# README mintlify Source: https://docs.phala.com/README_mintlify # Mintlify Usage ## Redirects Validation Run these two script to check and validate redirect links: .scripts/check\_links.py .scripts/validate-redirects.py # Batch check on-chain deployment allowance for all CVMs under an app Source: https://docs.phala.com/api-reference/apps/batch-check-on-chain-deployment-allowance-for-all-cvms-under-an-app /openapi.json post /api/v1/apps/{app_id}/cvms/is-allowed # Check if app deployment is allowed on-chain Source: https://docs.phala.com/api-reference/apps/check-if-app-deployment-is-allowed-on-chain /openapi.json post /api/v1/apps/{app_id}/is-allowed # Create a new instance under an existing app Source: https://docs.phala.com/api-reference/apps/create-a-new-instance-under-an-existing-app /openapi.json post /api/v1/apps/{app_id}/instances Deploy a new CVM instance under an existing app, optionally with a new Docker Compose file. For on-chain KMS apps, this uses a two-phase flow: the first request returns HTTP 465 with a commit token; after registering the compose hash on-chain, the second request commits with the token and transaction hash. # Create CVM replica Source: https://docs.phala.com/api-reference/apps/create-cvm-replica /openapi.json post /api/v1/apps/{app_id}/cvms/{vm_uuid}/replicas Clones a CVM to a target node. Uses source node if target not specified. # Get app attestation data Source: https://docs.phala.com/api-reference/apps/get-app-attestation-data /openapi.json get /api/v1/apps/{app_id}/attestations Returns TEE attestation quotes, event logs, and KMS configuration for running CVMs. # Get app details Source: https://docs.phala.com/api-reference/apps/get-app-details /openapi.json get /api/v1/apps/{app_id} Returns app metadata and current CVM configuration. Response schema varies by API version. # Get app filter options Source: https://docs.phala.com/api-reference/apps/get-app-filter-options /openapi.json get /api/v1/apps/filter-options Returns distinct values for each filterable field based on user's apps. # Get app metered usage Source: https://docs.phala.com/api-reference/apps/get-app-metered-usage /openapi.json get /api/v1/apps/{app_id}/usage Returns billing usage records from ClickHouse. Defaults to 7-day window if dates not specified. # Get app on-chain KMS info Source: https://docs.phala.com/api-reference/apps/get-app-on-chain-kms-info /openapi.json get /api/v1/apps/{app_id}/kms-info Returns on-chain KMS metadata for an app: chain, KMS contract, app contract, deployer address, and deployer type detection (EOA, Safe multisig, legacy multisig, ERC-4337, or generic contract). # Get device allowlist status Source: https://docs.phala.com/api-reference/apps/get-device-allowlist-status /openapi.json get /api/v1/apps/{app_id}/device-allowlist Returns the device_id allowlist status for an app's CVMs. For on-chain KMS apps, queries the blockchain in real-time to check which devices are registered in the app contract. # Get pending compose updates Source: https://docs.phala.com/api-reference/apps/get-pending-compose-updates /openapi.json get /api/v1/apps/{app_id}/pending-updates Returns provisioned compose file updates that have been signed on-chain but not yet applied to CVMs. # Get revision detail Source: https://docs.phala.com/api-reference/apps/get-revision-detail /openapi.json get /api/v1/apps/{app_id}/revisions/{revision_id} Returns full revision data including compose file, encrypted env, and user config. # List app CVMs Source: https://docs.phala.com/api-reference/apps/list-app-cvms /openapi.json get /api/v1/apps/{app_id}/cvms Returns CVM instances for an app. Admins see all instances; users see only their own. # List app revisions Source: https://docs.phala.com/api-reference/apps/list-app-revisions /openapi.json get /api/v1/apps/{app_id}/revisions Returns deployment history with compose hashes, operation types, and user refs. # List apps Source: https://docs.phala.com/api-reference/apps/list-apps /openapi.json get /api/v1/apps Returns paginated app list with filtering. Response schema varies by API version. # Preview a pending compose update Source: https://docs.phala.com/api-reference/apps/preview-a-pending-compose-update /openapi.json get /api/v1/apps/{app_id}/pending-updates/{vm_uuid}/{compose_hash}/preview Returns current and pending docker compose file and pre-launch script for diffing before applying an update. # Redeploy app from a specific revision Source: https://docs.phala.com/api-reference/apps/redeploy-app-from-a-specific-revision /openapi.json post /api/v1/apps/{app_id}/revisions/{revision_id}/redeploy Redeploy a CVM to a specific revision. If the app has exactly one CVM, it will be redeployed by default. If the app has multiple CVMs, you must specify target vm_uuids in request body. This schedules async CVM update task(s). # Run app TCB evaluation Source: https://docs.phala.com/api-reference/apps/run-app-tcb-evaluation /openapi.json get /api/v1/apps/{app_id}/attestations/tcb-evaluation Runs a live Intel PCS early collateral evaluation for each running CVM without using Redis collateral cache. # Check raw quote exists Source: https://docs.phala.com/api-reference/attestations/check-raw-quote-exists /openapi.json head /api/v1/attestations/raw/{checksum} Returns headers without body. Use to check existence and get Content-Length. # Download raw quote Source: https://docs.phala.com/api-reference/attestations/download-raw-quote /openapi.json get /api/v1/attestations/raw/{checksum} Returns raw quote bytes as binary file attachment. # Get quote collateral Source: https://docs.phala.com/api-reference/attestations/get-quote-collateral /openapi.json get /api/v1/attestations/collateral/{checksum} Returns TCB collateral data for quote verification. Cached for 24 hours. # List node device IDs Source: https://docs.phala.com/api-reference/attestations/list-node-device-ids /openapi.json get /api/v1/attestations/device_ids Public endpoint. Returns device_id list from nodes table (excluding deleted). # List node PPIDs Source: https://docs.phala.com/api-reference/attestations/list-node-ppids /openapi.json get /api/v1/attestations/ppids Public endpoint. Returns PPID list from nodes table (excluding deleted). # List nodes Source: https://docs.phala.com/api-reference/attestations/list-nodes /openapi.json get /api/v1/attestations/nodes Public endpoint. Returns NodeRef list derived from nodes table (excluding deleted). # List recent quotes Source: https://docs.phala.com/api-reference/attestations/list-recent-quotes /openapi.json get /api/v1/attestations/recent Public endpoint. Returns recently uploaded quotes ordered by timestamp. # Verify TEE quote Source: https://docs.phala.com/api-reference/attestations/verify-tee-quote /openapi.json post /api/v1/attestations/verify Accepts quote as file upload, form hex, or JSON hex. 0x prefix handled automatically. # View quote details Source: https://docs.phala.com/api-reference/attestations/view-quote-details /openapi.json get /api/v1/attestations/view/{checksum} Public endpoint. Returns parsed quote data with verification status. # Batch refresh CVM instance IDs Source: https://docs.phala.com/api-reference/cvms/batch-refresh-cvm-instance-ids /openapi.json patch /api/v1/cvms/instance-ids Resolves and persists missing instance_id values. Admin can pass cross-workspace CVM IDs. # Batch stop CVMs Source: https://docs.phala.com/api-reference/cvms/batch-stop-cvms /openapi.json post /api/v1/cvms/batch-stop Force stop multiple CVMs at once. Non-admin users can only stop CVMs in their current workspace. Admin users can stop any CVM. Max 50 per request. # Cancel in-progress CVM operation Source: https://docs.phala.com/api-reference/cvms/cancel-in-progress-cvm-operation /openapi.json post /api/v1/cvms/{cvm_id}/operations/cancel Cancel the current in-progress operation for a CVM. Revokes the Celery task and clears the operation lock. # Commit CVM replica (token-based, no auth required) Source: https://docs.phala.com/api-reference/cvms/commit-cvm-replica-token-based-no-auth-required /openapi.json post /api/v1/cvms/{cvm_id}/commit-replica Completes a two-phase CVM replication using a one-time commit token. The token was generated during a prepare-only or hash-required replica request. # Commit CVM update (token-based, no auth required) Source: https://docs.phala.com/api-reference/cvms/commit-cvm-update-token-based-no-auth-required /openapi.json post /api/v1/cvms/{cvm_id}/commit-update Completes a two-phase CVM update using a one-time commit token. The token was generated during a prepare-only PATCH request. No workspace authentication is required — the commit token is the credential. # Create CVM from provision Source: https://docs.phala.com/api-reference/cvms/create-cvm-from-provision /openapi.json post /api/v1/cvms Creates a CVM using cached provision data from /cvms/provision. Requires compose_hash from the provision response. # Create Eliza from provision Source: https://docs.phala.com/api-reference/cvms/create-eliza-from-provision /openapi.json post /api/v1/cvms/eliza Creates an Eliza CVM using cached provision data from /cvms/provision/eliza. Requires compose_hash from the provision response. # Create GPU workload CVM Source: https://docs.phala.com/api-reference/cvms/create-gpu-workload-cvm /openapi.json post /api/v1/cvms/workload Creates a CVM for GPU workloads. Node is auto-selected from available GPU nodes. # Delete CVM Source: https://docs.phala.com/api-reference/cvms/delete-cvm /openapi.json delete /api/v1/cvms/{cvm_id} Stops and permanently deletes CVM. Data is not recoverable. # Diagnose custom domain DNS Source: https://docs.phala.com/api-reference/cvms/diagnose-custom-domain-dns /openapi.json post /api/v1/cvms/{cvm_id}/customize-domain-diagnose Verify DNS records for custom domain configuration. Checks CNAME pointing to CVM, TXT records for app verification, and CAA records. # Force stop CVM Source: https://docs.phala.com/api-reference/cvms/force-stop-cvm /openapi.json post /api/v1/cvms/{cvm_id}/stop Immediately stops CVM (like pulling power). Cancels any in-progress operation. Triggers billing. # Get commit token info (no auth required) Source: https://docs.phala.com/api-reference/cvms/get-commit-token-info-no-auth-required /openapi.json get /api/v1/cvms/{cvm_id}/commit-update Read-only lookup of commit token metadata for the confirm-update page. # Get CVM attestation Source: https://docs.phala.com/api-reference/cvms/get-cvm-attestation /openapi.json get /api/v1/cvms/{cvm_id}/attestation Returns TEE attestation data including TCB info and app certificates. # Get CVM details Source: https://docs.phala.com/api-reference/cvms/get-cvm-details /openapi.json get /api/v1/cvms/{cvm_id} Returns full CVM details including status, network info, and KMS configuration. # Get CVM network info Source: https://docs.phala.com/api-reference/cvms/get-cvm-network-info /openapi.json get /api/v1/cvms/{cvm_id}/network Returns network status including internal IP, handshake time, and public URLs. # Get CVM system stats Source: https://docs.phala.com/api-reference/cvms/get-cvm-system-stats /openapi.json get /api/v1/cvms/{cvm_id}/stats Returns live system metrics (CPU, memory, disk) from the running CVM. # Get CVM user config Source: https://docs.phala.com/api-reference/cvms/get-cvm-user-config /openapi.json get /api/v1/cvms/{cvm_id}/user_config Returns the user_config stored in Teepod for this CVM, including injected SSH keys. # Get Docker Compose file Source: https://docs.phala.com/api-reference/cvms/get-docker-compose-file /openapi.json get /api/v1/cvms/{cvm_id}/docker-compose.yml Returns the raw Docker Compose YAML content from the CVM configuration. # Get firewall availability for this CVM's node Source: https://docs.phala.com/api-reference/cvms/get-firewall-availability-for-this-cvms-node /openapi.json get /api/v1/cvms/{cvm_id}/firewall-status Checks teescope capabilities.firewall on the node hosting this CVM. # Get network configuration Source: https://docs.phala.com/api-reference/cvms/get-network-configuration /openapi.json get /api/v1/cvms/{cvm_id}/network-config Returns real-time KMS URLs, Gateway URLs, and port mappings from VMM. # Get operation status Source: https://docs.phala.com/api-reference/cvms/get-operation-status /openapi.json get /api/v1/cvms/{cvm_id}/operation-status Returns current in-progress operation details including Celery task status. # Get port mappings Source: https://docs.phala.com/api-reference/cvms/get-port-mappings /openapi.json get /api/v1/cvms/{cvm_id}/ports Returns persisted port mappings for this CVM. # Get pre-launch script Source: https://docs.phala.com/api-reference/cvms/get-pre-launch-script /openapi.json get /api/v1/cvms/{cvm_id}/pre-launch-script Returns the shell script that runs before containers start. # Get raw compose file Source: https://docs.phala.com/api-reference/cvms/get-raw-compose-file /openapi.json get /api/v1/cvms/{cvm_id}/compose_file Returns the complete compose file as JSON or raw text based on query param. # Handle Batch Cvm Status Source: https://docs.phala.com/api-reference/cvms/handle-batch-cvm-status /openapi.json post /api/v1/status/batch Batch query CVM real-time status. Groups by teepod and queries in parallel, returns status updates with complete async operation information. # Handle Get Cvm State Source: https://docs.phala.com/api-reference/cvms/handle-get-cvm-state /openapi.json get /api/v1/cvms/{cvm_id}/state Get CVM state. Supports two modes: 1. Immediate mode (no target): Returns current state immediately 2. SSE mode (with target): Streams state updates until status matches target or timeout Example: GET /cvms/{cvm_id}/state - Returns current state GET /cvms/{cvm_id}/state?target=running - Streams until status becomes 'running' SSE connection limit: 5 concurrent connections per user # Handle Update Cvm Listed Source: https://docs.phala.com/api-reference/cvms/handle-update-cvm-listed /openapi.json patch /api/v1/cvms/{cvm_id}/listed # Handle Update Cvm Name Source: https://docs.phala.com/api-reference/cvms/handle-update-cvm-name /openapi.json patch /api/v1/cvms/{cvm_id}/name Update CVM name with RFC 1123 validation and availability check # Handle Update Cvm Scheduled Delete Source: https://docs.phala.com/api-reference/cvms/handle-update-cvm-scheduled-delete /openapi.json patch /api/v1/cvms/{cvm_id}/scheduled-delete Update CVM scheduled delete time # List available OS images Source: https://docs.phala.com/api-reference/cvms/list-available-os-images /openapi.json get /api/v1/cvms/{cvm_id}/available-os-images Returns OS images this CVM can upgrade to, filtered by node availability, KMS allowlist, dev/prod upgrade rules, and GPU compatibility. # List CVM operations Source: https://docs.phala.com/api-reference/cvms/list-cvm-operations /openapi.json get /api/v1/cvms/{cvm_id}/operations Returns operation records grouped by correlation_id. Only includes events with correlation_id (new format since 2024-01). Error messages are sanitized for non-admin users. # List CVMs Source: https://docs.phala.com/api-reference/cvms/list-cvms /openapi.json get /api/v1/cvms Returns CVMs in current workspace. Admins can filter by user_id or teepod_id. # List CVMs (paginated) Source: https://docs.phala.com/api-reference/cvms/list-cvms-paginated /openapi.json get /api/v1/cvms/paginated Returns paginated CVMs in current workspace with total count. # Provision compose file update Source: https://docs.phala.com/api-reference/cvms/provision-compose-file-update /openapi.json post /api/v1/cvms/{cvm_id}/compose_file/provision Validates new compose file and returns compose_hash for on-chain registration. Cache expires after 14 days. # Provision DStack app Source: https://docs.phala.com/api-reference/cvms/provision-dstack-app /openapi.json post /api/v1/cvms/provision Validates Docker Compose configuration and caches it for 14 days. Returns compose_hash and KMS info for on-chain registration before creation. # Provision Eliza app Source: https://docs.phala.com/api-reference/cvms/provision-eliza-app /openapi.json post /api/v1/cvms/provision/eliza Validates Eliza configuration and caches it for 14 days. Returns compose_hash and KMS info for on-chain registration. # Redeploy CVM from a specific revision Source: https://docs.phala.com/api-reference/cvms/redeploy-cvm-from-a-specific-revision /openapi.json post /api/v1/cvms/{cvm_id}/revisions/{revision_id}/redeploy Switch/rollback a CVM to a specified app revision (compose_hash). # Refresh CVM instance ID Source: https://docs.phala.com/api-reference/cvms/refresh-cvm-instance-id /openapi.json patch /api/v1/cvms/{cvm_id}/instance-id Resolve and persist instance_id for a CVM. Default behavior only backfills missing values. # Replicate CVM Source: https://docs.phala.com/api-reference/cvms/replicate-cvm /openapi.json post /api/v1/cvms/{cvm_id}/replicas Create a replica of the CVM on a target node. If request body is omitted, it replicates to the source node and reuses existing env. # Restart CVM Source: https://docs.phala.com/api-reference/cvms/restart-cvm /openapi.json post /api/v1/cvms/{cvm_id}/restart Stops and starts CVM. Use force=true to skip graceful shutdown. # Set port mappings Source: https://docs.phala.com/api-reference/cvms/set-port-mappings /openapi.json put /api/v1/cvms/{cvm_id}/ports Bulk-replace all port mappings for this CVM. Writes to DB and applies to VMM. # Shutdown CVM Source: https://docs.phala.com/api-reference/cvms/shutdown-cvm /openapi.json post /api/v1/cvms/{cvm_id}/shutdown Gracefully shuts down CVM (SIGTERM then SIGKILL after timeout). Triggers billing. # Start CVM Source: https://docs.phala.com/api-reference/cvms/start-cvm /openapi.json post /api/v1/cvms/{cvm_id}/start Powers on a stopped CVM. GPU allocation happens during start. # Submit compose file update Source: https://docs.phala.com/api-reference/cvms/submit-compose-file-update /openapi.json post /api/v1/cvms/{cvm_id}/compose_file Submit complete compose file with hash verification. Use when you have pre-computed compose_hash and want to skip provision step. # Trigger compose file update Source: https://docs.phala.com/api-reference/cvms/trigger-compose-file-update /openapi.json patch /api/v1/cvms/{cvm_id}/compose_file Applies a provisioned compose file update. Requires compose_hash from provision endpoint. Restarts the CVM with new configuration. # Update CVM Source: https://docs.phala.com/api-reference/cvms/update-cvm /openapi.json patch /api/v1/cvms/{cvm_id} Unified PATCH endpoint for CVM updates. Only fields present in the request body are applied. Handles resource resize, compose configuration, visibility, environment, and OS image updates. For contract-owned KMS, compose-hash-affecting changes require two-phase flow with on-chain hash registration. # Update Docker Compose file Source: https://docs.phala.com/api-reference/cvms/update-docker-compose-file /openapi.json patch /api/v1/cvms/{cvm_id}/docker-compose Replaces the docker-compose.yml and restarts the CVM. For contract-owned KMS, requires two-phase flow with on-chain hash registration. # Update environment variables Source: https://docs.phala.com/api-reference/cvms/update-environment-variables /openapi.json patch /api/v1/cvms/{cvm_id}/envs Updates encrypted environment variables. If env_keys changes for contract-owned KMS, requires two-phase flow with on-chain hash registration. # Update KMS/Gateway URLs Source: https://docs.phala.com/api-reference/cvms/update-kmsgateway-urls /openapi.json patch /api/v1/cvms/{cvm_id}/network-config Update KMS and/or Gateway URLs for this CVM. Requires VMM >= 0.5.7. # Update OS image Source: https://docs.phala.com/api-reference/cvms/update-os-image /openapi.json patch /api/v1/cvms/{cvm_id}/os-image Upgrades the CVM to a different OS image. Shuts down, updates, and restarts. Takes several minutes. Track progress via correlation_id. # Update pre-launch script Source: https://docs.phala.com/api-reference/cvms/update-pre-launch-script /openapi.json patch /api/v1/cvms/{cvm_id}/pre-launch-script Replaces the pre-launch script and restarts the CVM. For contract-owned KMS, requires two-phase flow with on-chain hash registration. # List family instance types Source: https://docs.phala.com/api-reference/instance-types/list-family-instance-types /openapi.json get /api/v1/instance-types/{family} Public endpoint. Returns all public instance types for a specific family. # List instance type families Source: https://docs.phala.com/api-reference/instance-types/list-instance-type-families /openapi.json get /api/v1/instance-types Public endpoint. Returns all families with their public instance types grouped. # Get app encryption pubkey Source: https://docs.phala.com/api-reference/kms/get-app-encryption-pubkey /openapi.json get /api/v1/kms/{kms_id}/pubkey/{app_id} Retrieve the public key for encrypting environment variables before deployment. # Get KMS details Source: https://docs.phala.com/api-reference/kms/get-kms-details /openapi.json get /api/v1/kms/{kms_id} Retrieve KMS instance details by hashed ID or slug. # Get on-chain KMS details by chain Source: https://docs.phala.com/api-reference/kms/get-on-chain-kms-details-by-chain /openapi.json get /api/v1/kms/on-chain/{chain} Returns KMS contracts on the specified chain with device and OS image whitelist status. # Inspect KMS instance Source: https://docs.phala.com/api-reference/kms/inspect-kms-instance /openapi.json get /api/v1/kms/{kms_id}/info Real-time inspection of a KMS instance. Queries KMS RPC, guest agent, associated Teepod VMMs, and on-chain contract data with cross-validation. # List KMS instances Source: https://docs.phala.com/api-reference/kms/list-kms-instances /openapi.json get /api/v1/kms Query available KMS instances with pagination. Filter by onchain status to find instances with blockchain integration. # Predict next app IDs Source: https://docs.phala.com/api-reference/kms/predict-next-app-ids /openapi.json get /api/v1/kms/phala/next_app_id Predict the next N app IDs that will be assigned to the team's wallet. Maximum 20 predictions allowed. # Add SSH key Source: https://docs.phala.com/api-reference/ssh-keys/add-ssh-key /openapi.json post /api/v1/user/ssh-keys Validates and stores an SSH public key. Minimum RSA 2048-bit, Ed25519 recommended. # Delete SSH key Source: https://docs.phala.com/api-reference/ssh-keys/delete-ssh-key /openapi.json delete /api/v1/user/ssh-keys/{key_id} Soft-deletes an SSH key. Key must belong to the authenticated user. # Import keys from GitHub profile Source: https://docs.phala.com/api-reference/ssh-keys/import-keys-from-github-profile /openapi.json post /api/v1/user/ssh-keys/github-profile Fetches public SSH keys from any GitHub user's profile via the unauthenticated GitHub API. Deduplicates against the user's existing keys by fingerprint. # List user SSH keys Source: https://docs.phala.com/api-reference/ssh-keys/list-user-ssh-keys /openapi.json get /api/v1/user/ssh-keys Returns all SSH keys belonging to the authenticated user. # Sync keys from GitHub Source: https://docs.phala.com/api-reference/ssh-keys/sync-keys-from-github /openapi.json post /api/v1/user/ssh-keys/github-sync Imports SSH keys from linked GitHub account. Adds new keys, updates existing, removes deleted. # Create workspace webhook Source: https://docs.phala.com/api-reference/webhooks/create-workspace-webhook /openapi.json post /api/v1/workspace/webhooks # Delete webhook Source: https://docs.phala.com/api-reference/webhooks/delete-webhook /openapi.json delete /api/v1/workspace/webhooks/{webhook_id} # Get webhook delivery stats Source: https://docs.phala.com/api-reference/webhooks/get-webhook-delivery-stats /openapi.json get /api/v1/workspace/webhooks/{webhook_id}/stats # Get webhook details Source: https://docs.phala.com/api-reference/webhooks/get-webhook-details /openapi.json get /api/v1/workspace/webhooks/{webhook_id} # List webhook deliveries Source: https://docs.phala.com/api-reference/webhooks/list-webhook-deliveries /openapi.json get /api/v1/workspace/webhooks/{webhook_id}/deliveries # List workspace webhooks Source: https://docs.phala.com/api-reference/webhooks/list-workspace-webhooks /openapi.json get /api/v1/workspace/webhooks # Resend a webhook delivery Source: https://docs.phala.com/api-reference/webhooks/resend-a-webhook-delivery /openapi.json post /api/v1/workspace/webhooks/{webhook_id}/deliveries/{event_id}/resend # Reveal webhook secret (requires 2FA) Source: https://docs.phala.com/api-reference/webhooks/reveal-webhook-secret-requires-2fa /openapi.json post /api/v1/workspace/webhooks/{webhook_id}/reveal-secret # Rotate webhook secret (requires 2FA) Source: https://docs.phala.com/api-reference/webhooks/rotate-webhook-secret-requires-2fa /openapi.json post /api/v1/workspace/webhooks/{webhook_id}/rotate-secret # Send test webhook event Source: https://docs.phala.com/api-reference/webhooks/send-test-webhook-event /openapi.json post /api/v1/workspace/webhooks/{webhook_id}/test # Update webhook Source: https://docs.phala.com/api-reference/webhooks/update-webhook /openapi.json put /api/v1/workspace/webhooks/{webhook_id} # Doc requirements Source: https://docs.phala.com/doc-requirements Act as a 10 year technical writer. Now you are performing an audit of the given doc page in this docs repo. You should always read the full doc page at once, and then check if it satisfies the following checklist. # Check list 1. Content is self contained 2. Content is helpful to the audience 3. Ensure no redundant info is repeated in the doc 4. Always use conversational language 5. No text-wall. Each paragraph should be less than 4 sentences. 6. Flow of the doc is simple and clear 7. Human-style natural writing rather than AI-style 8. Lead with the key takeaway 9. Tutorials should guide step by step with context and purpose (if applicable) 10. Respect documentation types and their promises 11. Avoid bullet point overload # Explaination of each criteria ## Ensure no redundant info is repeated in the doc Watch for sections that present the same information from different perspectives without adding new value. Common redundancy patterns: **Opposite perspectives that say the same thing:** * "What We Protect Against" (threat model) vs "What You Don't Need to Trust" (trust model) * If section A says "we protect against X" and section B says "you don't trust X", they're often redundant * Consider combining them into a single section **Same concept explained multiple times:** * Hardware encryption mentioned in overview, threat model, architecture, and trust model * Gateway TEE protection explained in 3+ different sections * Ask: Does each mention add new information, or just rephrase? **How to fix:** * Combine related sections when they cover the same ground * Keep one detailed explanation, use brief references elsewhere * If repetition serves a purpose (overview vs deep-dive), make sure the detail level differs significantly Good example: A single "Security Model" section with subsections for threats, trust requirements, and limitations. Bad example: Separate "Threat Model" and "Trust Model" sections that list the same protections from opposite angles. ## Content is self contained Ask a few questions to yourself and make sure the answer is yes: * Can you understand all the contents in this doc? * Do you feel confused why you are reading a piece of the content? ## Content is helpful to the audience Our audience are developers with CS foundamentals, knowledge of AI and Confidential Computing. Ensure anything we write in the doc is helpful to the audience. Ask a few questions: * Does the doc help the audience to learn more? (Should be Yes) * Does the language sounds like a sales pitch? (Should be No) ## Human-style natural writing rather than AI-style The doc should have a very natural flow. Ensure: * Conversational but precise. Write like you're explaining to a peer, not a professor. * Short sentences. One idea per sentence. Avoid comma chains and overlong clauses. * Use active voice. "TEE mode scales well" > "The scalability of TEE mode is underscored." * Include links/tables/figures only if they add real value (not filler). * Minimize em-dashes for better flow. Em-dashes create dramatic pauses but feel heavy when overused. Limit to 1-2 per page, reserved for high-impact emphasis. Replace most em-dashes with "because", "so", commas, or separate sentences. Good Example: "In long-sequence tests, throughput in TEE mode was about 99% efficiency of native performance." Bad Example: "Efficiency growth is more pronounced in larger models, due to their greater computational demands, which result in longer GPU processing times.” (dense, academic)" ## Lead with the key takeaway The doc should open with the most important result or insight, not the setup or background. Ask yourself: * Does the first sentence clearly tell the reader why this page matters? * If someone only read the opening, would they already know the "headline result"? Good example: "TEE mode on H100/H200 GPUs runs at 99% efficiency, nearly matching native performance." Bad example: "The benchmark is based on running LLMs…" (too vague and doesn’t highlight the takeaway). ## Tutorials should guide step by step with context and purpose Identify if the doc is a tutorial or a reference page. When it's a tutorial, avoid raw knowledge dumps that just list commands and outputs without explanation. Ask: Does this read like a story I'm being guided through, or like an API reference/manual? ## Respect documentation types and their promises Different documentation types serve different purposes. Never mix them up: * Tutorials: Learning-oriented content for new users * How-to guides: Task-oriented guidance for specific problems * Explanations: Understanding-oriented conceptual discussions * Reference: Information-oriented technical descriptions Bad example: A "quickstart" that includes full API specifications, conceptual explanations, and multiple advanced scenarios. Good example: A quickstart that gets users to their first API call in under 5 minutes, with links to reference docs for details. ## Avoid bullet point overload Too many bullet points make documentation feel like an outline, not documentation for humans. Rules: * Maximum 3-5 bullets before breaking with prose * Use bullets for true lists, not as an organizational crutch * Conversational flow > perfect organization Bad example: * Step 1: Do this * Step 2: Do that * Note: Remember this * Important: Don't forget * Also: Consider this * Furthermore: Think about that * Additionally: Keep in mind Good example: Here's how to verify your quote. First, you'll need either a binary file or hex string. Choose your method: * Binary file upload for quote.bin files * Hex string for data from APIs * Direct checksum lookup if already verified After verification, you'll get a unique checksum that serves as your quote's permanent identifier. Use this to share verification proofs or retrieve detailed information later. # API Reference Source: https://docs.phala.com/dstack-cloud/api-reference API reference for Guest Agent (Unix Socket), KMS Onboard API, and management endpoints. # API Reference dstack-cloud exposes three API surfaces: the Guest Agent's Unix socket (for apps inside the CVM), the KMS RPC (for key delivery over RA-TLS), and the Onboard HTTP endpoints (for first-time KMS bootstrap). ## Guest Agent API (Unix Socket) The Guest Agent runs inside each dstack CVM and provides local APIs via a Unix socket at `/var/run/dstack.sock`. ### Get TDX Quote Obtain a TDX attestation quote from the hardware. **GCP (TDX) only.** ```bash theme={"system"} curl --unix-socket /var/run/dstack.sock \ "http://localhost/GetQuote?report_data=0x1234deadbeef" ``` **Parameters:** | Parameter | Type | Description | | ------------- | ------------ | -------------------------------------------------------------------- | | `report_data` | string (hex) | Optional challenge value (32 bytes). Used to prevent replay attacks. | **Response:** ```json theme={"system"} { "quote": "", "rtmr0": "...", "rtmr1": "...", "rtmr2": "...", "rtmr3": "..." } ``` ### Get NSM Attestation Obtain a Nitro Attestation Document from the NSM. **AWS Nitro only.** ```bash theme={"system"} curl --unix-socket /var/run/dstack.sock \ "http://localhost/GetAttestation?user_data=0x1234deadbeef" ``` **Parameters:** | Parameter | Type | Description | | ----------- | ------------ | ------------------------- | | `user_data` | string (hex) | Optional challenge value. | **Response:** ```json theme={"system"} { "document": "" } ``` ### Get Attestation (HTTP) External attestation endpoint, accessible via HTTPS. ```bash theme={"system"} curl https://your-app.example.com/attestation ``` **Response:** Full attestation data (Quote or Document) for external verification. *** ## KMS API The KMS exposes an RPC interface for key management. All communication uses RA-TLS — the KMS verifies the workload's attestation before processing any request. ### getKey(name) Request a key from the KMS. ```bash theme={"system"} # Called from within the application via dstack SDK # Not directly callable via curl (requires RA-TLS handshake) ``` **Parameters:** | Parameter | Type | Description | | --------- | ------ | --------------------------------------------------------------- | | `name` | string | The name of the requested key. Keys are scoped per application. | **Returns:** The requested key as raw bytes. **Error Codes:** | Error | Description | | ----------------------- | ---------------------------------------------------------------------------------------- | | `UNAUTHORIZED` | Workload attestation verification failed. The workload is not running in a verified TEE. | | `MEASUREMENT_NOT_FOUND` | The workload's measurement is not registered on-chain. | | `KEY_NOT_FOUND` | No key exists for the requested name. | | `INTERNAL_ERROR` | KMS encountered an internal error. | ### How Applications Use getKey Applications retrieve keys through the dstack SDK, which handles attestation and the RA-TLS connection automatically: **Python:** ```python theme={"system"} from dstack import DstackClient client = DstackClient() key = client.get_key("my-api-key") ``` **TypeScript:** ```typescript theme={"system"} import { DstackClient } from '@dstack/sdk'; const client = new DstackClient(); const key = await client.getKey("my-api-key"); ``` **Rust:** ```rust theme={"system"} use dstack_sdk::DstackClient; let client = DstackClient::new()?; let key = client.get_key("my-api-key")?; ``` *** ## KMS Onboard API (HTTP, Bootstrap Only) These endpoints are only available during the first-time bootstrap (Onboard mode). After bootstrap is completed, KMS switches to RA-TLS-only mode. ### Onboard.Bootstrap Generate the KMS key pair and obtain attestation information. ```bash theme={"system"} curl -s "http://:12001/prpc/Onboard.Bootstrap?json" \ -d '{"domain": ""}' ``` **Parameters:** | Parameter | Type | Description | | --------- | ------ | ------------------------------------------------------------------------------------ | | `domain` | string | The domain name or IP address where KMS is accessible. Used in the attestation data. | **Response:** ```json theme={"system"} { "publicKey": "...", "attestation": "...", "measurement": "..." } ``` ### /finish Complete the bootstrap process. KMS restarts and switches to Normal mode (HTTPS + RA-TLS). ```bash theme={"system"} curl "http://:12001/finish" ``` **Response:** HTTP 200 on success. *** ## Docker Compose Volume for Guest Agent To access the Guest Agent from within a Docker container, mount the socket: ```yaml theme={"system"} services: my-app: image: my-app:latest volumes: - /var/run/dstack.sock:/var/run/dstack.sock ``` ## Next Steps * **[Configuration Reference](configuration)** — Configuration file formats * **[Glossary](glossary)** — Term definitions # Attestation Integration Source: https://docs.phala.com/dstack-cloud/attestation-integration How dstack-cloud integrates remote attestation mechanisms on GCP (TDX Quote) and AWS Nitro (NSM Attestation Document). # Attestation Integration Remote attestation is how a TEE proves it's running genuine, unmodified code on real hardware. dstack-cloud integrates two different attestation mechanisms — Intel TDX on GCP and AWS NSM on Nitro — to give KMS confidence before dispatching keys. > For the fundamentals of dstack Remote Attestation, the RA-TLS protocol design, and ZT-TLS / ACME certificate management, refer to the official Phala documentation: > > * [TEE Attestation Guide](https://docs.phala.com/dstack/trust-center-technical) > * [TEE-Controlled Domain Certificates](https://docs.phala.com/dstack/design-documents/tee-controlled-domain-certificates) ## What Is Remote Attestation Remote Attestation is the process by which a TEE proves its identity and integrity to an external party. The proof contains: 1. **Hardware signature** — Confirms the workload is running in genuine TEE hardware (not emulated) 2. **Workload measurements** — Cryptographic hash of the code and configuration running inside the TEE 3. **User data** — An optional challenge value to prevent replay attacks The verifier (e.g., KMS) checks the hardware signature and measurements to confirm the workload has not been tampered with. ## Attestation on GCP (Intel TDX + vTPM) On GCP, dstack uses a **dual-attestation approach** combining Intel TDX and Google-managed vTPM to build a comprehensive trust chain. ### Architecture Overview GCP Confidential VMs provide two independent attestation mechanisms: | Source | Type | Coverage | Trust Root | | -------------------- | -------------- | ------------------------------ | ------------------------------ | | **Intel TDX Module** | Hardware-based | Firmware + RTMR registers | Intel TDX hardware | | **Google vTPM** | Software-based | Boot loader + kernel integrity | Google's certificate authority | dstack binds these two mechanisms together to provide defense-in-depth attestation. ### Attestation Creation Flow When a dstack CVM starts on GCP, the Guest Agent performs the following steps: 1. **Collect report\_data** (64 bytes), optionally bound to an RA-TLS public key 2. **Generate TDX Quote** via `tdx-attest::get_quote(report_data)`: * Includes all measurement values (MRTD, RTMR0-3) * Signed by the Intel TDX module 3. **Read TDX event log** via `cc-eventlog::tdx::read_event_log()` 4. **Compute TPM qualifying data** as `sha256(tdx_quote)` — this binds the two attestations together 5. **Generate vTPM Quote** via `tpm-attest::TpmContext::create_quote(qualifying_data, policy)`: * Signed by Google's vTPM attestation key * Includes PCR values representing boot chain integrity 6. **Package attestation** as `DstackGcpTdxQuote { tdx_quote, tpm_quote }` ### Trust Chain Binding The key innovation is the **qualifying data binding**: ``` tpm_quote.qualifying_data = sha256(tdx_quote) ``` This creates a cryptographic link between the two independent attestation mechanisms: * **TDX Quote** proves the workload runs in genuine Intel TDX hardware with specific RTMR values * **vTPM Quote** proves the boot chain integrity via PCR values * The SHA-256 binding ensures both attestations come from the same CVM instance ### Verification Flow The verifier (e.g., KMS) performs dual verification: **TDX Verification:** 1. Retrieve TDX collateral and verify quote signature using Intel's TCB info 2. Verify TCB status (debug mode disabled, mr\_signer\_seam valid) 3. Replay runtime events and compare RTMR3 with quote's RTMR3 4. Verify `report_data` matches expected value **vTPM Verification:** 1. Retrieve vTPM collateral and verify quote signature using Google's certificate chain 2. Replay runtime events and compare PCR values 3. Verify `qualifying_data == sha256(tdx_quote)` — confirms binding Only when **both** verifications succeed is the attestation considered valid ### Measurement Registers (GCP) **TDX RTMR Registers:** | Register | Contents | | --------- | ------------------------------------------------------ | | **MRTD** | Measured firmware (virtual firmware measurement) | | **RTMR0** | OS kernel and initramfs | | **RTMR1** | OS kernel command-line parameters | | **RTMR2** | OS runtime and init scripts | | **RTMR3** | Application `compose-hash` + KMS binding + instance ID | **vTPM PCR Registers:** The vTPM provides an independent measurement of the boot chain. While dstack primarily relies on TDX RTMRs for application attestation, the vTPM PCRs offer an additional layer of verification for the boot process. > **Note:** For detailed PCR assignments and event log structure, refer to the [GCP Confidential VM Attestation Documentation](https://cloud.google.com/confidential-computing/confidential-vm/docs/attestation). ## Attestation on AWS Nitro (NSM) When a Nitro Enclave launches from its EIF (Enclave Image File): 1. **NSM auto-generation** — The Nitro Secure Module automatically generates an Attestation Document when the Enclave launches 2. **Document contents** — The Attestation Document includes: * PCR0-2 values (Platform Configuration Registers) — 3 measurements generated during EIF build * NSM signature * Enclave ID and launch timestamp 3. **OS\_IMAGE\_HASH** — The 3 PCR values (PCR0, PCR1, PCR2) are combined into a single `OS_IMAGE_HASH`, which serves as the unique workload identifier 4. **Verification** — KMS verifies the Attestation Document: * Validates the NSM signature using AWS root certificates * Checks that the `OS_IMAGE_HASH` matches an authorized value registered on-chain * Confirms the Enclave is running on genuine Nitro hardware ### Measurement Registers (Nitro) | Register | Contents | | -------- | ---------------------------------------- | | **PCR0** | OS image (kernel + initramfs) | | **PCR1** | Application code and runtime | | **PCR2** | Additional configuration and dstack-util | These 3 PCR values are combined into a single `OS_IMAGE_HASH`, which is the unique identifier used for on-chain authorization. Any change to the Dockerfile, application code, or `dstack-util` version produces a different `OS_IMAGE_HASH`. ## GCP vs. Nitro: Attestation Comparison Both platforms generate hardware-signed attestations that KMS verifies before dispatching keys. The main differences are in the attestation format (TDX Quote vs. NSM Document), verification method (Intel DCAP QVL vs. AWS certificate chain), and how measurements are stored (RTMR registers vs. PCR values). For platform-specific details, see the sections above. For a broader architectural comparison, see [AWS Nitro Enclave Integration](nitro-enclave). ## Measurements and On-chain Authorization Measurements are the foundation of on-chain key access control: * Before KMS dispatches keys to a workload, the workload's measurements must be **registered on-chain** * This ensures only audited, approved code versions can obtain keys * If you update your application code, the measurements change, and you must re-register them through governance Measurement Registration Flow For the registration process, see [Register Workload Measurements](/dstack-cloud/register-enclave-measurement). ## Verifying Attestation You can verify that a dstack CVM is running in genuine TEE by requesting its attestation: ```bash theme={"system"} # Request attestation from a running CVM curl https://your-app.example.com/attestation ``` The response includes the full attestation data that can be independently verified using: * **dstack-verifier** — HTTP service or CLI tool provided by dstack-cloud * **dcap-qvl** — Open-source Intel DCAP Quote Verification Library * **dstack SDK** — `replayRtmrs()` function for local verification For more details, see the [dstack-cloud Verification Guide](https://github.com/Phala-Network/dstack-cloud/blob/master/docs/verification.md). ## Next Steps * **[Security Model](security-model)** — Trust boundaries and security guarantees * **[Register Workload Measurements](/dstack-cloud/register-enclave-measurement)** — How to register measurements on-chain * **[Runbook](/dstack-cloud/runbook)** — Troubleshooting attestation failures # Code Walkthrough and KT Materials Source: https://docs.phala.com/dstack-cloud/code-walkthrough Code walkthrough and knowledge transfer materials for the dstack-cloud project. # Code Walkthrough and KT Materials This page is an internal knowledge-transfer resource. It maps the codebase structure, traces the key request path from application to KMS, and highlights the files you'll need to understand when contributing to dstack-cloud. > **Note:** This document is an internal deliverable for new contributors and team members. ## Guide for New Contributors If you're new to dstack-cloud, here's the recommended reading order: 1. **Understand the architecture** — Read [overview](overview) first to get the big picture 2. **Set up the development environment** — Follow the README in the repository root 3. **Start with a simple workload** — Deploy a basic nginx container on GCP using the Quick Start 4. **Explore the KMS** — Deploy KMS and observe the bootstrap flow 5. **Read the attestation code** — Understand how measurements are generated and verified 6. **Review the contracts** — Understand the governance model and on-chain authorization ## Repository Structure The codebase is organized into these key areas: ``` dstack-cloud/ ├── cli/ # dstack-cloud CLI tool ├── kms/ # dstack-kms service ├── packages/ │ ├── attestation/ # Attestation modules (TDX, NSM, TPM) │ ├── guest-agent/ # Guest Agent (runs inside CVM) │ ├── gateway/ # TLS termination and RA-TLS gateway │ └── vmm/ # Virtual Machine Monitor ├── contracts/ # Smart contracts (DstackKms, DstackApp) ├── scripts/ # Build and deployment scripts └── docs/ # Documentation ``` ## Core Request Paths Understanding how a key request flows through the system is the best way to learn the codebase. Here's what happens when an application asks KMS for a key: ### Key Request Flow **GCP (via Guest Agent):** ``` Application (in CVM) → dstack SDK (via /var/run/dstack.sock) → Guest Agent → RA-TLS connection → dstack-kms (in separate TEE) → Attestation verification → On-chain measurement check → Key derivation and dispatch → RA-TLS response → Guest Agent → Application receives key ``` **AWS Nitro (via dstack-util):** ``` dstack-util (in Enclave) → NSM attestation document obtained → VSOCK → VSOCK Proxy → dstack-kms (in separate TEE) → Attestation verification (NSM + OS_IMAGE_HASH) → On-chain measurement check → Key derivation and dispatch → Key returned to dstack-util → Application receives key (user decides usage) ``` Key Request Flow ### CVM Deployment Flow When you run `dstack-cloud deploy`, the CLI parses your configuration and orchestrates the creation of a TEE environment. The flow differs by platform: **GCP (dstack CVM):** ``` dstack-cloud deploy → Parse docker-compose.yaml → Build CVM image (dstack-os + containers) → Generate measurements (RTMR values) → Create Confidential VM with TDX → Guest Agent starts inside CVM → Attestation obtained from hardware ``` **AWS Nitro (Enclave):** ``` dstack-cloud deploy → Build Docker image from Dockerfile → Run nitro-cli build-enclave → generates EIF → 3 PCRs (PCR0-2) produced at build time → Combine 3 PCRs into 1 OS_IMAGE_HASH → Register OS_IMAGE_HASH on-chain (via governance) → Launch Enclave on EC2 instance → dstack-util handles attestation and key retrieval ``` CVM Deployment Flow ## Attestation Module The attestation module abstracts platform-specific hardware attestation behind a common interface. Each platform has its own module: | Platform | Module | Input | Output | | --------- | ------------ | ------------ | -------------------- | | GCP (TDX) | `tdx-attest` | TDX hardware | TDX Quote | | AWS Nitro | `nsm-attest` | NSM device | Attestation Document | | GCP (TPM) | `tpm-attest` | TPM device | TPM Quote | ## Key Files These are the files you'll spend the most time in when contributing: | File | Purpose | | ----------------------------------- | ------------------------------------------------------ | | `kms/src/main.rs` | KMS service entry point, RPC handlers, bootstrap logic | | `packages/guest-agent/src/main.rs` | Guest Agent entry point, local API server | | `packages/attestation/src/lib.rs` | Platform-agnostic attestation interface | | `cli/src/main.rs` | CLI entry point, deploy/status/logs commands | | `contracts/contracts/DstackKms.sol` | KMS policy contract | | `contracts/contracts/DstackApp.sol` | Application contract | ## Resources * [dstack-cloud GitHub](https://github.com/Phala-Network/dstack-cloud) * [dstack framework GitHub](https://github.com/Dstack-TEE/dstack) * [dstack Official Documentation](https://docs.phala.com/dstack/overview) * [dstack Whitepaper](https://docs.phala.com/dstack/design-documents/whitepaper) # Configuration Reference Source: https://docs.phala.com/dstack-cloud/configuration Configuration reference for dstack-cloud global config, app.json, docker-compose.yaml, and CLI commands. # Configuration Reference dstack-cloud uses three layers of configuration: a global CLI config, a per-project `app.json`, and a standard `docker-compose.yaml` for your application. This page documents every field and available option. ## dstack-cloud Global Configuration **Location:** `~/.config/dstack-cloud/config.json` ```json theme={"system"} { "image_search_paths": ["/path/to/your/images"], "gcp": { "project": "your-gcp-project-id", "zone": "us-central1-a", "bucket": "gs://your-bucket-name" }, "nitro": { "region": "us-east-1" } } ``` ### Fields | Field | Type | Description | | -------------------- | --------------- | -------------------------------------------------- | | `image_search_paths` | array of string | Local paths where dstack-cloud looks for OS images | | `gcp.project` | string | GCP project ID | | `gcp.zone` | string | GCP zone for VM deployment | | `gcp.bucket` | string | GCS bucket for storing CVM images | | `nitro.region` | string | AWS region for Nitro Enclave deployment | ## app.json (Project Configuration) **Location:** `/app.json` Generated by `dstack-cloud new` and edited manually for advanced configuration. ```json theme={"system"} { "os_image": "dstack-cloud-0.6.0", "key_provider": "local", "instance_name": "my-app", "platform": "nitro" } ``` ### Fields | Field | Type | Description | | --------------- | ------ | ----------------------------------------------------- | | `os_image` | string | The dstack OS image version to use | | `key_provider` | string | Key provider mode: `local` (default), `tpm`, or `kms` | | `instance_name` | string | Human-readable name for the instance | | `platform` | string | Target platform: `gcp` or `nitro` | ### key\_provider Values | Value | Description | Use Case | | ------- | ------------------------------------------- | ---------------------------------------------------- | | `local` | Keys generated locally within the CVM | Development, testing, single-node | | `tpm` | Use the platform TPM as root of trust | KMS instances on GCP | | `kms` | Use an external dstack-kms for key delivery | Production workloads that need attested key delivery | ## docker-compose.yaml The standard Docker Compose file defines your application. dstack-cloud reads this file and packages all containers into the CVM. ### dstack-specific Extensions dstack-cloud reads the standard `docker-compose.yaml` format. No special extensions are required. **Important notes:** * All images must be pullable by the build system (use public registries or pre-pull images) * Use SHA256 digests for pinned images (recommended for reproducible measurements): ```yaml theme={"system"} services: web: image: nginx:latest@sha256:abc123... ``` * The `runtime: nvidia` field is supported for GPU workloads on compatible instances * `volumes` that reference `/var/run/dstack.sock` are automatically mounted for Guest Agent access ### Example: Web Application ```yaml theme={"system"} services: web: image: nginx:latest ports: - "80:80" ``` ### Example: AI Inference with GPU ```yaml theme={"system"} services: vllm: image: vllm/vllm-openai:latest runtime: nvidia command: --model Qwen/Qwen2.5-7B-Instruct ports: - "8000:8000" volumes: - /var/run/dstack.sock:/var/run/dstack.sock ``` ### Example: KMS Instance ```yaml theme={"system"} services: dstack-kms: image: phalanetwork/dstack-kms:latest environment: - KMS_HTTPS_PORT=12001 - ETH_RPC_URL=https://sepolia.base.org - KMS_CONTRACT_ADDR=0x... - APP_CONTRACT_ADDR=0x... ports: - "12001:12001" ``` ## .env (Environment Variables) **Location:** `/.env` Standard dotenv format. Used for environment variables that are injected into the CVM. ```bash theme={"system"} API_KEY=your-api-key-here DATABASE_URL=postgres://user:pass@host:5432/db ``` ### KMS Environment Variables | Variable | Description | Example | | ------------------- | --------------------------------------------- | -------------------------- | | `KMS_HTTPS_PORT` | Port for KMS HTTPS/RA-TLS service | `12001` | | `ETH_RPC_URL` | Ethereum RPC endpoint URL | `https://sepolia.base.org` | | `KMS_CONTRACT_ADDR` | DstackKms contract address | `0x1234...abcd` | | `APP_CONTRACT_ADDR` | DstackApp contract address | `0x5678...efgh` | | `USE_LIGHT_CLIENT` | Use helios light client instead of direct RPC | `true` or `false` | ### Security of Environment Variables * Environment variables are encrypted before leaving your machine * They are decrypted only inside the CVM/TEE * The cloud provider and host OS cannot read them in plaintext ## prelaunch.sh (Pre-launch Script) **Location:** `/prelaunch.sh` An optional shell script that runs before the CVM/Enclave launches. Common uses: * Start the VSOCK proxy (Nitro) * Inject environment variables * Generate dynamic configuration ```bash theme={"system"} #!/bin/bash # prelaunch.sh — runs before CVM launch # Start VSOCK proxy (Nitro only) socat VSOCK-LISTEN:8000,reuseaddr,fork TCP:localhost:8000 & # Inject environment variables export KMS_HTTPS_PORT=12001 export ETH_RPC_URL=https://sepolia.base.org echo "Prelaunch complete" ``` ## dstack-cloud CLI Commands | Command | Description | | ---------------------------------------- | -------------------------------------------- | | `dstack-cloud new ` | Create a new project directory | | `dstack-cloud deploy` | Build and deploy the CVM/Enclave | | `dstack-cloud status` | Show deployment status and measurements | | `dstack-cloud logs [--follow]` | View container logs | | `dstack-cloud stop` | Stop the running CVM/Enclave | | `dstack-cloud start` | Start a stopped CVM/Enclave | | `dstack-cloud remove` | Remove the deployment and clean up resources | | `dstack-cloud fw allow ` | Allow inbound traffic on a port | | `dstack-cloud fw deny ` | Deny inbound traffic on a port | | `dstack-cloud pull --os-image ` | Download an OS image | | `dstack-cloud config-edit` | Edit the global configuration | ## Next Steps * **[API Reference](api-reference)** — Guest Agent and KMS APIs * **[Glossary](glossary)** — Term definitions # Deploy On-chain KMS Smart Contracts Source: https://docs.phala.com/dstack-cloud/deploy-onchain-kms Deploy DstackKms smart contracts and configure governance with Multisig and Timelock. # Deploy On-chain KMS Smart Contracts Deploy the `DstackKms` smart contract to enforce that only authorized workloads can receive keys. This page covers the full contract deployment workflow, including the recommended Safe + Timelock governance setup for production. ## Overview | Component | Required | Description | | ---------------------- | -------- | --------------------------------------------------------------------------- | | **DstackKms** | Yes | Stores authorized workload measurements, admin roles, and KMS configuration | | **ERC1967Proxy** | Yes | Proxy contract for upgradeable DstackKms (UUPS pattern) | | **TimelockController** | Optional | Enforces a delay on governance actions | | **Safe (Multisig)** | Optional | Multi-signature wallet for governance | > **Note:** Safe and Timelock are **optional security enhancements**, not part of dstack. They are recommended for production but not required for development. ## Governance Models ### Model A: Direct Admin (Simplest) * Admin is a single EOA (externally owned account) * No multisig, no timelock * Governance actions execute immediately * Suitable for development and testing ### Model B: Timelock Only * Admin is a TimelockController contract * Governance actions require a delay before execution * Anyone can execute after delay (or restricted to specific executors) * Suitable for simple production setups ### Model C: Safe + Timelock (Recommended for Production) * Admin is a Safe multisig wallet * Timelock enforces a delay * Requires multi-party approval + delay period * Maximum security and transparency *** ## Prerequisites * [Foundry](https://book.getfoundry.sh/getting-started/installation) installed (`forge`, `cast`) * A wallet with funds for deployment gas * Testnet: Use a faucet (e.g., [Base Sepolia faucet](https://www.alchemy.com/faucets/base-sepolia)) * Mainnet: Sufficient ETH for contract deployment * RPC endpoint for the target network *** ## Step 1: Set Up the Project ```bash theme={"system"} # Clone the dstack repository (contains KMS contracts) git clone https://github.com/Dstack-TEE/dstack.git cd dstack/kms/auth-eth/contracts # Install dependencies (OpenZeppelin) forge install OpenZeppelin/openzeppelin-contracts@v5.6.1 --no-git forge install OpenZeppelin/openzeppelin-contracts-upgradeable@v5.6.1 --no-git # Build contracts forge build ``` *** ## Step 2: Configure Environment Create a `.env` file: ```bash theme={"system"} # .env RPC_URL=https://sepolia.base.org PRIVATE_KEY=0x... ``` > **Security:** Never commit your private key. Add `.env` to `.gitignore`. *** ## Step 3: Deploy DstackKms (Basic, No Timelock) For development/testing, you can deploy DstackKms with direct EOA admin: ```bash theme={"system"} # Deploy DstackKms implementation KMS_IMPL=$(forge create src/DstackKms.sol:DstackKms \ --broadcast \ --rpc-url $RPC_URL \ --private-key $PRIVATE_KEY | grep "Deployed to:" | tail -1 | cut -d' ' -f3) echo "DstackKms implementation: $KMS_IMPL" # Encode initializer calldata DEPLOYER=$(cast wallet address --private-key $PRIVATE_KEY) INIT_DATA=$(cast calldata "initialize(address,address)" $DEPLOYER $ZERO_ADDRESS) # Deploy ERC1967Proxy KMS_PROXY=$(forge create lib/openzeppelin-contracts/contracts/proxy/ERC1967/ERC1967Proxy.sol:ERC1967Proxy \ --broadcast \ --rpc-url $RPC_URL \ --private-key $PRIVATE_KEY \ --constructor-args $KMS_IMPL $INIT_DATA | grep "Deployed to:" | tail -1 | cut -d' ' -f3) echo "DstackKms proxy: $KMS_PROXY" ``` > **Why ERC1967Proxy?** DstackKms uses UUPS upgradeable pattern. You must deploy a proxy to have a working upgradeable instance. The proxy is the actual application address. *** ## Step 4: Deploy with Timelock (Recommended for Production) ### Timelock Configuration The timelock delay depends on your deployment environment: | Parameter | Testnet | Mainnet | | ------------------ | ------------- | ------------------------------------------- | | **Timelock delay** | 1-4 hours | 24-72 hours | | **Safe signers** | 2-3 addresses | 5-7 addresses (from multiple organizations) | | **Safe threshold** | 2/3 | ≥ 2/3 | | **Executor role** | Open or EOA | Safe only (strict control) | | **Admin role** | EOA or 0x0 | 0x0 (self-managed by timelock) | ### 4.1 Prepare Timelock Configuration ```bash theme={"system"} # Set environment variables export MIN_DELAY=86400 # 1 day in seconds (production: 2-3 days) export PROPOSER=0x... # Address that can schedule operations (your Safe or EOA) export EXECUTOR=0x... # Address that can execute operations (Safe, or 0x0 for open execution) export ADMIN=0x... # Admin address (can grant/revoke roles) ``` ### 4.2 Deploy All Contracts ```bash theme={"system"} # Get deployer address DEPLOYER=$(cast wallet address --private-key $PRIVATE_KEY) # 1. Deploy DstackKms implementation KMS_IMPL=$(forge create src/DstackKms.sol:DstackKms \ --broadcast \ --rpc-url $RPC_URL \ --private-key $PRIVATE_KEY | grep "Deployed to:" | tail -1 | cut -d' ' -f3) # 2. Encode initializer (owner = deployer initially) INIT_DATA=$(cast calldata "initialize(address,address)" $DEPLOYER $ZERO_ADDRESS) # 3. Deploy ERC1967Proxy KMS_PROXY=$(forge create lib/openzeppelin-contracts/contracts/proxy/ERC1967/ERC1967Proxy.sol:ERC1967Proxy \ --broadcast \ --rpc-url $RPC_URL \ --private-key $PRIVATE_KEY \ --constructor-args $KMS_IMPL $INIT_DATA | grep "Deployed to:" | tail -1 | cut -d' ' -f3) # 4. Deploy TimelockController TIMELOCK=$(forge create lib/openzeppelin-contracts/contracts/governance/TimelockController.sol:TimelockController \ --broadcast \ --rpc-url $RPC_URL \ --private-key $PRIVATE_KEY \ --constructor-args $MIN_DELAY "[$PROPOSER]" "[$EXECUTOR]" $ADMIN | grep "Deployed to:" | tail -1 | cut -d' ' -f3) # 5. Transfer ownership to Timelock cast send $KMS_PROXY "transferOwnership(address)" $TIMELOCK \ --rpc-url $RPC_URL \ --private-key $PRIVATE_KEY # 6. Verify ownership OWNER=$(cast call $KMS_PROXY "owner()(address)" --rpc-url $RPC_URL) echo "=== Deployment Result ===" echo "DstackKms implementation: $KMS_IMPL" echo "DstackKms proxy: $KMS_PROXY" echo "TimelockController: $TIMELOCK" echo "DstackKms owner: $OWNER" ``` ### 4.3 Understanding TimelockController Roles | Role | Description | Who Should Have It | | ------------ | ---------------------------------- | ------------------------------------------------------------- | | **Proposer** | Can schedule operations | Safe multisig or trusted EOA | | **Executor** | Can execute operations after delay | Safe, or `address(0)` for open execution | | **Admin** | Can grant/revoke roles | Should be `address(0)` after setup (self-managed by timelock) | *** ## Step 5: Configure Safe (Optional but Recommended) For production, use a Safe multisig as the proposer/executor: ### 5.1 Create a Safe 1. Go to [Safe web app](https://app.safe.global) 2. Connect your wallet 3. Create a new Safe on your target network 4. Add signers (3-7 addresses recommended) 5. Set threshold (≥ 2/3 of signers) ### 5.2 Use Safe Address in Deployment When deploying the TimelockController, use your Safe address: ```bash theme={"system"} export PROPOSER= export EXECUTOR= export ADMIN=0x0000000000000000000000000000000000000000 # Let timelock manage itself ``` ### 5.3 Governance Flow with Safe + Timelock 1. **Draft transaction** — Use Safe web interface to create a transaction 2. **Collect signatures** — Required signers approve 3. **Schedule in timelock** — Safe calls `timelock.schedule()` 4. **Wait for delay** — Wait the configured delay period 5. **Execute** — Anyone (or only executor) calls `timelock.execute()` *** ## Step 6: Verify Deployment Verify on block explorer: ```bash theme={"system"} # Verify implementation forge verify-contract $KMS_IMPL src/DstackKms.sol:DstackKms \ --chain base-sepolia \ --verifier etherscan # Verify proxy forge verify-contract $KMS_PROXY lib/openzeppelin-contracts/contracts/proxy/ERC1967/ERC1967Proxy.sol:ERC1967Proxy \ --chain base-sepolia \ --verifier etherscan \ --constructor-args $(cast abi-encode "constructor(address,bytes)" $KMS_IMPL $INIT_DATA) ``` Check on block explorer: * `DstackKms` owner is set to the TimelockController address * TimelockController has correct proposer/executor roles *** ## How to Execute Governance Actions (With Timelock) Once deployed, all `onlyOwner` operations must go through the timelock: ### Schedule an Operation ```bash theme={"system"} # Example: Add an authorized measurement cast send $TIMELOCK "schedule(address,uint256,bytes,bytes32,bytes32,uint256)" \ $KMS_PROXY \ 0 \ $(cast calldata "addKmsAggregatedMr(bytes32)" $YOUR_MEASUREMENT) \ 0x0000000000000000000000000000000000000000000000000000000000000000 \ $(cast keccak "unique-operation-id") \ $MIN_DELAY \ --rpc-url $RPC_URL \ --private-key $PROPOSER_KEY ``` ### Execute After Delay ```bash theme={"system"} # Wait for MIN_DELAY to pass, then: cast send $TIMELOCK "execute(address,uint256,bytes,bytes32,bytes32)" \ $KMS_PROXY \ 0 \ $(cast calldata "addKmsAggregatedMr(bytes32)" $YOUR_MEASUREMENT) \ 0x0000000000000000000000000000000000000000000000000000000000000000 \ $(cast keccak "unique-operation-id") \ --rpc-url $RPC_URL \ --private-key $EXECUTOR_KEY ``` *** ## All Governance-Protected Methods Once ownership is transferred to timelock, these methods require timelock governance: | Method | Purpose | | ----------------------- | ------------------------------------------ | | `setKmsInfo` | Update KMS public key and attestation info | | `setKmsQuote` | Update KMS quote | | `setKmsEventlog` | Update KMS event log | | `setGatewayAppId` | Set gateway application ID | | `setAppImplementation` | Update app implementation address | | `addKmsAggregatedMr` | Authorize a KMS measurement | | `removeKmsAggregatedMr` | Revoke a KMS measurement | | `addKmsDevice` | Authorize a KMS device | | `removeKmsDevice` | Revoke a KMS device | | `addOsImageHash` | Authorize an OS image hash | | `removeOsImageHash` | Revoke an OS image hash | *** ## Common Issues | Issue | Solution | | --------------------------------------- | ------------------------------------------------------------------------ | | "Insufficient funds" | Get testnet ETH from faucet, or ensure mainnet wallet has enough ETH | | "Ownable: caller is not the owner" | Ownership already transferred to timelock. Use timelock.schedule/execute | | "Timelock: operation is not ready" | Wait for the delay period to pass before executing | | "Timelock: operation already scheduled" | Use a different salt (unique operation ID) | | Proxy verification fails | Use `cast abi-encode` to construct the constructor arguments | *** ## Next Steps * **[Register KMS Measurements](run-kms-on-gcp)** — Register KMS measurements before bootstrap * **[Register Workload Measurements](register-enclave-measurement)** — Authorize workloads to receive keys * **[Manage Governance](manage-governance)** — How to create proposals and execute governance actions * **[Concept: Governance](/dstack-cloud/governance)** — Understand the governance model # E2E Test Report Source: https://docs.phala.com/dstack-cloud/e2e-test-report End-to-end test report covering test coverage and pre-production validation results. # E2E Test Report **This document is a work in progress.** Test results will be populated as they become available. ## Test Coverage Tests are being conducted across the following scenarios. Results will be added as each test completes. | Category | Status | Notes | | -------------------------------------------------------- | ----------- | ----- | | GCP CVM deployment | In progress | | | GCP KMS deployment + bootstrap | In progress | | | Nitro Enclave deployment | In progress | | | RA-TLS connection (GCP KMS → GCP workload) | In progress | | | RA-TLS connection (GCP KMS → Nitro workload) | In progress | | | getKey() key retrieval | In progress | | | On-chain measurement registration | In progress | | | On-chain measurement revocation | In progress | | | Governance flow (propose → approve → timelock → execute) | In progress | | | KMS bootstrap (Onboard → Finish) | In progress | | | Multi-node key replication | Planned | | ## Test Environment | Component | Version | | ------------ | ---------------------- | | dstack-cloud | 0.6.0-test | | dstack OS | dstack-cloud-0.6.0 | | GCP zone | us-central1-a | | AWS region | us-east-1 | | Blockchain | Base Sepolia (testnet) | ## Known Limitations | Limitation | Impact | Mitigation | | ---------------------------- | --------------------------------------------------- | ------------------------------------------------------ | | KMS root key is single-point | If the KMS TEE is compromised, all keys are at risk | MPC distribution planned for future version | | Nitro Enclaves are stateless | Workloads lose state on restart | Use GCP for persistent workloads; KMS runs on GCP only | | Public RPC reliability | Testnet RPCs may return stale state | Use multiple RPC providers | ## Accepted Risks | Risk | Acceptance Criteria | | ------------------------------ | ---------------------------------------------------------- | | TEE hardware side-channels | Accept residual risk. Monitor Intel/AWS advisories. | | Smart contract vulnerabilities | Mitigate with audits. Re-audit after each contract change. | # Quick Start: Deploy Your First dstack App on GCP Source: https://docs.phala.com/dstack-cloud/get-started Quick start tutorial to deploy your first dstack CVM on GCP in 15-25 minutes. # Quick Start: Deploy Your First dstack App on GCP You'll have a Docker application running inside a hardware-encrypted Confidential VM on GCP within 25 minutes. This tutorial covers the full path: install the CLI, configure your project, deploy, and verify that attestation proves your workload runs in genuine TEE hardware. **Estimated time:** 15–25 minutes (first run). **What you will do:** 1. Install `dstack-cloud` CLI 2. Configure global GCP/KMS settings 3. Create a project and define workload 4. Deploy to GCP TDX CVM 5. Verify workload access and runtime status *** ## Prerequisites Before you begin: * GCP project with Intel TDX quota in target zone (for example `us-central1-a`) * `gcloud` authenticated ```bash theme={"system"} gcloud auth login gcloud config set project YOUR_PROJECT_ID ``` * **Linux host** (required — `dstack-cloud deploy` uses FAT32 disk images, which don't work on macOS) * Docker installed * `gsutil`, `jq`, `mtools` (for `mcopy`), `dosfstools` (for `mkfs.fat`) — these are needed by the deploy process to build a shared disk image *** ## Step 1: Install dstack-cloud CLI ```bash theme={"system"} curl -fsSL -o ~/.local/bin/dstack-cloud \ https://raw.githubusercontent.com/Phala-Network/meta-dstack-cloud/main/scripts/bin/dstack-cloud chmod +x ~/.local/bin/dstack-cloud dstack-cloud --help ``` *** ## Step 2: Configure global settings ```bash theme={"system"} dstack-cloud config-edit ``` Use JSON config (`~/.config/dstack-cloud/config.json`): ```json theme={"system"} { "services": { "kms_urls": ["https://kms.tdxlab.dstack.org:12001"], "gateway_urls": ["https://gateway.tdxlab.dstack.org:12002"], "pccs_url": "" }, "image_search_paths": ["/path/to/images"], "gcp": { "project": "YOUR_PROJECT_ID", "zone": "us-central1-a", "bucket": "gs://YOUR_BUCKET" } } ``` If bucket does not exist: ```bash theme={"system"} gcloud storage buckets create gs://YOUR_BUCKET --project YOUR_PROJECT_ID --location us-central1 ``` ### Optional: configure external KMS If you already deployed your own KMS, replace `services.kms_urls`: ```json theme={"system"} "services": { "kms_urls": ["https://YOUR_KMS_IP_OR_DOMAIN:12001"] } ``` *** ## Step 3: Pull OS image ```bash theme={"system"} dstack-cloud pull https://github.com/Phala-Network/meta-dstack-cloud/releases/download/v0.6.0-test/dstack-cloud-0.6.0.tar.gz dstack-cloud pull https://github.com/Phala-Network/meta-dstack-cloud/releases/download/v0.6.0-test/dstack-cloud-0.6.0-uki.tar.gz ``` Verify: ```bash theme={"system"} ls -lh /path/to/images/dstack-cloud-0.6.0/disk.raw ``` *** ## Step 4: Create project ```bash theme={"system"} dstack-cloud new my-first-app --os-image dstack-cloud-0.6.0 --instance-name dstack-first-app cd my-first-app ``` Project files include: ``` my-first-app/ ├── app.json # Application metadata ├── docker-compose.yaml # Your container definition ├── .env # Environment variables (encrypted) └── prelaunch.sh # Optional pre-launch script (e.g., setup, data download) ``` *** ## Step 5: Configure app Edit `app.json` and set: * `gcp_config.project = "YOUR_PROJECT_ID"` * `gcp_config.zone = "us-central1-a"` * `gcp_config.bucket = "gs://YOUR_BUCKET"` Default key mode is `kms`. If you want no external KMS for a basic quick test, switch to: * `"key_provider": "tpm"` * `"gateway_enabled": false` * remove `.env` file and remove `env_file` field from `app.json` *** ## Step 6: Define workload Edit `docker-compose.yaml`: ```yaml theme={"system"} services: web: image: nginx:latest ports: - "8080:80" ``` *** ## Step 7: Deploy ```bash theme={"system"} dstack-cloud deploy --delete ``` This will create a TDX CVM and start your workload. *** ## Step 8: Open firewall ```bash theme={"system"} dstack-cloud fw allow 8080 ``` *** ## Step 9: Verify Check that the CVM is running and your workload is accessible: ```bash theme={"system"} dstack-cloud status # Expected: shows "RUNNING" with measurements (RTMR values) dstack-cloud logs --follow # Get attestation from your app curl https://app-abc123.your-gateway-domain.com/attestation # Verify using dstack-verifier dstack-verifier verify ``` The attestation proves: * The workload runs in genuine Intel TDX hardware * The exact code and measurements match expectations * The boot chain integrity is verified via TDX + vTPM For detailed verification, see [Attestation Integration](/dstack-cloud/attestation-integration). **Test workload:** ```bash theme={"system"} curl http://:8080 ``` If gateway is enabled, use the URL shown by `dstack-cloud status`. ## Understanding What Happened When you deployed your application: 1. **Confidential VM Created** — A GCP VM with Intel TDX was provisioned 2. **dstack OS Booted** — A minimal, attested guest OS started inside the TEE 3. **Automatic Disk Encryption** — All disk I/O is encrypted with keys managed by the Guest Agent 4. **TEE Attestation** — The Guest Agent provides attestation proof via the TDX + vTPM mechanism 5. **TLS Certificate** — Gateway automatically provisions ACME certificates for your domain ### Key Delivery via KMS dstack uses an external **Key Management Service (dstack-kms)** to deliver keys to your confidential workloads. The KMS runs in its own TEE and only dispatches keys to workloads that pass attestation verification. *** ## Managing Your Deployment Your application now runs in a hardware-protected environment where even the cloud provider cannot access the memory or data. *** ## Troubleshooting | Issue | Fix | | --------------------------------------- | ----------------------------------------------------- | | `Boot image ... not found` | verify image path and `disk.raw` existence | | VM UEFI boot loop | use valid UKI boot image (`-uki.tar.gz`) | | `.env found but KMS is not enabled` | remove `.env` and remove `env_file` in `app.json` | | Port not reachable | ensure firewall rule exists and container has started | | missing `gsutil` / `mcopy` / `mkfs.fat` | install required dependencies | *** ## Next steps * Detailed guide: [Run a Workload on GCP with Self-hosted KMS](/dstack-cloud/run-on-gcp) * External KMS flow: [Run a dstack-kms CVM on GCP](/dstack-cloud/run-kms-on-gcp) * Concepts: [Attestation Integration](/dstack-cloud/attestation-integration) # Glossary Source: https://docs.phala.com/dstack-cloud/glossary Glossary of core terminology used in dstack-cloud documentation. # Glossary Core terminology used in this documentation. *** ## Infrastructure | Term | Definition | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **TEE (Trusted Execution Environment)** | A hardware-isolated compute environment that protects code and data from the rest of the system. Memory is encrypted, and the hardware proves the environment's integrity via attestation. | | **CVM (Confidential Virtual Machine)** | A virtual machine running in a TEE. In dstack-cloud, a CVM runs dstack-os with your Docker containers inside. | | **Intel TDX (Trust Domain Extensions)** | Intel's TEE technology that provides memory encryption and isolation at the VM level. Supported on GCP Confidential VMs and certain bare-metal servers. | | **SGX (Software Guard Extensions)** | Intel's earlier TEE technology. Provides memory encryption at the enclave (process) level. | | **AWS Nitro Enclaves** | AWS's TEE technology. Provides process-level isolation within an EC2 instance. Memory is encrypted and inaccessible to the host. | | **NSM (Nitro Secure Module)** | The hardware component in AWS Nitro that generates attestation documents and manages Enclave lifecycle. | | **VSOCK** | A socket interface for communication between a Nitro Enclave and its host EC2 instance. The Enclave cannot directly access the network — all traffic goes through VSOCK. | | **EIF (Enclave Image File)** | The image format used by AWS Nitro Enclaves. Built from a Docker image, it contains the OS and application code that runs inside the Enclave. | | **TPM (Trusted Platform Module)** | A hardware security module that provides secure key storage and measurement. Used on GCP Confidential VMs as the root of trust. | | **PCCS (Provisioning Certificate Caching Service)** | Intel's service for caching attestation certificate chains, used in DCAP (Data Center Attestation Primitives) workflows. | *** ## Security Mechanisms | Term | Definition | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Remote Attestation** | The process by which a TEE proves its identity and integrity to an external party. The TEE generates a cryptographic proof (signed by hardware) containing measurements of the running code. | | **RA-TLS (Remote Attestation TLS)** | An extension of TLS where both parties verify each other's TEE attestation during the handshake. Used for secure communication between workloads and KMS. | | **ZT-TLS (Zero Trust TLS)** | A TLS variant that requires TEE attestation for every connection, regardless of network location. | | **Measurement** | A cryptographic hash of the code and configuration running inside a TEE. Any change to the code produces a different measurement. | | **PCR (Platform Configuration Register)** | Measurement registers used by AWS Nitro Enclaves (PCR0-3). | | **RTMR (Runtime Measurement Register)** | Measurement registers used by Intel TDX (RTMR0-3). | | **OS\_IMAGE\_HASH** | A composite hash derived from all PCR values, used as the unique identifier for a Nitro Enclave image on-chain. | | **compose-hash** | The SHA256 hash of the `docker-compose.yaml` file, stored in RTMR3 on GCP TDX deployments. | | **Quote (TDX Quote)** | A hardware-signed attestation proof generated by Intel TDX. Contains measurement values (RTMR0-3) and an Intel hardware signature. | | **Attestation Document** | The attestation proof generated by AWS NSM. Contains PCR values and an NSM signature. | *** ## dstack Components | Term | Definition | | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **dstack** | The open-source confidential computing framework. Provides a CLI, KMS, attestation, and SDK libraries (Python, TypeScript, Rust, Go) for building confidential applications. [GitHub](https://github.com/Dstack-TEE/dstack) | | **dstack-sdk** | Client libraries (Python, TypeScript, Rust, Go) that applications use to interact with dstack features such as attestation, key retrieval, and storage encryption. Part of the dstack framework. | | **dstack-cloud** | An extension of dstack that enables deployment on GCP Confidential VMs and AWS Nitro Enclaves. Provides the `dstack-cloud` CLI. [GitHub](https://github.com/Phala-Network/dstack-cloud) | | **dstack-os** | The minimal operating system that runs inside each dstack CVM. Built from a reproducible Docker image. | | **Guest Agent** | A process that runs inside each dstack CVM alongside your application containers. Handles attestation, key retrieval from KMS, and storage encryption. Exposes a local API at `/var/run/dstack.sock`. | | **KMS (Key Management Service)** | A standalone service that runs in its own TEE. Verifies workload attestation and dispatches keys to authorized workloads. Also known as DeRoT (Decentralized Root-of-Trust). | | **Gateway** | An edge component that handles TLS termination, automatic ACME certificate provisioning, and RA-TLS support for incoming connections. | | **VMM (Virtual Machine Monitor)** | The component that parses Docker Compose files and boots CVMs from reproducible OS images. | *** ## On-chain Governance | Term | Definition | | ------------------- | --------------------------------------------------------------------------------------------------------------------- | | **DstackKms** | The on-chain KMS policy contract. Stores authorized workload measurements and admin roles. | | **DstackApp** | The on-chain application entry contract. Holds a reference to DstackKms. | | **Multisig (Safe)** | A multi-signature wallet that controls governance actions. Requires multiple parties to approve any transaction. | | **Timelock** | A delay mechanism that enforces a mandatory waiting period between approval and execution of governance transactions. | | **GovernanceSafe** | The specific Safe wallet instance used for dstack governance. | *** ## Cryptography | Term | Definition | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **MPC (Multi-Party Computation)** | A cryptographic technique where multiple parties jointly compute a result without any single party learning the inputs. Planned for future KMS root key generation. | | **KDF (Key Derivation Function)** | A function that derives application-specific keys from a root key. Each workload gets a unique key derived this way. | | **SealingKey** | A key used to encrypt data at rest (disk storage) inside the CVM. Derived from the KMS-delivered key. | | **RootKey** | The top-level key in the KMS hierarchy. All application keys are derived from the RootKey. | # On-chain Governance Model Source: https://docs.phala.com/dstack-cloud/governance The on-chain governance model using Multisig + Timelock to control KMS authorization and prevent covert upgrades. # On-chain Governance Model On-chain governance ensures that key policy changes — like adding or revoking workload measurements — cannot happen silently. Every authorization change goes through a Multisig + Timelock process and is recorded on-chain for anyone to audit. ## Core Smart Contracts ### DstackKms The KMS policy contract. It stores: * **Authorized measurements** — The list of workload measurements (RTMR / OS\_IMAGE\_HASH) that are allowed to receive keys * **Configuration parameters** — KMS-related settings controlled by governance * **Admin roles** — Addresses authorized to perform governance operations The off-chain dstack-kms service queries DstackKms to determine whether a workload is authorized before dispatching keys. ### DstackApp The application-level entry contract. It: * Holds a reference to DstackKms * Enforces application-specific checks before delegating to the KMS * Acts as the on-chain identity for a specific deployment ### GovernanceSafe **Optional.** A multisig wallet (typically [Safe](https://safe.global)) that you can introduce to enhance governance security: * Owns DstackKms and DstackApp * Is the only entity allowed to execute governance operations (registering measurements, changing admin, upgrading contracts) * Requires multiple signers to approve any transaction > **Note:** GovernanceSafe is not part of dstack or dstack-cloud. It is an optional governance mechanism you can add to prevent unilateral control over key policies. ### Timelock **Optional.** A delay module attached to the multisig that: * Enforces a mandatory waiting period between approval and execution * Maintains a queue of pending governance actions * Gives the community time to review and respond to proposed changes > **Note:** Like GovernanceSafe, Timelock is an optional enhancement. You can use it with a multisig to add a delay before governance actions take effect. ## Governance Workflow ### Basic Workflow (without Multisig/Timelock) At minimum, governance operations (registering measurements, updating configuration) are performed by the admin address directly: 1. Admin submits a transaction to DstackKms or DstackApp 2. Transaction executes immediately 3. Changes take effect right away This is suitable for development and testing, but lacks the security guarantees needed for production. ### Enhanced Workflow (with Multisig + Timelock) For production deployments, we recommend introducing a multisig wallet (e.g., Safe) and a timelock. The workflow becomes: Governance Workflow 1. **Draft** — A governance action is proposed (e.g., register new measurement) 2. **Submit to Multisig** — The action is submitted to the multisig wallet 3. **Approve** — Required number of signers approve the transaction 4. **Queue in Timelock** — The approved action is queued and must wait for the delay period 5. **Wait** — The delay period passes (e.g., 48 hours), allowing time for review 6. **Execute** — After the delay, anyone can execute the action This ensures: * No single party can unilaterally modify key policies * All changes are visible and reviewable before taking effect * Stakeholders have time to respond to suspicious proposals ## Role of Multisig + Timelock | Mechanism | Purpose | | ------------ | ------------------------------------------------------------------------------------------------------ | | **Multisig** | Prevents unilateral actions. No single person can modify key policies — requires multi-party approval. | | **Timelock** | Enforces a delay, giving all stakeholders time to detect and respond to suspicious changes. | ### Recommended Configuration | Parameter | Production | Non-production | | -------------------------- | ----------------------------------- | -------------- | | **Number of signers** | 5-7 (from at least 2 organizations) | 2-3 | | **Signature threshold** | ≥ 2/3 (e.g., 4-of-6) | ≥ 2/3 | | **Timelock delay** | 24-72 hours | 1-4 hours | | **Transaction expiration** | 7-14 days | 3-7 days | ## Typical Governance Scenarios ### Register New Measurements (New Version Deploy) When you update your application code or dstack OS version, the measurements change. You must register the new measurements before KMS will dispatch keys: 1. Build the updated CVM image 2. Extract new measurements (RTMR3 / OS\_IMAGE\_HASH) 3. Draft a governance transaction to add the new measurements to DstackKms 4. Submit to multisig for approval 5. Wait for timelock 6. Execute — KMS now authorizes workloads with the new measurements 7. Deploy the updated CVM ### Revoke a Compromised Measurement (Emergency) If a measurement is found to be vulnerable or unauthorized: 1. Draft a governance transaction to remove the measurement from DstackKms 2. Submit to multisig for expedited approval 3. Wait for timelock (cannot be bypassed — this is by design) 4. Execute — KMS immediately stops dispatching keys to workloads with that measurement > **Note:** Even in emergencies, the timelock cannot be skipped. This is intentional — it ensures all changes are visible and reviewable. Plan your security incident response accordingly. ### Upgrade Smart Contracts If the contracts need to be upgraded (e.g., bug fix, new feature): 1. Deploy the new contract implementation 2. Draft a governance transaction to update the proxy's implementation address 3. Submit to multisig for approval 4. Wait for timelock 5. Execute — the proxy now points to the new implementation ## Security Considerations * **Use hardware wallets:** All multisig signers should use hardware wallets (e.g., Ledger, Trezor) to protect their signing keys. * **Set appropriate thresholds:** A threshold of ≥ 2/3 ensures that compromising a minority of signers is insufficient to push through malicious changes. * **Regular health checks:** Periodically verify that all signers are still accessible and the governance configuration is as expected. * **Monitor governance activity:** Set up alerts for governance transactions (proposals, approvals, executions) to detect suspicious activity early. ## Next Steps * **[Deploy On-chain KMS Smart Contracts](/dstack-cloud/deploy-onchain-kms)** — Set up contracts on testnet or mainnet * **[Manage Governance](/dstack-cloud/manage-governance)** — How to create proposals and execute governance actions * **[Security Model](security-model)** — Trust boundaries and security guarantees # KMS and Key Delivery Source: https://docs.phala.com/dstack-cloud/kms-and-key-delivery Understand KMS operating modes, key delivery flow, and GCP vs Nitro differences in dstack-cloud deployments. # KMS and Key Delivery KMS is the component that decides whether a workload is trustworthy enough to receive keys. This page explains the two operating modes (local vs. external KMS), the key delivery flow on each platform, and how on-chain smart contracts add an auditable authorization layer. > For the core design of dstack's Key Management System — including the Decentralized Root-of-Trust (DeRoT) protocol, MPC key generation, key derivation, and rotation — refer to the official Phala documentation: > > * [Key Management Protocol](https://docs.phala.com/dstack/design-documents/key-management-protocol) > * [Decentralized Root-of-Trust](https://docs.phala.com/dstack/design-documents/decentralized-root-of-trust) ## Operating Modes dstack supports two KMS operating modes: ### Local Key Provider Keys are generated and managed locally within each dstack CVM. No separate KMS service is needed. * **How it works:** The Guest Agent generates keys using hardware-rooted entropy (TPM on GCP, NSM on Nitro). Keys are sealed to the TEE and never leave the CVM. * **Use when:** Development, testing, or single-node deployments where key sharing between instances is not required. * **Limitation:** Keys cannot be shared across multiple CVMs. If you need the same key in different workloads, use KMS mode. ### KMS Mode A standalone **dstack-kms** service runs in its own TEE (a dedicated dstack CVM). It verifies workload identity before dispatching keys. * **How it works:** When a dstack application CVM starts, it obtains a hardware attestation and sends it to KMS. KMS verifies the attestation and checks whether the workload is authorized. If verification passes, KMS dispatches the requested key. * **Use when:** Production deployments, multi-node setups, or when multiple CVM instances need access to the same keys. * **Requirement:** A dstack-kms instance must be deployed and running before application's CVM can request keys. ### On-chain Smart Contracts (Security Enhancement) KMS mode can optionally integrate with on-chain smart contracts (`DstackKms`, `DstackApp`) to provide an additional layer of security: * On-chain contracts control which application CVM are authorized to receive keys (via registered measurements) * All authorization changes are visible on-chain — anyone can audit which measurements are approved * Changes follow a Multisig + Timelock process, preventing covert key policy modifications * This is a **security enhancement** to KMS mode, not a separate operating mode For details on governance, see [On-chain Governance Model](governance). ## Key Delivery Flow (KMS Mode) The following describes the high-level key delivery flow: ### GCP (dstack CVM with Guest Agent) 1. **CVM Startup** — A dstack CVM boots and its Guest Agent obtains a TDX Quote from hardware. 2. **Secure Channel** — The CVM establishes a secure connection with KMS using RA-TLS (Remote Attestation TLS). Both sides verify each other's attestation during the TLS handshake. 3. **Attestation Verification** — KMS verifies the TDX Quote: * Confirms the CVM is running in genuine TEE hardware * Extracts the workload measurement (RTMR values) * Checks whether the measurement matches an authorized value 4. **On-chain Check** *(if on-chain contracts are configured)* — KMS queries the blockchain to verify that the workload's measurement is registered and authorized. 5. **Key Dispatch** — If all checks pass, KMS derives the requested key and sends it to the CVM over the encrypted channel. 6. **Key Usage** — The Guest Agent manages the key lifecycle. Your application retrieves the key via the dstack SDK (Python, TypeScript, Rust, or Go) by connecting to the Guest Agent at `/var/run/dstack.sock`. The Guest Agent can also automatically handle disk encryption. ### AWS Nitro (Enclave with dstack-util) 1. **Enclave Startup** — A Nitro Enclave launches from the EIF. `dstack-util` inside the Enclave obtains an NSM Attestation Document. 2. **Secure Channel** — `dstack-util` establishes a connection with KMS through the VSOCK proxy on the host. The connection uses TLS (RA-TLS if available). 3. **Attestation Verification** — KMS verifies the NSM Attestation Document: * Confirms the Enclave is running on genuine Nitro hardware * Extracts the PCR values (measurements) * Checks whether the `OS_IMAGE_HASH` matches an authorized value 4. **On-chain Check** *(if on-chain contracts are configured)* — KMS queries the blockchain to verify that the measurement is registered and authorized. 5. **Key Dispatch** — If all checks pass, KMS derives the requested key and sends it back through the encrypted channel. 6. **Key Usage** — `dstack-util` makes the key available to your application. **You decide how to use the key** — common patterns include encrypting model weights in memory, decrypting configuration secrets, or encrypting a RAM-disk. Unlike GCP, there is no automatic disk encryption on Nitro. > **Key difference:** On GCP, the Guest Agent manages the entire key lifecycle including automatic disk encryption. On Nitro, `dstack-util` only retrieves the key — your application is responsible for using it. ## GCP vs. Nitro: Key Delivery Differences The key delivery flow is similar on both platforms, but the implementation details differ: * **Attestation format:** TDX Quote (GCP) vs. NSM Attestation Document (Nitro) * **Communication with KMS:** Direct network on GCP vs. VSOCK proxy on Nitro * **Key usage:** Guest Agent manages the full key lifecycle on GCP (including automatic disk encryption), while on Nitro `dstack-util` only retrieves the key — your application decides how to use it * **Persistent storage:** Available on GCP, not on Nitro (stateless Enclave) For a complete comparison, see [AWS Nitro Enclave Integration](nitro-enclave). ### VSOCK Proxy (Nitro-specific) On AWS Nitro, the Enclave cannot directly access the external network. A **VSOCK proxy** runs on the host EC2 instance to forward network requests: * `dstack-util` inside the Enclave sends requests to the VSOCK proxy * The proxy forwards them to the destination (KMS, RPC endpoints, etc.) * Responses are routed back through the proxy to the Enclave This means that on Nitro, you must deploy and configure a VSOCK proxy on the host machine before `dstack-util` can communicate with KMS or any external service. ## KMS Bootstrap (First-time Setup) When a dstack-kms CVM starts for the first time, it enters **Onboard mode** (HTTP, unauthenticated). This is a one-time initialization process: 1. Call `Onboard.Bootstrap` to generate the KMS key pair and obtain attestation info. 2. Register the KMS attestation on-chain (if using on-chain contracts). 3. Call `/finish` to complete initialization. KMS restarts and switches to HTTPS mode. After bootstrap, KMS operates in **Normal mode** and only accepts RA-TLS connections from verified workloads. > **Implementation:** The KMS bootstrap logic is implemented in the [dstack repository](https://github.com/Phala-Network/dstack). ## Key Lifecycle The full key lifecycle is documented in the [Key Management Protocol](https://docs.phala.com/dstack/design-documents/key-management-protocol). At a high level: | Stage | Description | | -------------- | ---------------------------------------------------------------------------------------- | | **Generation** | Root keys are generated within the KMS TEE using hardware-rooted entropy | | **Derivation** | Application-specific keys are derived from root keys using KDF (Key Derivation Function) | | **Delivery** | Keys are delivered to verified CVMs over RA-TLS encrypted channels | | **Rotation** | Keys can be rotated by updating the derivation inputs; old keys are naturally phased out | | **Revocation** | Workloads can be de-authorized by revoking their on-chain measurement registration | ## Next Steps * **[Run a dstack-kms CVM on GCP](/dstack-cloud/run-kms-on-gcp)** — Deploy KMS on GCP * **[Deploy On-chain KMS Smart Contracts](/dstack-cloud/deploy-onchain-kms)** — Set up on-chain authorization # Governance Operations Source: https://docs.phala.com/dstack-cloud/manage-governance Create governance proposals, manage Multisig signing, and execute Timelock transactions. # Governance Operations Every governance action — registering measurements, revoking access, upgrading contracts — follows the same pattern: propose, collect signatures, wait for timelock, execute. This page walks through that pattern step by step. ## Prerequisites * Access to a signer wallet in the GovernanceSafe * The Safe web interface ([https://app.safe.global](https://app.safe.global)) or Safe CLI * Understanding of the governance model — see [On-chain Governance Model](/dstack-cloud/governance) ## Creating a Governance Proposal All governance changes start as a transaction submitted to the GovernanceSafe. ### Via Safe Web Interface 1. Go to [https://app.safe.global](https://app.safe.global) and connect your signer wallet 2. Select your governance Safe 3. Click **"New Transaction"** 4. Choose the transaction type: * **Contract interaction** — Call a function on DstackKms, DstackApp, or other contracts * **Transfer** — Send funds (if needed for gas or contract operations) * **Raw transaction** — For complex multi-call transactions ### Common Governance Actions | Action | Contract | Function | | ------------------------------- | --------- | ---------------------------- | | Register new measurement | DstackKms | `addOsImageHash(bytes32)` | | Revoke measurement | DstackKms | `removeOsImageHash(bytes32)` | | Update admin role | DstackKms | `setAdmin(address)` | | Update KMS reference | DstackApp | `setKms(address)` | | Upgrade contract implementation | Proxy | `upgradeTo(address)` | ### Example: Register a New Measurement 1. **New Transaction** → **Contract interaction** 2. Select `DstackKms` contract 3. Select `addMeasurement(bytes32)` function 4. Enter the measurement hash: `0x1234abcd...` 5. Click **"Create"** 6. The transaction enters the Safe queue ## Multisig Signing and Approval After a transaction is created, it must be approved by enough signers to meet the threshold. ### Approving a Transaction 1. Each signer connects to the Safe web interface 2. Opens the pending transaction 3. Reviews: * Target contract address * Function name and parameters * Simulation result (if available) 4. Clicks **"Confirm"** (signs with their wallet) ### Tracking Approval Progress The Safe interface shows: * Number of confirmations collected * Required threshold (e.g., "3 of 5 confirmations") * Status: Pending / Awaiting execution / Executed ### Rejecting a Transaction If a signer disagrees with a proposal, they should: 1. Not sign the transaction 2. Communicate concerns to other signers 3. If the transaction has already been approved, wait for the timelock — there is no "cancel" after execution ## Timelock After the required number of signatures is collected, the transaction enters the Timelock queue. ### How It Works 1. The transaction is queued with a mandatory delay 2. No one can execute the transaction until the delay expires 3. During the delay, anyone can review the transaction on-chain 4. After the delay, any authorized address (typically a signer) can execute it ### Timelock Duration | Environment | Typical Duration | | ----------- | ---------------- | | Testnet | 1-4 hours | | Staging | 4-24 hours | | Production | 24-72 hours | ### Monitoring the Timelock * **Safe web interface:** Shows the queue position and remaining time * **Block explorer:** Look for the Timelock contract's `QueuedTransactions` event ## Execution After the timelock expires: 1. Open the Safe web interface 2. Find the transaction (status: "Ready to execute") 3. Click **"Execute"** 4. Confirm the transaction in your wallet 5. The transaction is submitted on-chain After execution, the change takes effect immediately. KMS syncs the latest on-chain state on its next query. ## Emergency Operations ### Revoke a Compromised Measurement If you discover that a measurement is compromised: 1. Draft a transaction to call `DstackKms.removeOsImageHash(bytes32)` with the compromised hash 2. Submit to the Safe for priority approval 3. Collect signatures as quickly as possible 4. **Wait for the timelock** — this cannot be bypassed 5. Execute after the delay > **Important:** The timelock cannot be skipped, even in emergencies. This is by design — it prevents covert changes. Plan your incident response to account for the delay. ### Replace a Signer If a signer key is compromised or a signer needs to be replaced: 1. Draft a transaction to call `Safe.addOwner(address)` and `Safe.removeOwner(address, address)` 2. Submit for multisig approval 3. Wait for timelock 4. Execute The replacement signer should use a hardware wallet. ### Governance Health Check Periodically verify the health of your governance setup: | Check | How | | -------------------------------- | ------------------------------------------------------------- | | All signers are reachable | Contact each signer; confirm they have access to their wallet | | Signer keys are secure | Verify signers are using hardware wallets | | Threshold is appropriate | Review the threshold against the number of active signers | | Timelock duration is appropriate | Adjust if the threat landscape has changed | | No stale transactions in queue | Review and cancel any outdated proposals | | Contract ownership is correct | Verify DstackKms and DstackApp owners point to the Safe | ## Common Issues | Issue | Solution | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Transaction not appearing in queue | Ensure the transaction was submitted correctly. Check the nonce. Verify gas is sufficient. | | Signer cannot confirm | Ensure the signer is connected to the correct network. Check that the wallet has ETH for gas. | | Timelock delay seems too long | Adjust the Timelock parameters through a governance action (which itself goes through the current timelock). | | Transaction execution reverts | Check the contract state. The transaction may depend on state that changed during the timelock period. Simulate the transaction before executing. | | Lost signer key | Immediately add a new signer and remove the lost one through governance. Use the remaining signers to approve. | ## Next Steps * **[Register Workload Measurements](register-enclave-measurement)** — Most common governance action * **[Deploy On-chain KMS Smart Contracts](deploy-onchain-kms)** — Initial contract setup * **[Concept: Governance](/dstack-cloud/governance)** — Understand the governance model # Monitoring and Alerting Source: https://docs.phala.com/dstack-cloud/monitoring-alerting Set up monitoring, alerting, and dashboards for dstack-cloud deployments. # Monitoring and Alerting You can't secure what you can't see. This page covers the key metrics to watch, log collection patterns, and alert rules that help you catch attestation failures, governance anomalies, or KMS downtime before they become incidents. ## Key Metrics ### KMS Metrics These metrics tell you whether KMS is healthy and delivering keys promptly: | Metric | Description | Alert Threshold | | ----------------------------------------- | -------------------------------------------------------- | --------------- | | **Key request success rate** | Percentage of key requests that succeed | \< 99% | | **Key dispatch latency (p50/p99)** | Time from request to key delivery | p99 > 5s | | **Attestation verification success rate** | Percentage of attestation verifications that pass | \< 99% | | **On-chain sync lag** | Time between on-chain state change and KMS picking it up | > 60s | | **KMS uptime** | Percentage of time KMS is reachable | \< 99.9% | ### CVM / Enclave Metrics Track workload health and resource usage: | Metric | Description | Alert Threshold | | --------------------------------- | ------------------------------------------- | --------------- | | **CVM uptime** | Percentage of time the CVM is running | \< 99% | | **CVM boot time** | Time from deploy command to CVM ready | > 5 minutes | | **RA-TLS handshake success rate** | Percentage of successful RA-TLS connections | \< 99% | | **Container restart count** | Number of container restarts within the CVM | > 0 | | **Memory usage** | CVM memory utilization | > 90% | ### Governance Metrics Governance metrics help you catch stalled proposals or signer inactivity: | Metric | Description | Alert Threshold | | ----------------------------- | ------------------------------------------------- | --------------- | | **Pending proposals** | Number of governance proposals awaiting execution | > 0 for > 24h | | **Signer participation rate** | Percentage of signers active in last 7 days | \< 80% | | **Timelock queue depth** | Number of transactions in the timelock queue | > 3 | ## Log Collection ### dstack-cloud Logs Use the built-in log viewer: ```bash theme={"system"} # View recent logs dstack-cloud logs # Follow logs in real-time dstack-cloud logs --follow # View logs for a specific container dstack-cloud logs --container ``` ### KMS Logs KMS logs are available through `dstack-cloud logs` when the KMS is deployed as a dstack CVM. Key log patterns to monitor: | Log Pattern | Meaning | | --------------------------------- | ----------------------------------------------------- | | `attestation verification failed` | A workload's attestation was rejected | | `measurement not authorized` | The workload's measurement is not registered on-chain | | `key dispatched` | A key was successfully delivered to a workload | | `on-chain sync completed` | KMS synced its on-chain state | | `RA-TLS handshake failed` | TLS connection with attestation failed | ### Infrastructure Logs **GCP:** * Cloud Logging: `gcloud logging read "resource.type=gce_instance AND labels.instance_id="` * Serial port output for boot diagnostics **AWS Nitro:** * EC2 instance system logs * VSOCK proxy logs (if running as a systemd service) ## Dashboard Configuration ### Recommended Dashboard Panels 1. **KMS Health** * Request rate (requests/min) * Success rate (%) * Latency histogram (p50, p95, p99) * Active connections 2. **CVM Health** * Number of running CVMs * Per-CVM status (running/stopped/error) * Boot time trend 3. **Governance** * Pending proposals count * Recent governance transactions * Signer activity heatmap ### Integration with Datadog To integrate with Datadog, you'll need a Datadog agent on each host machine that collects custom metrics from dstack-cloud. The key metric namespaces are `dstack.cloud.cvm.*`, `dstack.cloud.kms.*`, and `dstack.cloud.governance.*`. 1. Set up a Datadog agent on each host machine 2. Configure custom metrics collection for KMS, CVM, and governance events 3. Create dashboards and monitors based on the metrics above ## Alert Rules ### Critical Alerts (Page immediately) | Alert | Condition | Response | | -------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------ | | **KMS down** | KMS unreachable for > 2 minutes | Check CVM/Enclave status. Restart if needed. | | **Attestation failure spike** | > 10 attestation failures in 5 minutes | May indicate a compromised or misconfigured workload. Investigate immediately. | | **Governance: suspicious transaction** | New proposal to revoke critical measurements | Review the proposal. Alert all signers. | ### Warning Alerts (Page during business hours) | Alert | Condition | Response | | ------------------------------- | --------------------------------- | ---------------------------------------------------- | | **High key dispatch latency** | p99 > 5s for > 10 minutes | Check KMS load. Scale if needed. | | **CVM restart loop** | CVM restarted > 3 times in 1 hour | Investigate container health. Check resource limits. | | **On-chain sync lag** | KMS on-chain state > 60s behind | Check RPC provider health. Switch to backup RPC. | | **VSOCK proxy failure** (Nitro) | VSOCK proxy process not running | Restart the proxy. Check host health. | ### Info Alerts (Log only) | Alert | Condition | | --------------------------- | -------------------------------------- | | **New governance proposal** | Any new proposal submitted to the Safe | | **CVM deployed** | New CVM deployment completed | | **Measurement registered** | New measurement added on-chain | ## Escalation Policies | Severity | Escalation | Response Time | | ------------ | --------------------------------------- | ----------------- | | **Critical** | Page on-call SRE + notify security team | 15 minutes | | **Warning** | Notify platform team via Slack/Teams | 2 hours | | **Info** | Log to incident channel | Next business day | Use [Incident.io](https://incident.io) or similar tools to manage incident lifecycle. ## On-chain Monitoring Monitor the blockchain for governance activity: * **Safe Transaction Service:** Subscribe to the Safe's transaction feed for real-time notifications * **The Graph / Dune Analytics:** Query governance transaction history for reporting * **Block explorer alerts:** Set up watch-only notifications for DstackKms and DstackApp contract events ## Next Steps * **[Runbook](runbook)** — Step-by-step troubleshooting for common incidents * **[Upgrade Procedures](upgrade)** — How to upgrade CVMs, KMS, and contracts # AWS Nitro Enclave Integration Source: https://docs.phala.com/dstack-cloud/nitro-enclave How AWS Nitro Enclaves work with dstack-cloud, including EIF, VSOCK proxy, and key architectural differences from GCP. ## What Are Nitro Enclaves AWS Nitro Enclaves provide process-level isolation on EC2: the host OS cannot read or write enclave memory, even with root access. dstack-cloud packages your application into an Enclave Image File (EIF) and uses `dstack-util` for attestation and key retrieval — replacing the Guest Agent used on GCP. Key properties: * **Memory-isolated:** The host OS cannot read or write Enclave memory * **No persistent storage:** Every Enclave launch starts from a clean state * **No external network access:** Enclaves communicate through VSOCK (a socket interface to the host machine) * **Cryptographic attestation:** AWS Nitro Secure Module (NSM) generates an Attestation Document proving the Enclave's identity ## How dstack Uses Nitro Enclaves The Nitro integration differs significantly from GCP. On GCP, dstack-cloud deploys a full virtual machine (dstack CVM) with a Guest Agent. On AWS Nitro, the workload runs inside an AWS Enclave OS image — there is no Guest Agent and no dstack OS. Nitro Enclave Architecture On AWS Nitro, the flow is: 1. You define your application in a `Dockerfile` 2. AWS Nitro CLI (`nitro-cli build-enclave`) converts the Dockerfile into an **Enclave Image File (EIF)** — the Enclave OS image 3. dstack packages **`dstack-util`** into the EIF. This tool handles attestation and key retrieval from KMS 4. The Enclave runs your application code inside the AWS-provided Enclave OS — not dstack OS 5. Inside the Enclave, `dstack-util` communicates with KMS through the VSOCK proxy on the host to obtain keys 6. Your application decides how to use the key (e.g., encrypting disk in-memory, decrypting model weights, etc.) ### Key Differences from GCP | Aspect | GCP (dstack CVM) | AWS Nitro (Enclave) | | ---------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | **OS inside TEE** | dstack OS (custom Linux with Guest Agent) | Enclave OS (provided by AWS) | | **Guest Agent** | Yes — handles attestation, key management, storage encryption | **No** — replaced by `dstack-util` | | **Disk encryption** | Automatic (Guest Agent handles it) | **User-managed** — `dstack-util` provides the key, your application decides what to encrypt | | **Image build** | dstack-cloud builds a CVM disk image | AWS Nitro CLI builds EIF from a Dockerfile | | **Application format** | `docker-compose.yaml` | `Dockerfile` (single container) | ## Enclave Image (EIF) Build Process A Nitro Enclave is launched from an **Enclave Image File (EIF)**. The build process has two stages: ### Stage 1: Build Docker Image First, build a Docker image from your Dockerfile: ```bash theme={"system"} docker build -t my-app -f Dockerfile . ``` Your Dockerfile should include your application and `dstack-util`. ### Stage 2: Build EIF from Docker Image Then, convert the Docker image into an EIF using the Nitro CLI: ```bash theme={"system"} nitro-cli build-enclave --docker-uri my-app:latest \ --output-file my-app.eif ``` This command: * Uses the AWS-provided Enclave OS as the base * Packages your Docker image content into an EIF * Generates 3 PCR values (PCR0, PCR1, PCR2) that uniquely identify the image ### Stage 3: Register OS\_IMAGE\_HASH The 3 PCR values are combined into a single `OS_IMAGE_HASH` for on-chain authorization: ```bash theme={"system"} # Example: combine PCRs into OS_IMAGE_HASH OS_IMAGE_HASH=$(sha256sum <<< "${PCR0}${PCR1}${PCR2}" | cut -d' ' -f1) ``` The `OS_IMAGE_HASH` must be registered via [governance](governance) before KMS will deliver keys to this Enclave. > **Note:** Any change to the Dockerfile, application code, or `dstack-util` version produces different PCR values and a different `OS_IMAGE_HASH`. ### Automated Deployment In practice, these steps are automated by deployment scripts. See [Run a Workload on AWS Nitro](/dstack-cloud/run-on-nitro) for the full workflow using the `deploy_host.sh` and `get_keys.sh` scripts. ## Key Retrieval with dstack-util On Nitro, `dstack-util` replaces the Guest Agent for key management. It is a lightweight tool that: * Obtains the Enclave's attestation document from NSM (Nitro Secure Module) * Sends the attestation to dstack-kms through the VSOCK proxy * Receives the requested key if the attestation is valid (and the measurement is authorized on-chain, when using on-chain KMS mode) * Makes the key available to your application (e.g., via stdout, file, or shared memory) > **Note:** Unlike GCP where the Guest Agent automatically handles disk encryption, on Nitro you are responsible for deciding how to use the key. Common patterns include encrypting model weights in memory, decrypting configuration secrets, or encrypting a RAM-disk. ## Networking and Communication Nitro Enclaves have no direct network access. All communication goes through VSOCK: VSOCK Communication * The Enclave communicates with the host via VSOCK (AF\_VSOCK socket family) * A VSOCK proxy on the host forwards TCP connections to the outside world (including KMS) * The proxy must be started before the Enclave launches * If the proxy crashes, the Enclave loses external connectivity ### VSOCK Proxy Setup The VSOCK proxy translates between VSOCK (Enclave↔Host) and TCP (Host↔Network): ```bash theme={"system"} # Example: forward Enclave VSOCK port 8000 to host TCP port 8000 socat VSOCK-LISTEN:8000,reuseaddr,fork TCP:localhost:8000 ``` dstack-cloud provides helper scripts and configuration for setting up the VSOCK proxy as part of the deployment process. ## Resource Constraints Nitro Enclaves have static resource allocation: | Resource | Behavior | | ----------- | ---------------------------------------------------------------- | | **CPU** | Allocated at launch (in vCPUs), cannot be changed during runtime | | **Memory** | Allocated at launch (in MB), cannot be changed during runtime | | **Maximum** | Depends on the host EC2 instance type | ### Supported EC2 Instance Types Nitro Enclaves are supported on instances built on the AWS Nitro System. Common choices: | Instance Type | vCPUs | Memory | Notes | | ------------- | ----- | ------ | ----------------- | | `m5.xlarge` | 4 | 16 GB | General purpose | | `m5.2xlarge` | 8 | 32 GB | General purpose | | `c5.2xlarge` | 8 | 16 GB | Compute optimized | | `r5.2xlarge` | 8 | 64 GB | Memory optimized | Note: The EC2 instance must have Nitro Enclave support enabled in its Nitro configuration, and you must allocate enough CPU/memory to accommodate both the host OS and the Enclave. ## When to Use Nitro vs. GCP Choose **AWS Nitro Enclaves** when: * Your existing infrastructure is on AWS * You need the strongest possible isolation (process-level, no host access) * Your workload fits within static resource limits * You don't need persistent disk storage or you want to do it with the key by yourself * You want full control over how keys are used (encryption strategy, key usage pattern) Choose **GCP Confidential VMs** when: * Your existing infrastructure is on GCP * You need automatic disk encryption managed by dstack-cloud * You need direct network access without a VSOCK proxy * You need persistent disk storage * You need elastic resource allocation * You prefer a more managed experience with the Guest Agent handling attestation and key lifecycle ## Next Steps * **[Quick Start: Deploy on AWS Nitro](/dstack-cloud/get-started)** — Hands-on tutorial * **[Run a Workload on AWS Nitro](/dstack-cloud/run-on-nitro)** — Detailed deployment guide * **[Attestation Integration](attestation-integration)** — How attestation works on Nitro # dstack-cloud Overview Source: https://docs.phala.com/dstack-cloud/overview dstack-cloud is an extension of the dstack framework that enables deploying confidential Docker applications on GCP Confidential VMs and AWS Nitro Enclaves. # dstack-cloud Overview dstack-cloud lets you run Docker applications inside hardware-encrypted Trusted Execution Environments (TEEs) on **GCP Confidential VMs** and **AWS Nitro Enclaves** — without owning specialized bare-metal servers. Built on the open-source [dstack framework](https://github.com/Dstack-TEE/dstack), it brings TEE security to mainstream cloud platforms. Your code and data stay encrypted in memory, even from the cloud provider. ## What is dstack dstack is an open-source confidential computing framework originally designed to run on TDX BareMetal servers. It provides: * Hardware-level memory encryption and isolation via TEE (Trusted Execution Environment) * A key management system (KMS) that delivers secrets exclusively to verified workloads * Remote attestation to prove that code is running in genuine TEE hardware * SDK libraries in Python, TypeScript, Rust, and Go for application integration (the **dstack-sdk**) ## What is dstack-cloud dstack-cloud extends dstack to run on major cloud platforms. Instead of requiring bare-metal TDX hardware, it enables deployment on: * **GCP Confidential VMs** (Intel TDX) * **AWS Nitro Enclaves** (NSM) This means you can use the full dstack security model — hardware isolation, attested key delivery, encrypted storage — without owning specialized hardware. ## How It Works How It Works You define your application, and `dstack-cloud deploy` handles the rest: building a TEE environment, managing attestation, and delivering keys. ### On GCP: 1. `dstack-cloud deploy` creates a **Confidential Virtual Machine** running dstack OS with Intel TDX enabled. 2. A **Guest Agent** inside the CVM handles attestation, key retrieval from KMS, and automatic disk encryption. 3. Your application communicates with the Guest Agent via a Unix socket (`/var/run/dstack.sock`). ### On AWS Nitro: 1. AWS Nitro CLI converts your Dockerfile into an **Enclave Image File (EIF)** using Enclave OS (AWS-provided). 2. The EIF runs as a Nitro Enclave — there is no dstack OS and no Guest Agent. 3. **`dstack-util`** inside the Enclave handles attestation and key retrieval from KMS. Your application decides how to use the key. > The two platforms have different architectures. See [Nitro Enclave Integration](nitro-enclave) for a detailed comparison. ## Key Components | Component | Role | | ---------------------------- | ----------------------------------------------------------------------------------------------- | | **dstack-cloud CLI** | Build and deployment tool for both GCP and Nitro | | **dstack-kms** | Key Management Service running in its own TEE; verifies workloads and dispatches keys | | **Gateway** | Edge TLS termination, automatic ACME certificate provisioning, RA-TLS support | | **Guest Agent** (GCP only) | Process inside each CVM that handles attestation, key management, and automatic disk encryption | | **dstack-util** (Nitro only) | Lightweight tool packaged into the EIF; handles attestation and key retrieval from KMS | | **VSOCK Proxy** (Nitro only) | Runs on the EC2 host; forwards network traffic between the Enclave and external services | ## Typical Use Cases ### AI Inference with Model Protection Deploy AI models with model weights fully encrypted in memory. The cloud provider cannot access or copy your proprietary models. ### Sensitive Data Processing Process healthcare, financial, or personal data in a hardware-isolated environment. Memory encryption ensures that even system administrators cannot read your data. ### DeFi and Web3 Applications Run smart contract backend services or trading bots with private keys that never leave the TEE. The on-chain governance model provides verifiable, auditable key management. ## Supported Environments | Platform | TEE Technology | Attestation | Status | | -------------------- | ------------------- | ------------------------ | --------- | | GCP Confidential VMs | Intel TDX + TPM | TDX Quote | Available | | AWS Nitro Enclaves | Nitro Secure Module | NSM Attestation Document | Available | ## What You Need * An AWS or GCP account with appropriate permissions * `dstack-cloud` CLI installed * A `docker-compose.yaml` defining your application * (Optional) [dstack-sdk](https://github.com/Dstack-TEE/dstack/tree/master/sdk) integration for key retrieval from within your containers ## Next Steps * **[Quick Start Tutorial](/dstack-cloud/get-started)** — Deploy your first dstack CVM in 30 minutes * **[Nitro Enclave](nitro-enclave)** — How AWS Nitro Enclaves work in dstack-cloud * **[KMS and Key Delivery](kms-and-key-delivery)** — Understand how keys are managed and delivered * **[Security Model](security-model)** — Trust boundaries and security guarantees * **[Attestation Integration](attestation-integration)** — How remote attestation works across platforms # Register Workload Measurements On-chain Source: https://docs.phala.com/dstack-cloud/register-enclave-measurement Register workload measurements (OS_IMAGE_HASH / RTMR) on-chain for KMS authorization. # Register Workload Measurements On-chain This guide explains how to register workload measurements (RTMR / OS\_IMAGE\_HASH) on-chain so that KMS will authorize your workloads to receive keys. ## Background When a TEE workload starts — whether a dstack CVM (GCP) or a Nitro Enclave (AWS) — it generates a hardware attestation that includes measurements (cryptographic hashes of the code and configuration). KMS checks these measurements against an on-chain allowlist before dispatching keys. If you update your application code, Docker images, or the base image version, the measurements change. You must register the new measurements before KMS will authorize the updated workload. ### Two Layers of On-chain Registration There are two distinct registration flows in the dstack ecosystem: | Registration Type | What Gets Registered | Purpose | Where to Find | | -------------------------------------- | -------------------------------------------- | -------------------------------------------- | ---------------------------------------------- | | **KMS Registration** | KMS's `mrAggregated` (GCP) or public key | Proves KMS runs in a genuine TEE | [Run dstack-kms on GCP](run-kms-on-gcp) Step 8 | | **Workload Registration** (this guide) | Workload's `compose-hash` or `OS_IMAGE_HASH` | Authorizes workload to receive keys from KMS | This document | **This guide covers workload registration** — the process of registering your application's measurement so that KMS will dispatch keys to it. This applies to: * **GCP workloads** — dstack CVMs running applications that need keys from KMS * **Nitro workloads** — Enclaves that need keys from KMS (KMS itself runs on GCP only) Both types of workloads must register their measurements on-chain before KMS will authorize key delivery. | Platform | Measurement Field | Where It Comes From | | --------- | ------------------------- | ----------------------------------- | | GCP (TDX) | `compose-hash` (in RTMR3) | `dstack-cloud deploy` output | | AWS Nitro | `OS_IMAGE_HASH` | `nitro-cli describe-enclave` output | ## Prerequisites * A deployed dstack-kms instance with on-chain governance configured * Governance contracts (DstackKms, DstackApp) deployed * Multisig Safe with signer access * The new measurement value (from your build/deploy output) ## Step 1: Build and Deploy Your Workload First, build and deploy your application to get the measurements: ```bash theme={"system"} # On GCP dstack-cloud deploy dstack-cloud status # Note the compose-hash value in RTMR3 # On Nitro ./scripts/build-eif.sh # Note the OS_IMAGE_HASH value from output ``` ## Step 2: Extract the Measurement **GCP (TDX):** The measurement is the `compose-hash` value in RTMR3, displayed in the `dstack-cloud status` output. This value represents the hash of your docker-compose.yaml configuration. **AWS Nitro:** The measurement is the `OS_IMAGE_HASH`, calculated as `sha256(PCR0 || PCR1 || PCR2)`. Run: ```bash theme={"system"} ./scripts/build-eif.sh ``` The script outputs: ``` PCR0: PCR1: PCR2: OS_IMAGE_HASH: ``` **PCR values are derived from:** * **PCR0** — Complete EIF content hash (enclave image) * **PCR1** — Linux kernel and boot ramdisk * **PCR2** — Application layer (Docker image filesystem) > **Note:** The `OS_IMAGE_HASH` is the value you register on-chain. Any change to the Dockerfile, application code, or configuration produces a different hash. **Alternative: Preview measurements without full build:** ```bash theme={"system"} dstack-util get-keys --show-mrs \ --kms-url "https://your-kms:12001" \ --app-id "0xYOUR_APP_ID" ``` > **Important:** When using `--show-mrs` to preview, you must use the exact same `KMS_URL`, `APP_ID`, and `root_ca.pem` as the production build. Any difference will produce different PCR values. ## Step 3: Prepare the Governance Transaction You need to call `DstackKms.addOsImageHash()` with the new measurement value. ### 3.1 Draft the Transaction Using the Safe web interface ([https://app.safe.global](https://app.safe.global)): 1. Connect your signer wallet 2. Go to your Safe 3. Click "New Transaction" → "Contract interaction" 4. Select the `DstackKms` contract 5. Select the `addOsImageHash` function 6. Enter the measurement value (hex string) 7. Review and submit ### 3.2 Alternative: Using CLI ```bash theme={"system"} # Using cast (from Foundry) cast send \ "addOsImageHash(bytes32)" \ 0xYOUR_MEASUREMENT_HASH \ --rpc-url $RPC_URL \ --private-key $SIGNER_KEY ``` > **Note:** In production, do not use a single private key. Submit the transaction through the multisig Safe instead. ## Step 4: Collect Signatures The transaction enters the multisig queue. Other signers must approve it. 1. Each signer connects to the Safe web interface 2. Opens the pending transaction 3. Reviews the measurement value 4. Confirms (signs) the transaction The transaction is executed once the required threshold of signatures is collected. ## Step 5: Wait for Timelock After multisig approval, the transaction enters the Timelock queue. * **Production:** Wait 24-72 hours (depending on your configuration) * **Testnet:** Wait 1-4 hours During this period, any signer or observer can review and, if necessary, raise concerns. ## Step 6: Execute After the Timelock expires, the transaction can be executed: 1. Open the Safe web interface 2. Find the transaction in the queue 3. Click "Execute" The measurement is now registered on-chain. KMS will authorize workloads with this measurement. ## Step 7: Verify Verify that the measurement is registered: ```bash theme={"system"} # Using cast cast call \ "isAuthorized(bytes32)(bool)" \ 0xYOUR_MEASUREMENT_HASH \ --rpc-url $RPC_URL ``` Expected output: `true` ## Workflow Summary Measurement Registration Flow ## Registering Multiple Measurements If you are deploying multiple workloads (e.g., different application versions for canary testing), you can register multiple measurements in a single governance transaction: 1. Call `addOsImageHash` for each measurement value 2. Bundle them into a batch transaction in the Safe 3. Submit for approval as usual ## Revoking a Measurement If a measurement is found to be compromised or no longer needed: ```bash theme={"system"} cast send \ "removeOsImageHash(bytes32)" \ 0xCOMPROMISED_MEASUREMENT_HASH \ --rpc-url $RPC_URL ``` Follow the same governance flow (multisig → timelock → execute). ## Common Issues | Issue | Solution | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | KMS refuses to dispatch keys | Verify the measurement is registered: `cast call ... isAuthorized(...)`. Check that the measurement matches exactly (case-sensitive hex). | | Governance transaction stuck | Verify the timelock has expired. Check that the Safe has sufficient gas. | | Wrong measurement registered | You must revoke the wrong measurement and register the correct one through separate governance transactions. | | Measurement changes on every deploy | The `compose-hash` / `OS_IMAGE_HASH` is derived from your Docker images and configuration. Use pinned image tags (SHA256 digests) for reproducible builds. | ## Next Steps * **[Manage Governance](manage-governance)** — Other governance operations * **[Concept: Attestation Integration](/dstack-cloud/attestation-integration)** — How measurements are generated * **[Concept: KMS and Key Delivery](/dstack-cloud/kms-and-key-delivery)** — How KMS uses measurements # Release Notes Source: https://docs.phala.com/dstack-cloud/release-notes Release notes, version history, compatibility notes, and known issues. # Release Notes ## v0.6.0 (Current) ### Features * Initial release with support for GCP Confidential VMs and AWS Nitro Enclaves * dstack-cloud CLI for launching CVMs from Docker Compose configurations * KMS with two modes: Local Key Provider and standalone KMS * On-chain governance (DstackKms, DstackApp, Multisig + Timelock) * RA-TLS for secure KMS ↔ workload communication * VSOCK proxy support for Nitro Enclave networking * KMS bootstrap (Onboard → Finish) flow * Attestation verification (TDX Quote on GCP, NSM Document on Nitro) ### Compatibility Notes * Requires Intel TDX-capable GCP zones or Nitro Enclave-capable EC2 instances *** ## Changelog All notable changes will be documented here. Format follows [Keep a Changelog](https://keepachangelog.com/). # Run a dstack-kms CVM on GCP Source: https://docs.phala.com/dstack-cloud/run-kms-on-gcp Deploy a dstack-kms (Key Management Service) instance on GCP as a Confidential VM with on-chain governance. # Run a dstack-kms CVM on GCP Run your own dstack-kms instance on GCP so you have full control over key policy. This page covers the production deployment workflow: bootstrap the KMS, register its attestation on-chain, and verify it's serving keys correctly. > **Reference:** This guide is based on the [official deployment guide](https://github.com/Phala-Network/dstack-cloud-deployment-guide/blob/main/guide_EN.md). > **Platform Requirement:** `dstack-cloud deploy` must be run on **Linux**. macOS is not supported because the FAT32 shared disk image created by macOS `dosfstools` fails GCP image validation. *** ## Prerequisites * A GCP project with Confidential VM (TDX) quota * `gcloud` CLI installed and authenticated (`gcloud auth login`) * `gsutil` available in PATH (required by `dstack-cloud deploy` for GCS upload) * `docker` / `docker compose` * `node` + `npm` (for smart contract deployment) * `jq` * `mtools` (provides `mcopy`) * `dosfstools` (provides `mkfs.fat`) * **`dstack-cloud` CLI** installed: ```bash theme={"system"} curl -fsSL -o ~/.local/bin/dstack-cloud \ https://raw.githubusercontent.com/Phala-Network/meta-dstack-cloud/main/scripts/bin/dstack-cloud chmod +x ~/.local/bin/dstack-cloud ``` * **For on-chain governance mode:** * A blockchain RPC endpoint (e.g., `https://sepolia.base.org`) * A wallet with sufficient balance * Deployed smart contracts (DstackKms, DstackApp) — see [Deploy On-chain KMS Smart Contracts](deploy-onchain-kms) * You’ll need `KMS_CONTRACT_ADDR` and `APP_CONTRACT_ADDR` before KMS bootstrap/finish > **Note:** This guide uses pre-built KMS Docker images. If you need to build KMS from source or customize configuration, see the [official KMS build guide](https://github.com/Phala-Network/dstack-cloud/blob/master/docs/tutorials/kms-build-configuration.md). *** ## Step 1: Configure dstack-cloud Follow the same steps as [Run a Workload on GCP](run-on-gcp#step-1-configure-dstack-cloud) to set up your global config and GCS bucket. *** ## Step 2: Pull the OS Images Follow the same steps as [Run a Workload on GCP](run-on-gcp#step-2-pull-the-os-image) to download the `dstack-cloud-0.6.0` images. > If `disk.raw` is missing, deployment may create a VM that cannot boot (UEFI loop: `Failed to load image`). *** ## Step 3: Create a KMS Project ```bash theme={"system"} mkdir -p workshop-run dstack-cloud new workshop-run/kms-prod \ --os-image dstack-cloud-0.6.0 \ --key-provider tpm \ --instance-name dstack-kms cd workshop-run/kms-prod ``` Generated files include: * `app.json` * `docker-compose.yaml` * `prelaunch.sh` *** ## Step 4: Build or Select KMS Docker Image Use pre-built image or build your own: ```bash theme={"system"} # Option 1: Pre-built # KMS_IMAGE=cr.kvin.wang/dstack-kms:latest # Option 2: Build from source (from deployment-guide workshop repo) cd workshop/kms/builder ./build-image.sh dstack-kms:latest docker push dstack-kms:latest ``` *** ## Step 5: Configure KMS Runtime Replace generated compose/prelaunch with KMS config. > The compose templates below come from:\ > `Phala-Network/dstack-cloud-deployment-guide`\ > Clone that repo first if you only cloned this docs repo. ### Option A: Direct RPC ```bash theme={"system"} cp /path/to/dstack-cloud-deployment-guide/workshop/kms/docker-compose.direct.yaml docker-compose.yaml ``` Create `prelaunch.sh`: ```bash theme={"system"} cat > prelaunch.sh <<'EOF' #!/bin/sh cat > .env <<'ENVEOF' KMS_HTTPS_PORT=12001 AUTH_HTTP_PORT=18000 KMS_IMAGE=cr.kvin.wang/dstack-kms:latest ETH_RPC_URL=https://sepolia.base.org KMS_CONTRACT_ADDR= APP_CONTRACT_ADDR= DSTACK_REPO=https://github.com/Phala-Network/dstack-cloud.git DSTACK_REF=14963a2ccb0ec7bef8a496c1ac5ac40f5593145d ENVEOF EOF chmod +x prelaunch.sh ``` ### Option B: Light Client (Helios) ```bash theme={"system"} cp /path/to/dstack-cloud-deployment-guide/workshop/kms/docker-compose.light.yaml docker-compose.yaml ``` `prelaunch.sh` can stay the same unless your template requires different vars. *** ## Step 6: Deploy KMS CVM ```bash theme={"system"} dstack-cloud deploy --delete ``` If bucket does not exist, create explicitly first: ```bash theme={"system"} gcloud storage buckets create gs:// \ --project \ --location us-central1 ``` *** ## Step 7: Open Firewall ```bash theme={"system"} dstack-cloud fw allow 12001 dstack-cloud fw allow 18000 ``` For light-client mode: ```bash theme={"system"} dstack-cloud fw allow 18545 ``` Port notes: * `12001/tcp`: KMS API (**required**) * `18000/tcp`: auth-api debug endpoint (**optional**) * `18545/tcp`: helios RPC (light mode only) *** ## Step 8: Bootstrap (First-time) ### 8.1 Check KMS endpoint ```bash theme={"system"} dstack-cloud status ``` ### 8.2 Get attestation info ```bash theme={"system"} KMS_URL="http://:12001" curl -s "$KMS_URL/prpc/Onboard.GetAttestationInfo?json" | jq . ``` Capture: * `device_id` * `mr_aggregated` * `os_image_hash` ### 8.3 Register measurements on-chain (required for on-chain mode) ```bash theme={"system"} npx hardhat kms:add-image 0x --network custom npx hardhat kms:add 0x --network custom npx hardhat kms:add-device 0x --network custom ``` > `device_id` must come from `Onboard.GetAttestationInfo` (not dummy serial-console value). ### 8.4 Run bootstrap ```bash theme={"system"} curl -s "$KMS_URL/prpc/Onboard.Bootstrap?json" \ -d '{"domain":""}' | tee bootstrap-info.json | jq . ``` ### 8.5 Finish ```bash theme={"system"} curl "$KMS_URL/finish" sleep 5 ``` *** ## Step 9: Verify ```bash theme={"system"} # Onboard mode check (before finish) curl -s "http://:12001/prpc/Onboard.GetAttestationInfo?json" | jq . # Normal mode check (after finish) curl -sk "https://:12001/prpc/GetMeta?json" -d '{}' | jq . ``` *** ## Common Issues | Issue | Solution | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | macOS image/shared-disk errors | Use Linux host for deploy | | `Boot image ... not found` | Ensure image search path is correct and `disk.raw` exists under `/dstack-cloud-0.6.0/` | | VM `RUNNING` but 12001/18000 unreachable + serial shows `UEFI: Failed to load image` | Wrong boot image format/content. Ensure `disk.raw` comes from `dstack-cloud-0.6.0-uki.tar.gz` (do **not** rename `rootfs.img.parted.verity` as `disk.raw`) | | `gsutil` missing | Install Google Cloud SDK/gsutil and ensure in PATH | | `mcopy` missing | Install `mtools` | | `mkfs.fat` missing | Install `dosfstools` | | `missing field 'status'` from KMS flow | Usually indicates auth-eth / on-chain config mismatch (RPC/contract addresses/registrations). Verify `ETH_RPC_URL`, `KMS_CONTRACT_ADDR`, `APP_CONTRACT_ADDR`, and on-chain image/MR/device registration | | KMS port responds but APIs return 404 | Shared disk config stale/wrong; `dstack-cloud deploy --delete` | *** ## Cleanup ```bash theme={"system"} dstack-cloud stop dstack-cloud remove ``` Or manually: ```bash theme={"system"} gcloud compute instances delete dstack-kms --zone=us-central1-a gcloud compute images delete dstack-kms-boot gcloud compute images delete dstack-kms-shared ``` *** ## Next Steps * [Register Workload Measurements](register-enclave-measurement) * [KMS and Key Delivery](/dstack-cloud/kms-and-key-delivery) * [Attestation Integration](/dstack-cloud/attestation-integration) * [Deploy On-chain KMS Smart Contracts](deploy-onchain-kms) # Run a Workload on GCP Source: https://docs.phala.com/dstack-cloud/run-on-gcp Step-by-step guide to deploy a Docker application as a dstack CVM on GCP with Intel TDX. # Run a Workload on GCP Deploy a Docker application as a dstack CVM on GCP with Intel TDX. This page covers the full workflow — from configuration to verification — including how to choose between managed and self-hosted KMS endpoints. *** ## Prerequisites * A GCP project with Confidential VM quota enabled * Intel TDX Confidential VMs are available in select zones (for example `us-central1-a`) * `gcloud` CLI installed and authenticated ```bash theme={"system"} gcloud auth login gcloud config set project YOUR_PROJECT_ID ``` * Linux host for deployment (recommended) * Docker installed * `gsutil` available in PATH * `mtools` (`mcopy`) and `dosfstools` (`mkfs.fat`) installed * `dstack-cloud` CLI installed ```bash theme={"system"} curl -fsSL -o ~/.local/bin/dstack-cloud \ https://raw.githubusercontent.com/Phala-Network/meta-dstack-cloud/main/scripts/bin/dstack-cloud chmod +x ~/.local/bin/dstack-cloud ``` > Why Linux + mtools + dosfstools? `dstack-cloud deploy` builds a shared FAT image and needs these tools in local environment. *** ## Step 1: Configure `dstack-cloud` Edit global config: ```bash theme={"system"} dstack-cloud config-edit ``` `dstack-cloud` uses **JSON** config (`~/.config/dstack-cloud/config.json`). Example: ```json theme={"system"} { "services": { "kms_urls": ["https://kms.tdxlab.dstack.org:12001"], "gateway_urls": ["https://gateway.tdxlab.dstack.org:12002"], "pccs_url": "" }, "image_search_paths": ["/path/to/images"], "gcp": { "project": "YOUR_PROJECT_ID", "zone": "us-central1-a", "bucket": "gs://YOUR_BUCKET_NAME" } } ``` Create bucket if needed: ```bash theme={"system"} gcloud storage buckets create gs://YOUR_BUCKET_NAME --project YOUR_PROJECT_ID --location us-central1 ``` ### KMS Options | Option | Description | When to Use | | ---------------------- | ------------------------------------------------ | ------------------------------------ | | **Phala Official KMS** | Use managed KMS endpoints in `services.kms_urls` | Quick start, testing | | **Self-hosted KMS** | Use your own KMS endpoint in `services.kms_urls` | Production, compliance, full control | For Self-hosted KMS, point `services.kms_urls` to your deployed KMS URL (see [Run a dstack-kms CVM on GCP](run-kms-on-gcp) for how to set one up). If you run with key provider `tpm`/`none` (no external KMS), remove `.env` in project and remove `env_file` from `app.json`. *** ## Step 2: Pull the OS Image For `dstack-cloud-0.6.0`, download both archives: ```bash theme={"system"} dstack-cloud pull https://github.com/Phala-Network/meta-dstack-cloud/releases/download/v0.6.0-test/dstack-cloud-0.6.0.tar.gz dstack-cloud pull https://github.com/Phala-Network/meta-dstack-cloud/releases/download/v0.6.0-test/dstack-cloud-0.6.0-uki.tar.gz ``` Verify boot image file exists: ```bash theme={"system"} ls -lh /path/to/images/dstack-cloud-0.6.0/disk.raw ``` > If `disk.raw` is missing, VM may boot-loop with UEFI `Failed to load image`. *** ## Step 3: Create a Project ```bash theme={"system"} dstack-cloud new my-gcp-app --os-image dstack-cloud-0.6.0 --instance-name dstack-my-app cd my-gcp-app ``` *** ## Step 4: Configure Project (`app.json`) Update key fields in `app.json`: * `gcp_config.project` * `gcp_config.zone` * `gcp_config.bucket` * `gcp_config.instance_name` Choose key provider mode: * **External KMS mode (recommended):** `"key_provider": "kms"` * **No external KMS mode:** `"key_provider": "tpm"` (or `none`) Gateway options: * If you use dstack gateway URL routing, keep `gateway_enabled: true` * If you access service directly via VM public IP + opened port, set `gateway_enabled: false` *** ## Step 5: Define Your Application Edit `docker-compose.yaml`: ```yaml theme={"system"} services: web: image: nginx:latest ports: - "8080:80" ``` *** ## Step 6: (Optional) Add Environment Variables If your app needs secrets/config, create `.env`: ```env theme={"system"} API_KEY=your-api-key DATABASE_URL=postgres://user:pass@host:5432/db ``` In KMS mode, env values are encrypted client-side and only decrypted inside CVM after attestation. *** ## Step 7: Deploy ```bash theme={"system"} dstack-cloud deploy --delete ``` `dstack-cloud` will: 1. Prepare shared config files 2. Upload image artifacts to GCS 3. Create a GCP TDX Confidential VM 4. Start VM and run compose workload First deployment usually takes several minutes. *** ## Step 8: Open Firewall Open app port(s): ```bash theme={"system"} dstack-cloud fw allow 8080 ``` List firewall rules: ```bash theme={"system"} dstack-cloud fw list ``` *** ## Step 9: Verify Check status: ```bash theme={"system"} dstack-cloud status ``` View logs: ```bash theme={"system"} dstack-cloud logs --follow ``` Access app: * Direct VM mode (`gateway_enabled=false`): ```bash theme={"system"} curl http://:8080 ``` * Gateway mode (`gateway_enabled=true`): use URL printed in `dstack-cloud status`. *** ## Managing Your Deployment ```bash theme={"system"} dstack-cloud logs dstack-cloud stop dstack-cloud start dstack-cloud remove ``` *** ## Common Issues | Issue | Solution | | --------------------------------------------------- | -------------------------------------------------------------------------------------- | | `Boot image 'dstack-cloud-0.6.0' not found locally` | Ensure `disk.raw` exists under `/dstack-cloud-0.6.0/` | | VM RUNNING but serial log shows UEFI load failures | Wrong boot image source; use official `-uki.tar.gz` image containing `disk.raw` | | `gsutil` not found | Install Google Cloud SDK / ensure PATH | | `mcopy` not found | Install `mtools` | | `mkfs.fat` not found | Install `dosfstools` | | `.env found but KMS is not enabled` | Remove `.env` and remove `env_file` from `app.json`, or set key provider back to `kms` | | App not reachable immediately | Wait for compose startup to complete; check `dstack-cloud logs` | *** ## Next Steps * [Run a dstack-kms CVM on GCP](run-kms-on-gcp) * [Attestation Integration](/dstack-cloud/attestation-integration) * [Run a Workload on AWS Nitro](run-on-nitro) # Run a Workload on AWS Nitro Source: https://docs.phala.com/dstack-cloud/run-on-nitro Step-by-step guide to deploy a Docker application as a Nitro Enclave on AWS. # Run a Workload on AWS Nitro Run a Docker application inside an AWS Nitro Enclave — one of the strongest isolation guarantees available in the public cloud. The host OS cannot read your enclave's memory, even with root access. > **Important:** On AWS Nitro, your workload runs inside an **AWS Enclave OS image** — not a dstack CVM. There is no Guest Agent. `dstack-util` is packaged into the Enclave for attestation and key retrieval. You control how keys are used (for example disk/data decryption in your app). ## Prerequisites * AWS account permissions for EC2 + Nitro Enclaves * AWS CLI configured (`aws configure`) * Docker installed * Git + `gh` CLI (if using GitHub template/release flow) * The [dstack-nitro-enclave-app-template](https://github.com/Phala-Network/dstack-nitro-enclave-app-template) repository ## Overview Deployment flow: 1. Create app from template 2. Replace KMS root CA certificate (**required before build**) 3. Build EIF and get measurements 4. Register OS\_IMAGE\_HASH on-chain 5. Deploy/run on Nitro host and retrieve keys ### Key Delivery via KMS On Nitro, `dstack-util` inside Enclave reaches KMS through host-side VSOCK proxy. KMS verifies Nitro attestation and policy before returning keys. **KMS Options** | Option | Description | When to Use | | ---------------------- | -------------------- | ---------------------------------- | | **Phala Official KMS** | Managed KMS by Phala | Quick start/testing | | **Self-hosted KMS** | Your own dstack-kms | Production/compliance/full control | Self-hosted KMS can run on: * GCP TDX CVM — see [Run a dstack-kms CVM on GCP](run-kms-on-gcp) * Intel TDX bare metal *** ## Step 1: Create Your App from Template ```bash theme={"system"} gh repo create my-enclave-app \ --template Phala-Network/dstack-nitro-enclave-app-template \ --private git clone https://github.com/YOUR_USER/my-enclave-app.git cd my-enclave-app ``` *** ## Step 2: Replace KMS Root CA Certificate > ⚠️ Required. Template `app/root_ca.pem` is placeholder only. `root_ca.pem` is baked into image and affects measured hash. ```bash theme={"system"} # From your running KMS endpoint curl -sk https://:12001/prpc/GetTempCaCert?json \ | jq -r .temp_ca_cert > app/root_ca.pem ``` > Use a **domain name** in KMS URL when possible. Avoid raw IP in production. If KMS CA rotates, rebuild EIF and re-register new image hash. *** ## Step 3: Configure and Build EIF ### 3.1 Template Variables `app/entrypoint.sh` uses placeholders: ```bash theme={"system"} KMS_URL="__KMS_URL__" APP_ID="__APP_ID__" ``` Do not hardcode manually; pass via build inputs. ### 3.2 Local Build ```bash theme={"system"} DSTACK_UTIL=/path/to/dstack-util \ KMS_URL=https://your-kms-domain:12001 \ APP_ID=0xYOUR_APP_ID \ ./scripts/build-eif.sh ``` or source build variant: ```bash theme={"system"} KMS_URL=https://your-kms-domain:12001 \ APP_ID=0xYOUR_APP_ID \ DSTACK_COMMIT=14963a2ccb0ec7bef8a496c1ac5ac40f5593145d \ ./scripts/build-eif.sh ``` ### 3.3 GitHub Actions Build ```bash theme={"system"} git add . git commit -m "configure enclave app" git push origin main git tag v0.1.0 git push origin v0.1.0 ``` ### 3.4 Output `./output/`: * `enclave.eif` * `measurements.json` * `measurements.sigstore.json` (CI) Output includes: * `PCR0`, `PCR1`, `PCR2` * `OS_IMAGE_HASH = sha256(PCR0 || PCR1 || PCR2)` *** ## Step 4: Register OS\_IMAGE\_HASH On-chain Before key retrieval, register measured image hash. ### Development ```bash theme={"system"} cd dstack/kms/auth-eth npx hardhat kms:add-image --network ``` ### Production Use governance/multisig/timelock flow. See [Register Workload Measurements](register-enclave-measurement). ### App policy check (important) For successful `GetAppKey`, app policy must also allow your runtime attestation: * compose hash/image hash allowed (`app:add-hash`) * device policy satisfied (either `allowAnyDevice=true` or device explicitly added) Useful commands: ```bash theme={"system"} npx hardhat app:add-hash --app-id --network # Choose one: npx hardhat app:set-allow-any-device --app-id true --network # or npx hardhat app:add-device --app-id --network ``` *** ## Step 5: Deploy on EC2 ### 5.1 Launch Nitro-capable instance | Setting | Value | | -------------- | ----------------------------------------- | | Instance type | `c5.xlarge` or larger (c5/m5/r5 families) | | AMI | Amazon Linux 2023 | | Nitro Enclaves | enabled | ### 5.2 Install Nitro tooling ```bash theme={"system"} sudo yum install -y aws-nitro-enclaves-cli sudo systemctl enable nitro-enclaves-allocator.service sudo systemctl start nitro-enclaves-allocator.service ``` ### 5.3 Allocator resources `/etc/nitro_enclaves/allocator.yaml`: ```yaml theme={"system"} memory_mib: 2048 cpu_count: 2 ``` ```bash theme={"system"} sudo systemctl restart nitro-enclaves-allocator.service ``` ### 5.4 Run EIF ```bash theme={"system"} gh release download v0.1.0 -p 'enclave.eif' nitro-cli run-enclave --eif-path enclave.eif --cpu-count 2 --memory 2048 ``` ### 5.5 Verify ```bash theme={"system"} nitro-cli describe-enclaves nitro-cli console ``` *** ## Entrypoint Notes The template uses `dstack-util get-keys` and a VSOCK proxy for key retrieval. Keys are returned to `/var/run/dstack/keys.json` inside the enclave, or captured on the host via VSOCK in helper scripts. > **Important:** The KMS URL, APP\_ID, and `root_ca.pem` are all baked into the enclave image and affect the measured hash. You must use the exact same inputs between `--show-mrs` preview and the real run — any difference produces a different `OS_IMAGE_HASH`. *** ## Troubleshooting | Issue | Solution | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `Boot denied: OS image is not allowed` | register the exact runtime OS\_IMAGE\_HASH; ensure runtime build inputs match show-mrs inputs | | Raw IP in KMS URL causes TLS/SAN issues | use DNS name (`kms.example.com`), not bare IP | | `get_keys.sh` completes but key file is empty | treat as failure; inspect enclave console and KMS policy response | | `DEBUG_ENCLAVE=1` run cannot pass policy | expected for production validation; debug mode alters attestation behavior and should be used only for diagnostics | | `Insufficient CPUs available in the pool` | terminate leftover enclaves and adjust allocator CPU count | *** ## Differences from GCP Nitro and GCP take fundamentally different approaches. On GCP, the Guest Agent handles attestation, key management, and automatic disk encryption. On Nitro, `dstack-util` only retrieves the key — your application decides what to encrypt. For a full comparison, see the [overview](overview) and [KMS and Key Delivery](kms-and-key-delivery). *** ## Next Steps * [Register Workload Measurements](register-enclave-measurement) * [Run a dstack-kms on GCP](run-kms-on-gcp) * [Nitro Enclave concept](/dstack-cloud/nitro-enclave) # Troubleshooting Runbook Source: https://docs.phala.com/dstack-cloud/runbook Troubleshooting runbook for common dstack-cloud deployment issues. # Troubleshooting Runbook When something breaks in a dstack-cloud deployment, the root cause usually falls into one of these categories: attestation mismatch, KMS unavailability, governance hold-up, or infrastructure issues. This runbook covers the most common failure modes and how to diagnose them. ## RA-TLS Connection Failures ### Symptoms * Workload logs show "RA-TLS handshake failed" * KMS logs show "connection from unverified peer" * Workload cannot obtain keys ### Diagnosis ```bash theme={"system"} # Check workload logs dstack-cloud logs # Check KMS logs cd kms-prod dstack-cloud logs ``` ### Common Causes and Fixes | Cause | Fix | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Workload attestation invalid | Verify the workload's measurements match what is registered on-chain. Run `dstack-cloud status` to get current measurements. | | KMS attestation invalid | Verify KMS is running in a genuine TEE. Check `dstack-cloud status` for the KMS instance. | | Clock skew between workload and KMS | RA-TLS requires relatively synchronized clocks. Check NTP configuration on both sides. | | Certificate expired | Check that the RA-TLS certificates have not expired. Restart the CVM to regenerate. | ## Attestation Verification Failures ### Symptoms * KMS refuses to dispatch keys * Logs show "measurement not authorized" or "attestation verification failed" ### Diagnosis ```bash theme={"system"} # Get current measurements dstack-cloud status # Note the RTMR3 / OS_IMAGE_HASH # Check on-chain authorization cast call \ "isAuthorized(bytes32)(bool)" \ 0xYOUR_MEASUREMENT_HASH \ --rpc-url $RPC_URL ``` ### Common Causes and Fixes | Cause | Fix | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Measurement not registered on-chain | Register the measurement via governance. See [Register Workload Measurements](/dstack-cloud/register-enclave-measurement). | | Measurement changed after update | Application code or Docker image changed. Register the new measurement. | | KMS pointing to wrong contract | Verify `KMS_CONTRACT_ADDR` environment variable. | | RPC returns stale state | Check RPC provider health. Switch to a backup RPC endpoint. | ## CVM / Enclave Startup Failures ### Symptoms * `dstack-cloud deploy` succeeds but CVM exits immediately * `dstack-cloud status` shows "ERROR" or "STOPPED" ### Diagnosis ```bash theme={"system"} # Check logs for the reason dstack-cloud logs # Check resource allocation (GCP) gcloud compute instances describe # Check resource allocation (Nitro) sudo amazon-nitro-enclaves-cli describe-enclaves ``` ### Common Causes and Fixes | Cause | Fix | | --------------------- | -------------------------------------------------------------------------------------------------------------- | | Insufficient memory | Allocate more memory. On GCP, use a larger machine type. On Nitro, increase `--memory` in `allocate-enclaves`. | | Invalid Docker image | Verify the image exists and is accessible. Use SHA256 digests for pinned images. | | Container crash loop | Check application logs. The container may have a runtime error. | | OS image incompatible | Ensure the OS image version matches the dstack-cloud CLI version. | ### GCP-specific | Cause | Fix | | ------------------------------ | ----------------------------------------------------------------- | | Confidential VM quota exceeded | Request quota increase in GCP Console. | | VM not booting as TDX | Verify the VM has `confidential-compute: enabled` in GCP Console. | ### Nitro-specific | Cause | Fix | | ------------------------------- | --------------------------------------------------------- | | Enclave image (EIF) too large | Reduce Docker image size. Use multi-stage builds. | | Nitro driver not installed | Install: `sudo apt-get install -y aws-nitro-enclaves-cli` | | Enclave resource limit exceeded | Run `allocate-enclaves` with higher values and retry. | ## On-chain Authorization Failures ### Symptoms * KMS logs show "workload not authorized" * Keys are not dispatched despite correct attestation ### Diagnosis ```bash theme={"system"} # Check if measurement is authorized on-chain cast call "isAuthorized(bytes32)(bool)" 0xHASH --rpc-url $RPC_URL # Check DstackKms contract state cast call "owner()(address)" --rpc-url $RPC_URL ``` ### Common Causes and Fixes | Cause | Fix | | ---------------------------------------- | -------------------------------------------------------------------------------------- | | Measurement registered on wrong contract | Verify the KMS is configured to use the correct `DstackKms` address. | | Governance transaction not yet executed | Check the Safe for pending transactions. Wait for timelock. | | Measurement was revoked | Check the Safe transaction history. If revoked by mistake, re-register via governance. | ## KMS Unavailable ### Symptoms * Workloads cannot connect to KMS * `dstack-cloud status` shows KMS as stopped or unreachable ### Diagnosis ```bash theme={"system"} # Check KMS status cd kms-prod dstack-cloud status # Check KMS logs dstack-cloud logs # Test connectivity curl -k https://:12001/health ``` ### Common Causes and Fixes | Cause | Fix | | --------------------------- | -------------------------------------------------------------------------------------------------- | | KMS CVM stopped | Restart: `dstack-cloud start` | | KMS bootstrap not completed | Complete the bootstrap procedure. See [Run a dstack-kms CVM on GCP](/dstack-cloud/run-kms-on-gcp). | | Network issue | Verify firewall rules. Check VSOCK proxy on Nitro. | | KMS out of memory | Allocate more resources. Check `dstack-cloud logs` for OOM errors. | ## Governance Transactions Stuck ### Symptoms * Governance proposal not advancing * Transaction in Safe queue not executing ### Diagnosis 1. Check the Safe web interface for transaction status 2. Check if the timelock has expired 3. Verify the Safe has sufficient gas ### Common Causes and Fixes | Cause | Fix | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Not enough signatures | Contact missing signers. If a signer is unavailable, consider adding a new signer (requires governance). | | Timelock not yet expired | Check the exact expiry time. Wait. | | Safe out of gas | Send ETH to the Safe address. | | Transaction will revert | Simulate the transaction before executing. The contract state may have changed since the proposal was created. Cancel and re-submit. | | Stale transaction in queue | Cancel the stale transaction through the Safe interface. Submit a new one. | ## VSOCK Proxy Failures (Nitro-specific) ### Symptoms * Enclave cannot reach KMS or external services * `dstack-cloud logs` shows network timeout errors ### Diagnosis ```bash theme={"system"} # Check if socat is running ps aux | grep socat # Check VSOCK proxy logs journalctl -u vsock-proxy -f # if running as systemd service # Test VSOCK connectivity from the host echo "test" | socat - VSOCK-CONNECT:1:8000 ``` ### Common Causes and Fixes | Cause | Fix | | ----------------- | ----------------------------------------------------------------------------------------------------------- | | socat not running | Start the VSOCK proxy. Check `prelaunch.sh` for the proxy startup command. | | Wrong VSOCK port | Verify the VSOCK port matches between the Enclave and the proxy. | | socat crashed | Restart socat. Check system logs for crash reason. Consider running as a systemd service with auto-restart. | | Port conflict | Another process is using the same port. Change the proxy port configuration. | ## Emergency Operations ### Revoke a Compromised Measurement 1. Draft a governance transaction to remove the measurement from `DstackKms` 2. Request expedited approval from all signers 3. Wait for the timelock (cannot be bypassed) 4. Execute after the delay 5. Verify the measurement is no longer authorized ### KMS Key Compromise If the KMS root key may have been compromised: 1. Stop the KMS immediately: `dstack-cloud stop` 2. Audit all workloads that received keys from the compromised KMS 3. Rotate affected application keys 4. Deploy a new KMS instance with fresh measurements 5. Register the new KMS measurements on-chain 6. Revoke the old KMS measurements 7. Restart workloads against the new KMS ### Full System Recovery 1. Stop all CVMs and KMS instances 2. Verify blockchain state is consistent 3. Redeploy from known-good configuration 4. Re-register measurements if needed 5. Verify end-to-end key delivery 6. Review governance activity for suspicious transactions ## Diagnostic Commands Cheat Sheet ```bash theme={"system"} # Check deployment status dstack-cloud status # View logs dstack-cloud logs dstack-cloud logs --follow dstack-cloud logs --container # Check measurements dstack-cloud status | grep -E "measurement|hash|rtmr" # On-chain queries (using cast) cast call "isAuthorized(bytes32)(bool)" 0xHASH --rpc-url $RPC_URL cast call "owner()(address)" --rpc-url $RPC_URL # GCP diagnostics gcloud compute instances describe gcloud logging read "resource.type=gce_instance" # Nitro diagnostics sudo amazon-nitro-enclaves-cli describe-enclaves sudo amazon-nitro-enclaves-cli allocate-enclaves --cpu-count 2 --memory 4096 ``` ## Next Steps * **[Monitoring and Alerting](monitoring-alerting)** — Set up proactive monitoring * **[Upgrade Procedures](upgrade)** — Upgrade versions to fix known issues # Threat Model and Security Guarantees Source: https://docs.phala.com/dstack-cloud/security-model Trust boundaries, threat categories, security guarantees, and residual risks for dstack-cloud deployments. # Threat Model and Security Guarantees ## Trust Boundaries Understanding what you must trust — and what's protected by hardware or blockchain — is the foundation of evaluating dstack-cloud's security. This page maps out trust boundaries, threat categories, and the guarantees the system provides. Trust Boundaries ### Untrusted | Entity | Assumption | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | **Cloud platform** (GCP / AWS) | May attempt to read workload memory, inspect traffic, or modify configurations. TEE hardware prevents memory access. | | **Host machine** (EC2 instance on Nitro) | Has root access to the host OS. Cannot access Enclave memory or modify Enclave code. | | **Network attackers** | May intercept, modify, or replay network traffic. Defended by TLS / RA-TLS. | | **RPC providers** | May return stale or malicious blockchain state. KMS should use multiple RPC sources. | ### Protected by Hardware (TEE) | Entity | Protection | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | **dstack CVM** (GCP workload) | Memory encrypted by TDX. Host and cloud platform cannot read or modify it. Guest Agent handles attestation and key management. | | **Nitro Enclave** (AWS workload) | Memory encrypted by Nitro. Host and cloud platform cannot read or modify it. `dstack-util` handles attestation and key retrieval. | | **dstack-kms** (KMS) | Runs in its own TEE. Keys are generated and stored inside; never exposed outside. | ### Protected by Blockchain Consensus | Entity | Protection | | --------------------------------------------- | ------------------------------------------------------------------------------------- | | **On-chain contracts** (DstackKms, DstackApp) | Immutable unless governance process is followed. Changes require multisig + timelock. | ### Partially Trusted | Entity | Risk | | -------------------- | ------------------------------------------------------------------------------------------------------------------ | | **Multisig signers** | Can collude to push through unauthorized changes. Impact is limited by the signature threshold and timelock delay. | ## Threat Categories ### T1: Malicious Cloud Platform Operator or Compromised Host OS * **Attack:** Cloud provider or host OS administrator attempts to read workload memory or extract keys. * **Impact:** Data breach, key compromise. * **Mitigation:** TEE hardware encryption prevents memory access on both GCP (TDX) and AWS (Nitro Enclave). Attestation proves hardware authenticity. * **Residual risk:** Microarchitectural side-channel attacks (speculative execution, etc.). See Residual Risks. ### T3: Malicious or Compromised Workload * **Attack:** An attacker gains control of a workload container inside the CVM or Enclave. * **Impact:** Data within that container is compromised. The attacker may try to escalate to the Guest Agent (GCP) or dstack-util (Nitro). * **Mitigation:** Container isolation within the CVM/Enclave. The Guest Agent (GCP) or dstack-util (Nitro) validates attestation before delivering keys. * **Residual risk:** If the attacker can modify the CVM/Enclave image itself, the measurements change and KMS will refuse to deliver keys. On Nitro, since encryption strategy is user-controlled, a compromised workload may misuse any keys it has already obtained. ### T4: Man-in-the-Middle / Network Attack * **Attack:** Attacker intercepts communication between CVM and KMS, or between CVM and external services. * **Impact:** Key interception, data theft, configuration tampering. * **Mitigation:** All communication uses TLS or RA-TLS. RA-TLS additionally verifies both parties' attestation. * **Residual risk:** TLS implementation vulnerabilities, certificate authority compromise. ### T5: Compromised RPC Provider * **Attack:** Attacker operates a malicious RPC node that returns false blockchain state. * **Impact:** KMS may accept unauthorized measurements or reject authorized ones. * **Mitigation:** Use multiple independent RPC providers. KMS should verify blockchain state across sources. * **Residual risk:** If all RPC providers are colluding or compromised. ### T6: Compromised or Colluding Multisig Signers * **Attack:** Multiple signers collude to push through unauthorized governance changes (e.g., register malicious measurements). * **Impact:** Unauthorized workloads receive keys from KMS. * **Mitigation:** Signature threshold (≥ 2/3) limits the number of signers that must be compromised. Timelock provides a window for detection. * **Residual risk:** If enough signers collude to meet the threshold, the system is compromised. ### T7: Covert Deployer Attack * **Attack:** A workload deployer secretly modifies the application code after deployment. * **Impact:** The workload behaves differently from what was approved. * **Mitigation:** On-chain measurement registration. Any code change produces new measurements. KMS refuses to deliver keys to unregistered measurements. * **Residual risk:** If the attacker can register the new measurements through governance without being detected. ## Security Guarantees | Guarantee | Mechanism | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | **Keys never leave verified TEE** | KMS runs in its own TEE. Keys are generated, stored, and dispatched entirely within TEE. The cloud provider cannot access them. | | **Only approved code receives keys** | Workload measurements must be registered on-chain. KMS verifies measurements before dispatching keys. | | **Governance changes are auditable** | All governance actions go through Multisig + Timelock and are recorded on-chain. Anyone can verify the history. | | **Memory is encrypted** | TEE hardware encrypts all memory. The host OS and cloud platform cannot read CVM (GCP) or Enclave (Nitro) memory. | | **Code integrity is verifiable** | Attestation proves the exact code and configuration running in the TEE. External parties can independently verify. | ## Residual Risks These are risks that the current architecture does not fully mitigate: | Risk | Description | Mitigation | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | **Hardware side-channels** | TEE hardware may be vulnerable to microarchitectural side-channel attacks (e.g., Spectre, Meltdown variants). | Keep TCB (Trusted Computing Base) firmware updated. Monitor Intel / AWS security advisories. | | **Smart contract vulnerabilities** | Bugs in DstackKms, DstackApp, or governance contracts could lead to unauthorized access. | Conduct formal smart contract audits. Use well-tested contract libraries (Safe, Timelock). | | **KMS root key** | The KMS root key is currently a single point of trust within the KMS TEE. | Future plans include MPC (Multi-Party Computation) to distribute root key generation. | | **Denial of service** | The cloud provider or host operator can shut down CVMs or Enclaves, denying service. | Use cross-region, cross-provider redundancy for high-availability deployments. | ## Security Checklist for Deployments Before going to production, verify: **TEE and Attestation:** * [ ] dstack OS image is built from audited source code * [ ] All measurements (RTMR / OS\_IMAGE\_HASH) are registered on-chain * [ ] TLS certificates are valid and properly configured **Governance:** * [ ] Multisig signers are using hardware wallets * [ ] Signature threshold is ≥ 2/3 * [ ] Timelock delay is appropriate for your risk profile **Operations:** * [ ] Multiple independent RPC providers are configured * [ ] Monitoring and alerting are set up for attestation failures and governance events * [ ] Runbook exists for common failure scenarios ## Next Steps * **[Glossary](/dstack-cloud/glossary)** — Definitions of security-related terms * **[Runbook](/dstack-cloud/runbook)** — Troubleshooting security-related issues * **[dstack Security Model](https://github.com/Phala-Network/dstack-cloud/blob/master/docs/security/security-model.md)** — Official security model document # Upgrade Procedures Source: https://docs.phala.com/dstack-cloud/upgrade Upgrade procedures for KMS images, CVM applications, and smart contracts. # Upgrade Procedures Upgrading in a TEE environment is different from normal cloud upgrades: changing your Docker image or OS version changes the measurements, which means you need to go through governance before KMS will deliver keys to the new version. This page covers the upgrade procedures for KMS, CVMs/Enclaves, and smart contracts. ## Pre-upgrade Checklist Before any upgrade: * [ ] Review the changelog for the new version * [ ] Test the upgrade in a non-production environment first * [ ] Ensure you have access to all required signers (for governance changes) * [ ] Back up current configuration (`app.json`, `docker-compose.yaml`, `.env`) * [ ] Verify monitoring and alerting is operational * [ ] Notify stakeholders of planned downtime (if applicable) * [ ] Prepare a rollback plan ## KMS Image Upgrade ### When to Upgrade * New dstack-cloud version available with security fixes * New KMS features needed * KMS vulnerability disclosed ### Procedure (GCP) 1. **Pull the new OS image:** ```bash theme={"system"} dstack-cloud pull --os-image dstack-cloud-0.7.0 ``` 2. **Stop the current KMS CVM:** ```bash theme={"system"} cd kms-prod dstack-cloud stop ``` 3. **Update the OS image reference in `app.json`:** ```json theme={"system"} { "os_image": "dstack-cloud-0.7.0" } ``` 4. **Redeploy:** ```bash theme={"system"} dstack-cloud deploy ``` 5. **Re-bootstrap if the KMS image changed:** * If the new KMS image produces different measurements, you must register them on-chain before KMS will work * Follow the bootstrap procedure in [Run a dstack-kms CVM on GCP](/dstack-cloud/run-kms-on-gcp) 6. **Verify:** ```bash theme={"system"} dstack-cloud status dstack-cloud logs --follow ``` ### Procedure (Nitro) The process is similar, but note: * The Enclave is stateless — after stopping and restarting, key material must be re-derived * Ensure the VSOCK proxy is running before deploying the new Enclave * New measurements may require on-chain registration ### Rollback 1. Stop the new KMS CVM: `dstack-cloud stop` 2. Revert `app.json` to the previous OS image version 3. Redeploy: `dstack-cloud deploy` 4. If measurements changed during upgrade, the old measurements should still be registered on-chain ## CVM / Enclave Application Upgrade ### When Upgrading Requires Governance If your application upgrade changes the measurements (different Docker images, different `docker-compose.yaml`, different dstack OS version), you must go through governance: 1. **Build and deploy the updated CVM/Enclave** (in a test environment first) 2. **Extract the new measurements:** ```bash theme={"system"} dstack-cloud status # Note the RTMR3 / OS_IMAGE_HASH ``` 3. **Register new measurements on-chain:** * See [Register Workload Measurements](/dstack-cloud/register-enclave-measurement) 4. **Wait for governance approval and timelock** 5. **Deploy to production:** ```bash theme={"system"} dstack-cloud deploy ``` ### When Upgrading Does NOT Require Governance If the upgrade only changes application logic without changing the Docker images or configuration (e.g., updating application code via a mounted volume, which is not possible in TEE), no governance is needed. In practice, most upgrades in a TEE environment require governance because the code is measured at build time. ### Procedure (GCP) 1. Update `docker-compose.yaml` with new image versions: ```yaml theme={"system"} services: web: image: my-app:v2.0.0@sha256:abcd1234... ``` 2. Redeploy: ```bash theme={"system"} dstack-cloud deploy ``` 3. Verify: ```bash theme={"system"} dstack-cloud logs --follow ``` ### Procedure (Nitro) 1. Update `docker-compose.yaml` 2. Redeploy: ```bash theme={"system"} dstack-cloud deploy ``` 3. The Enclave is rebuilt from the new image 4. Verify via VSOCK proxy ## Smart Contract Upgrade ### When to Upgrade * Security vulnerability in contract code * New governance feature needed * Bug fix in contract logic ### Procedure (Using Proxy Pattern) If contracts use an upgradeable proxy pattern: 1. **Deploy the new implementation contract:** ```bash theme={"system"} npx hardhat run scripts/deploy-kms-v2.ts --network ``` 2. **Draft a governance transaction** to call `proxy.upgradeTo(newImplementation)`: * Use the Safe web interface * The transaction must go through multisig + timelock 3. **Test the upgrade on testnet first:** * Deploy new implementation on testnet * Run through governance on testnet * Verify the upgraded contract works as expected 4. **Execute governance on mainnet:** * Collect signatures * Wait for timelock * Execute 5. **Verify:** ```bash theme={"system"} # Check the implementation address cast call "implementation()(address)" --rpc-url $RPC_URL ``` ### Rollback To rollback, follow the same procedure with the previous implementation address. ### Important Notes * The proxy admin must be the governance Safe — never an EOA * Upgrades cannot be rushed — the timelock delay applies * Plan upgrades during low-traffic windows * Have a tested rollback procedure ready before upgrading ## Coordinated Upgrade: Application + Measurements + Contracts A full-stack upgrade requires careful coordination because each step may depend on the previous one completing. The general order is: 1. Test new application + new KMS + new contracts in staging 2. Deploy new contracts to mainnet (through governance) 3. Register new measurements on-chain (through governance) 4. Wait for all governance actions to complete 5. Deploy new KMS image to production 6. Deploy new application CVMs to production 7. Verify end-to-end key delivery 8. Optionally revoke old measurements after verification ## Downtime Considerations | Component | Downtime During Upgrade | | --------------- | -------------------------------------- | | KMS (GCP) | 1-2 minutes (stop + redeploy) | | CVM (GCP) | 2-3 minutes | | CVM (Nitro) | 30 seconds to 1 minute | | Smart contracts | No downtime (upgrade is a transaction) | To minimize downtime: * Deploy new CVMs first, then switch traffic * Keep old CVMs running until new ones are verified * For KMS, consider running multiple instances for high availability ## Next Steps * **[Monitoring and Alerting](monitoring-alerting)** — Monitor upgrade impact * **[Runbook](runbook)** — Troubleshoot upgrade issues # Acknowledgement Source: https://docs.phala.com/dstack/acknowledgement Deploy confidential applications with dstack — a TEE-based infrastructure platform. ## Contributors The inspiration for this work stems from [Andrew Miller](https://github.com/amiller)’s pioneering concept of a [Docker-based P2P TEE SDK](https://collective.flashbots.net/t/dstack-speedrunning-a-p2p-confidential-vm/3876). Special acknowledgment to [Flashbots](https://github.com/flashbots) for building a community around TEE. The TEE Hacker House initiative, organized by [Flashbots](https://github.com/flashbots) and led by [Tina](https://github.com/CarboClanC), has brought together TEE builders to develop tools for TEE-Web3 integration. This collaborative journey has generated invaluable insights for advancing secure, confidential environments within Web3. Special recognition goes to the Pi-rateship builders who contributed their expertise: * Teleport: [Sxy Sun](https://github.com/sxysun) * Flashbots: [Tina](https://github.com/CarboClanC), [Mateusz](https://github.com/Ruteri), [Dmarz](https://github.com/dmarzzz), [Moe](https://github.com/MoeMahhouk) * Ithaca: [Georgios](https://github.com/gakonst) * Fabric: [@gaoist](https://x.com/gaoist) * Phala Network: [Kevin Wang](https://github.com/kvinwang), [Shelven Zhou](https://github.com/shelvenzhou) * And many more... This project cannot be built without standing on the shoulders of giants: * [konvera/meta-confidential-compute](https://github.com/konvera/meta-confidential-compute) Together, we’re shaping the future of TEE in Web3, paving the way for more secure and developer-accessible confidential computing! For a full list of the direct contributors to this repo, see [Contributors](https://github.com/Dstack-TEE/dstack/contributors) on GitHub. # Decentralized Root-of-Trust Source: https://docs.phala.com/dstack/design-documents/decentralized-root-of-trust Technical design documents and architecture details for the dstack TEE platform. Root-of-Trust (RoT) is the cornerstone of the TEE chain-of-trust. The end user verifies Remote Attestations signed by the CPU, which ultimately rely on keys derived from a set of hardware-held secrets. The hardware component that manages these root secrets, verifies firmware and applications, and issues Remote Attestations is called the Root-of-Trust. ## Hardware Root-of-Trust However, hardware RoT has several drawbacks due to the limitations of physical hardware: * **Unrecoverable TCB**: Usually, bugs in TEE can be fixed by microcode upgrades with a TCB Recovery process. However, when the secret keys in the RoT are extracted, an attacker can simulate a RoT outside of the hardware and issue arbitrary Remote Attestations using the secrets. This invalidates the TCB Recovery process. * **Vendor/Physical Chip Locked**: The app running in the TEE relies on keys derived from the RoT to seal data and maintain identities (e.g., for signing signatures and establishing TLS connections). When the RoT is lost, the corresponding data and identities are lost as well. This prevents workloads from migrating between TEEs and blocks users from building services with high availability. ## Software Root-of-Trust Given these significant drawbacks of hardware RoT, it’s imperative to explore alternative approaches that offer greater flexibility and resilience. One such approach is abstracting the RoT into software. To understand how a software RoT can address these challenges, let’s reconsider its role in the context of a TEE and explore how it can enhance security: 1. **Measurement**: Verify the authenticity of the child layer in the chain-of-trust. * In Intel SGX, RoT verifies the version of the CPU microcode, BIOS configuration, and the code loaded into the SGX enclave. 2. **Key Derivation Service**: Provide a service to derive keys for the child layer. The derived keys can be used to implement data sealing, message signing, and Remote Attestation when combined with the measurements. These two properties can also be implemented in software, offering better flexibility and resilience. A simple software RoT can be implemented as follows: * A KMS that maintains a set of secrets. * Implement **Measurement**: Verify the authenticity of a physical TEE by requesting and verifying the native Remote Attestation from the TEE (e.g., Intel SGX DCAP Remote Attestation). Once this measurement is taken, the RoT can trust that the desired software is running on the desired hardware environment with all the proper properties, usually integrity and confidentiality. * Implement **Key Derivation Service**: Once verification passes, the KMS can provide services allowing TEE applications to obtain derived keys from the RoT. The communication channel between the RoT and the TEE application can be secured using end-to-end encryption protocols like RA-TLS. Similar to a hardware RoT, the service provides capabilities to seal data, sign certificates, and implement Remote Attestation for TEE applications. Implementing a software RoT offers several advantages but also introduces new challenges that must be carefully considered: * **Pros** * **Always Recoverable TCB**: TEEs with a compromised hardware RoT can be rejected by the RoT system when measuring their Remote Attestations. In case of a hardware RoT breach, the software RoT remains secure. The software RoT can blacklist the entire product line of the affected hardware to recover the TCB of the system. * **Dynamic Measurement**: Instead of relying on static secrets stored in the hardware RoT, a software RoT can measure the physical TEE periodically and detect vulnerabilities as they occur. When a vulnerability is detected, the RoT can trigger a TEE *application migration*, remove the affected TEE from the network, and perform a key rotation to reduce the risk (with forward and backward secrecy). * **Application Migration**: The keys the application relies on (for sealing, signing, etc.) are no longer associated with the hardware RoT. Regardless of which hardware TEE the application runs on, it can always obtain the same set of derived keys from the software RoT, decoupling TEE applications from the hardware implementation. * **Cons** * **Centralization Risk**: It’s challenging to run a software RoT securely. The RoT must (1) keep the secrets confidential, (2) ensure the integrity of its code, and (3) maintain the liveness of the service. These three properties are especially challenging without relying on trust in a centralized party. ## Decentralized Root-of-Trust While a software-based RoT offers significant benefits, it introduces the risk of centralization, which compromises security. To address this, a decentralized RoT can eliminate such risks and offer a more secure, distributed alternative. We propose a decentralized RoT model. This approach distributes trust across multiple independent nodes, eliminating reliance on a single entity. A decentralized RoT serves as the foundational trust anchor for the entire TEE network, removing dependencies on specific TEE manufacturers or cloud providers. A decentralized RoT can be implemented as follows: * A software RoT using Secure Multi-party Computation (MPC) to manage and use the root secrets. MPC distributes trust across multiple independent nodes, ensuring that no single entity can compromise the RoT. Attackers must control a majority (typically two-thirds) of nodes to access the secrets. * The RoT nodes run in TEEs to (1) measure the integrity of the code and (2) provide defense-in-depth to reduce the risk of MPC collusion attacks. With reproducible builds, anyone can verify that the RoT nodes have implemented the protocol properly. * Smart contracts on the blockchain act as the governance layer for the decentralized RoT, enforcing rules around RoT node behavior, including blacklisting certain hardware from joining the TEE network. By verifying Remote Attestations on-chain, smart contracts ensure that only nodes running approved code are part of the system. * Additionally, staking mechanisms can be introduced to add cryptoeconomic security, where node operators must stake tokens as collateral. Any malicious behavior risks losing this stake, incentivizing honest participation. The decentralized architecture solves the centralization risk in software RoT because: 1. **Confidentiality**: The secrets are protected by threshold MPC, eliminating the single point of failure, with TEE providing defense-in-depth. 2. **Integrity**: The code is protected by both TEE Remote Attestation and threshold MPC for redundancy. 3. **Liveness**: The MPC setup can tolerate the failure of up to one-third of the nodes. Cryptoeconomics can be introduced to further incentivize operators to behave well. This decentralized model effectively overcomes the vulnerabilities of centralized RoT by distributing control across a secure, blockchain-governed network of MPC nodes. Additionally, the architecture reduces the complexity for users in verifying the security properties of TEE applications by providing a chain-of-trust. Users can easily verify the chain-of-trust by checking the signatures originating from the smart contract. # Key Management Service Source: https://docs.phala.com/dstack/design-documents/key-management-protocol Technical design documents and architecture details for the dstack TEE platform. In [Decentralized Root-of-Trust](/dstack/design-documents/decentralized-root-of-trust), we described our motivation to update the TEE Root-of-Trust from hardware-kept unrecoverable secret keys to a smart-contract-governed software (i.e., DeRoT). In this article, we’ll delve deeper into the details of its key management protocol. ## Threat Model A fundamental assumption in our threat model is that *TEE can be compromised*. Therefore, our primary goal is to ensure that even if the TEE hardware is compromised, the application should still function correctly with minimal data leakage. This is why we introduce Decentralized Root-of-Trust (DeRoT), a verifiable Key Management Service (KMS) for TEE governed by smart contracts, to replace the hardware-kept unrecoverable secret keys in TEE. Based on this assumption, we discuss potential threats and corresponding countermeasures. Specifically, both the confidentiality and integrity guarantees provided by TEE hardware can be compromised. * If confidentiality is compromised, encrypted memory values may be leaked, and this will further affect the sealed data on persistent storage. The countermeasures aim to: * Limit the damage scope when a single worker is compromised to only the minimal amount of data it knows. * Ensure forward and backward secrecy so that historical and future data remains safe. * If the integrity promise is broken, an adversary can instruct the TEE to produce false results. In this case, all outputs from the compromised TEE should not be trusted or used by other TEE hardware and clients. While detecting TEE vulnerabilities and compromises is beyond the scope of DeRoT, it should be able to revoke authentication and prevent dirty data from being persisted. ## Introducing DeRoT Compared to traditional key management services, DeRoT differs in two aspects: * **Access Control on Clients:** DeRoT verifies its clients to ensure they are genuine TEE hardware running valid applications. * **Verifiable Chain-of-Trust:** The DeRoT program, along with all its generated keys, must be verifiable through blockchain technology. We will explain how to verify the integrity of the DeRoT program and check whether a key is genuinely generated by DeRoT in the following sections. DeRoT is designed to enhance the functionalities of the current hardware-kept secret key in TEE in two major scenarios: 1) identifying the TEE hardware, and 2) [data sealing](https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-intel-sgx-sealing.html) (i.e., the encryption of persistent storage). Its advantages include: * **Vendor- and Hardware-Agnostic TEE Verification:** DeRoT is responsible for distributing secret keys only to valid TEE hardware and controlling the key lifecycle. Clients can delegate the Remote Attestation (RA) report verification to DeRoT, needing only to verify whether the key held by the TEE is valid. This simplifies client implementation and abstracts away the details of RA reports. Additionally, this approach can be easily extended to support different TEE hardware and get integrated into existing certificate chains. * **Key Rotation in Data Sealing with Enhanced Security Features:** With key rotation, the time window for attackers is significantly limited. Furthermore, this approach ensures forward and backward secrecy for sealed data, enhancing overall security. * **Support for Upgradable and Migratable Applications:** Since the sealed data is now encrypted with keys generated by the standalone DeRoT, storage can be entirely separated from TEE hardware. This allows upgraded applications to decrypt saved states in a controllable way, and enables dynamic addition or removal of TEE hardware without concerns about data loss. Additionally, this architecture aligns with the existing cloud paradigm, facilitating easier migration. ## Unique Challenge: Stealthy Deployer Attack Unlike in smart contracts that every operation is observable on-chain, the introduce of DeRoT enables a blockchain-TEE hybrid system where the interactions can happen both on-chain and invisibly off-chain. Also, the support for upgradable application code will introduce more risks since the benign code may be changed to dump the encrypted data in an unconscious way. Considering the following attack from a malicious application deployer: 1. An open-source benign application is deployed to TEE and trusted by its users to process the privacy data; 2. The deployer asks the DeRoT to upgrade the application code. It seems no reason to reject it since he is the owner; 3. The deployer injects malicious logic to dump all the privacy data in the new code, but doesn’t announce this to the public. Since this upgrade is between the TEE and DeRoT, it is not observable from the on-chain perspective; 4. The deployer can make another upgrade to remove the malicious logic to erase the evidence. Actually, similar attacks on the upgradable smart contracts happened before, but the difference is that such attack will not be naturally perceived when the application and the DeRoT are running off-chain. It’s worth noting that adding more access control on the application upgrade operation (e.g. enforce a multi-sig smart contract) cannot mitigate this risk. This attack motivates an important principle in our design: *every operation involving key authorization changes must be initiated on-chain to ensure observability*. This will ensure the security equivalence to existing smart contracts. ## Design ### Verifiable DeRoT and Key Chain To enable the verification on the DeRoT program integrity and correctness, its measurement will be specified in the governance smart contract. Several steps need to be followed to ensure the smart contract’s governance over the DeRoT program. 1. The DeRoT code is open-source for anyone to do code and security review to ensure the correctness of key management logic and no backdoors; 2. The executable must be produced with reproducible build so the verifier can know it matches the source code; 3. The digest of valid executable is published through on-chain governance; 4. Multiple DeRoT instances run inside TEE. Here we only rely on TEE to provide the measurement over the executable, and ensure it matches the digest on chain. Apart from RA report verification, the core logic of key management in DeRoT is quite compact. It just involves two operations: * the creation of the RootKey; * the derivation of each application key using RootKey and nonce including deployer identity and application measurement. Because of the simple logic, it can use MPC/FHE-based threshold key generation and derivation with no concern about the performance overhead. The public key of RootKey must be published on-chain to enable any party to verify the validity of application keys with it. We will show that such root-key-derivation pattern is important to enable an efficient key rotation in the following section. ### Key Derivation The key management of DeRoT follows the *Principle of Least Privilege* that the secrecy known by each entity in the system is strictly limited to what it requires to finish the jobs. We first explain the keys used during program execution, and then discuss the privileges of different entities. DeRoT turns the previous hardware-centric key management to application-centric key management. That is, each application has its own keys to seal the states and encrypt the databases, and they can be distributed to different TEE workers to enable the applications to be migrated between them. There are three entities involved in the key management protocol, and they have different knowledge of the keys: * DeRoT, which knows the RootKey and is able to generate any key; * Storage service, which can be the external storage server or service containing the data and states of the application. The storage service holds the StorageKey and can rotate it independently to keep re-encrypting the data. It’s worth noting that the DeRoT can rotate it synchronously if they share the same timestamp; * The TEE worker running the applications. Every key it holds must be distributed from DeRoT. An important feature here is that *every key received by TEE worker is kept in rotation*, so the DeRoT can passively “revoke” the worker’s access to certain program states by not to deliver the newly rotated keys. Once DeRoT rejects to deliver the latest keys, the TEE worker will not able to decrypt the local sealed data or read/write to the external storage. During program execution, the DeRoT keeps delivering the latest AppKeys to functional workers so they can correctly decrypt the program states. Although this makes DeRoT the single point of failure, we want to argue that since the DeRoT only focuses on key management, it is easier to do thorough code review to ensure its security. Further, as discussed above, the DeRoT only listens to the application deployment request from blockchain and is not accessible to others, which greatly reduces the attack interface. The RootKey is only known by DeRoT and a Key Derivation Function (KDF) is used to generate all the other keys. The AppRootKey above is merely a logical entity and can be skipped in the actual key generate. For example, $SealingKey_{epoch}=KDF(RootKey,(deployer\_id,app\_hash,nonce,"seal",epoch))$ ### Key Rotation There are two rotation variables above: the RootKey itself and by changing `epoch` during key derivation. The rotation of the RootKey will immediately changes all the keys in the system, while the interval of updating `epoch` can be configured independently for different storage services so they can be rotated in different frequencies based on the requirements. The update of the applications, which will change the `app_hash` above, can be regarded as a special key rotation for the application. #### Forward and Backward Secrecy Ensurance in TEE Workers On one hand, the TEE worker runtime will only keep the latest AppKeys and remove all the used keys from its memory to preserve the forward secrecy; on the other hand, once a worker is considered compromised, the DeRoT will stop delivered the rotated new keys, so the worker will not be able to get further access to the following code and state updates of the program, preserving the backward secrecy. # TEE-Controlled Domain Certificates Source: https://docs.phala.com/dstack/design-documents/tee-controlled-domain-certificates Technical design documents and architecture details for the dstack TEE platform. ## The Problem In this article, we answer a practical question: *How to ensure that the certificate for a specific domain is controlled and can only be controlled by a TEE program (in our case, the DeRoT).* Once we can achieve this, DeRoT, as the root of trust, can then sign more certificates for every application it has verified. This will be an important building block to enable users to seamlessly verify whether an application is actually running within valid TEEs using the existing TLS-based protocols (like HTTPS). The DeRoT-controlled certificate can not only prove its identity under this paradigm, but extend the chain of trust from the blockchain ecosystem to any website. Given the key management protocol mentioned above, the DeRoT becomes the root-of-trust of the whole system as it is responsible for verifying all the TEE hardware and managing the data encryption. Its identity can be verified with the *RootPubKey* published on-chain. If the DeRoT can use this *RootKey* as the certificate’s private key, then it owns a TLS certificate which is verifiable on chain and can issue certificates to valid TEE applications running in verified hardware. Since the DeRoT is running inside TEE, it is not hard to generate a certificate that never leaves TEE. The real challenge here is how to ensure this certificate is tightly bound with a domain and cannot be replaced, given that the DNS servers are centralized services that cannot be controlled by smart contract or TEE application. ## Building Blocks Here we introduce two extra building blocks: * Certification Authority Authorization (CAA) in DNS record. This is used to specify which certificate authorities (CAs) are permitted to issue certificates for their domain. It prevents unauthorized issuance of SSL/TLS certificates. With CAA, we can ensure that the DeRoT-owned certificate is the only valid one for its domain. * Certificate Transparency (CT) is an Internet security standard for monitoring and auditing the issuance of digital certificates. Although the CAA ensures the DeRoT ownership over the certificate, the CAA itself, which can be configured in centralized service providers, may be changed. The CT logs can be used for two purposes: 1) to verify whether the CAA is really set to the expected certificate, and 2) to enable users to detect and reject unauthorized CAA modifications. ## System Design Here we introduce the certificate managing module in the DeRoT serving as a reverse proxy for applications running in the network. It’s worth noting that as a part of DeRoT, it runs in TEE and is also governed by the smart contract. ### DeRoT Certificate Registration The DeRoT will generate a x.509 certificate using the *RootKey* as the private key. Compared with other TLS certificates, the only difference is that the *RootPubKey* is available on-chain, so any users can verify that this certificate is really owned by DeRoT. This certificate is sent to CA (e.g. Let’s Encrypt) for signing, then a DNS CAA record is registered. In the example above, the CAA record means “only [letsencrypt.org](http://letsencrypt.org/) is allowed to issue certificates for [derot.com](http://derot.com/)”. We further utilize CAA records to restrict the authentication methods, as some of them are not secure. For example, when verifying with HTTPS, Let’s Encrypt only requires you to place a specific text file at **`http:///verify.txt`** to confirm domain ownership. However, Let’s Encrypt cannot verify whether the endpoint is within a TEE. To address this, we limit the authentication method to API access only, which can be fully controlled by a TEE. This ensures that there is no possibility of issuing the certificate to an incorrect target. ### TEE App Registration Although a senior user can request to verify the Remote Attestation report from a TEE app directly, most of the users could delegate the verification to DeRoT by verifying its certificate. To get reverse proxy from DeRoT, a TEE app needs to register itself to DeRoT, during which DeRoT will do the RA verification on it to ensure the DeRoT policy is followed. ### Protection against Unauthorized Certificate Changes Since the DNS servers are centralized services which cannot be controlled by smart contracts, there is risk that the attackers will change the DNS record to break the limitation on the certificate owner. Although we cannot technically prevent this, we can monitor the unexpected DNS record updates and notify the clients to avoid trusting invalid certificates. # Whitepaper Source: https://docs.phala.com/dstack/design-documents/whitepaper Technical design documents and architecture details for the dstack TEE platform. ## dstack Whitepaper ## Call for Next Generation Zero Trust Platform The Web3 ecosystem represents a paradigm shift in digital trust models, fundamentally transforming the relationship between users and application developers. Unlike traditional systems, Web3 empowers users with unprecedented control over their data through the principle of “Code is Law” — once deployed, code operates autonomously beyond external control. This transformation demands robust execution platforms that maintain confidentiality and integrity without relying on centralized trust authorities. The contrast between Web2 and Web3 architectures in Figure 1 illustrates this fundamental shift.\ In Web2 systems, developers maintain complete control through proprietary applications and centralized servers, retaining administrative privileges to access and modify user data at will. Web3’s smart contracts, however, represent the first significant advancement toward true trustlessness by executing programs on decentralized platforms (blockchains). Once deployed, administrators cannot affect program execution, and lifecycle management (e.g., code updates) follows coded agreements through democratic governance mechanisms like multi-signature wallets and Decentralized Autonomous Organizations (DAOs). This evolving trust model extends beyond basic applications to mission-critical scenarios such as AI model training, where data providers and model developers often have competing interests, and autonomous AI agents designed to operate independently of human intervention. In these contexts, Zero Trust platforms become essential to host applications and enforce verification mechanisms that appropriately constrain developer authority. A Zero Trust platform must enforce the following key principles: * **Code is Law**: This manifests in two dimensions: (1) the application logic, once deployed, cannot be changed unexpectedly; and (2) the lifecycle management of code — including deployment, upgrades, and deletion — follows predefined governance rules. * **Censorship Resistance**: User data remains beyond the control of any single entity, with data\ availability protected against Denial-of-Service attacks. * **Full Chain-of-Trust**: When users interact with applications on a Zero Trust platform, they\ must be able to verify all aspects of that application, including network setup, application identity and code logic, underlying hardware, and execution environment. * **Assume Breach**: The platform operates under the assumption that compromises will occur,\ implementing mechanisms for damage containment, rapid recovery, and minimization of data\ exposure during security incidents. ## From TEE to Zero Trust Platform Confidential Computing, particularly through Trusted Execution Environments (TEEs), has emerged as a promising foundation for extending blockchain capabilities to support complex computational tasks with confidentiality guarantees. Recent advances in VM-level TEE solutions, including Intel TDX and AMD SEV, have demonstrated substantial performance improvements and developer-friendly interfaces, making them compelling candidates for Web3 deployment. The integration of GPUs within TEEs now enables verifiable execution of both Large Language Models and AI agents, providing cryptographic guarantees of computational integrity. However, there exists a significant gap between raw TEE technology and the requirements of a true Zero Trust platform. Current TEE implementations face several critical limitations when applied to Web3 contexts: * Security Reliability: Recent vulnerability discoveries in TEE systems have raised significant concerns about their effectiveness as trust anchors. The persistence of side-channel attacks and micro-architectural vulnerabilities undermines confidence in TEE security guarantees, particularly for high-value Web3 applications where a single compromise could lead to catastrophic data exposure. * Censorship Vulnerability: Conventional TEE encryption schemas bind encryption keys to specific hardware instances, creating single points of failure that contradict Web3’s requirement\ for censorship resistance. This architectural limitation means that data availability cannot be\ guaranteed if specific hardware instances are compromised or become unavailable. * Vendor Dependencies: Current trust models heavily rely on centralized hardware manufacturers and verification processes, introducing risks of vendor lock-in and potential censorship vectors. This centralization contradicts Web3’s core principles of decentralization and trustlessness, creating systemic risks for the ecosystem. * Incomplete Verifiability: While TEEs provide Remote Attestation capabilities to verify application and hardware identity, they fail to deliver the comprehensive chain of trust required in Zero Trust environments. Users need guarantees that programs process their data as expected and cannot be modified without proper authorization — capabilities beyond current attestation mechanisms. To bridge this gap between raw TEE technology and Zero Trust requirements, we present [dstack](https://github.com/Dstack-TEE/dstack),\ a comprehensive framework for confidential containers that introduces three key innovations: * **Portable Confidential Container**: Our architecture enables seamless migration of confidential workloads across different TEE instances and hardware vendors, significantly reducing vendor lock-in risks while maintaining robust security guarantees. This innovation directly addresses the censorship vulnerability and vendor dependency limitations by providing hardware abstraction and state continuity across heterogeneous TEE environments. * **Decentralized Code Management**: We implement a comprehensive governance framework\ that leverages smart contracts for transparent and decentralized management of TEE applications. This system ensures verifiable deployment and upgrade processes, enforcing the “Code is Law” principle by binding application lifecycles to on-chain governance decisions. This innovation addresses the incomplete verifiability limitation by creating an immutable audit trail of application changes. * **Verifiable Domain Management**: Our novel approach to certificate management ensures that\ confidential containers can exclusively control domains and provide native HTTPS support to\ TEE applications without relying on centralized authorities. This system serves as a middleware between TEE applications and existing TLS-based networks, enabling users to establish end-to-end encrypted channels to TEE applications without modifications to client software. ## dstack Components These innovations are implemented through a cohesive system architecture comprising three core components: * *dstack-os*: A hardware abstraction layer with a minimized operating system image that eliminates differences in underlying TEE hardware while reducing the attack surface. This component provides a consistent, secure runtime environment across diverse TEE implementations. * *dstack-kms*: A blockchain-controlled key management service that replaces hardware-based encryption schemas with an independent service for generating and managing secret keys. This component enables secure data migration across TEE instances and supports key rotation to provide forward and backward data secrecy, mitigating the impact of potential TEE compromises. * *dstack-ingress* and *dstack-gateway*: Complementary systems providing TEE-controlled domain management through different approaches — dstack-ingress enables applications to specify custom domains with minimal integration requirements, while dstack-gateway offers pre-registered wildcard domains requiring no application code changes. By addressing the critical limitations of current TEE implementations, dstack establishes a comprehensive framework that transforms raw TEE technology into a true Zero Trust platform aligned with Web3’s fundamental principles. Our approach not only enhances the security and reliability of confidential computing but also provides a practical path toward truly decentralized, censorship-resistant infrastructure for next-generation applications. The remainder of this article details the technical design of dstack, evaluates its security characteristics, and demonstrates how our innovations collectively create a Zero Trust platform\ that maintains the performance advantages of VM-level TEE solutions while addressing their inherent limitations in Web3 contexts. ## System Design ### Portable Confidential Container Containerization has revolutionized cloud-native application deployment by providing unparalleled portability and scalability through effective data separation. These capabilities are particularly critical for Trusted Execution Environment (TEE) applications, as they directly address the inherent limitations above: * **Security Reliability**: Containerized applications can be seamlessly migrated from compromised TEE hardware to secure instances, preserving code integrity. Combined with our secure data migration mechanism that ensures both forward and backward secrecy, this approach significantly mitigates privacy leakage risks even when individual TEE instances are compromised. * **Censorship Resistance**: The architecture’s inherent scalability provides robust defense against Denial-of-Service attacks. Additionally, by decoupling data storage from execution environments, established data availability solutions can be readily integrated to ensure persistent access to application data. * **Vendor Independence**: Our hardware abstraction layer enables confidential containers to\ operate across diverse TEE hardware platforms without modification, eliminating vendor lock-in while facilitating rapid adoption of emerging TEE technologies as they become available. However, implementing truly portable confidential containers presents substantial technical challenges that differentiate them from conventional containerization. Two fundamental obstacles must be overcome: * First, existing TEE implementations generate data sealing keys derived from hardware-bound root keys unique to each TEE instance. This architectural design creates an inherent barrier to portability — data encrypted by one TEE instance cannot be decrypted by any other, even when running identical application code. Prior research has largely circumvented this limitation by restricting TEEs to stateless applications, but this approach severely constrains their utility in complex real-world scenarios. * Second, TEE implementations from different vendors impose disparate specifications for deployable programs. This fragmentation forces developers to create and maintain multiple artifacts across platforms, each requiring separate (and expensive) code reviews to verify functional equivalence — a process that undermines the trust guarantees central to Web3 applications. To address these challenges comprehensively, we introduce two complementary components: dstack-kms and dstack-os. #### dstack-kms: Blockchain-Controlled Secret Derivation dstack-kms is a secret derivation service that fundamentally transforms how encryption keys are managed in confidential computing environments. Its primary function is to generate a unique and stable secret (the *Root Key*) for each application based on its code and configurations. This Root Key serves as the foundation for deriving additional application-specific secrets used for data encryption and verifiable random number generation. Unlike hardware-based encryption approaches, dstack-kms deliberately decouples key generation from specific TEE hardware instances. This architectural decision enables the critical capability of encrypted data migration between different TEE instances, with decryption possible after proper authorization. This feature is essential for achieving true censorship resistance. Given its position as the root-of-trust for the entire system, the verifiability and availability of\ dstack-kms are paramount. Unlike previous approaches that assume TEE infallibility, our threat\ model explicitly acknowledges that ***TEEs can be compromised***. To address this reality, dstack-kms implements comprehensive key rotation capabilities that provide both backward and forward secrecy for application data. This ensures that even if a specific TEE hardware instance is compromised, applications can be migrated with minimal data exposure. To understand the limitations of conventional approaches, this figure illustrates the key derivation hierarchy in Intel SGX. This design uses a hardware-bound key to both identify the TEE instance (for Remote Attestation) and generate encryption keys for data sealing. While this approach effectively establishes hardware identity, it fundamentally undermines censorship resistance: * The root key cannot be updated, meaning a compromised TEE with a leaked root key remains\ permanently vulnerable even after patching the underlying vulnerability. * Encrypted data becomes intrinsically bound to specific TEE hardware instances, creating single points of failure that contradict Web3’s censorship resistance requirements. Our dstack-kms addresses these limitations by delegating secret generation to an independent\ service that ensures encrypted data remains portable across TEE instances. As illustrated in figure above, dstack-kms consists of an on-chain governance smart contract coupled with an off-chain peer-to-peer network where the actual secret derivation service operates. Every off-chain node here runs a light client to continuously synchronize with the governance contract. To ensure comprehensive verifiability of the dstack-kms program’s integrity and correctness, we implement a multi-stage verification process: 1. The dstack-kms codebase is fully open-sourced, enabling thorough code and security reviews to verify key management logic and ensure the absence of backdoors. 2. Executable binaries are produced with Github Actions, allowing verifiers to confirm that the\ runtime code matches the reviewed source code. 3. The cryptographic digest of valid executables is published through the governance smart contract, creating an immutable reference for verification. 4. Each service node operates within its own TEE instance. Critically, we rely on the TEE solely for measuring the executable and ensuring it matches the on-chain digest — not for the fundamental security of the key management process. This verification framework ensures that dstack-kms remains trustworthy to any third-party verifier while avoiding dependency on the absolute security of any single TEE instance. The core logic of each dstack-kms service node is intentionally minimalist, focusing exclusively on the generation and secure distribution of the root key, along with deriving encryption keys for TEE applications. This simplicity enables the implementation of Multi-Party Computation (MPC) based threshold key generation and derivation without concerning performance overhead. We have developed two complementary implementations: * **Simple Duplication**: The first node in the P2P network generates a cryptographically secure\ random number as the root key, then shares it with other nodes after verifying their attestation reports. All nodes maintain identical copies of the root key and derive consistent encryption keys for applications. This approach maximizes availability — as long as at least one node remains operational, the root key can be recovered. * **MPC-Based Key Generation**: While simple duplication provides robust availability, it creates vulnerability to single-node compromises. Our MPC implementation uses Shamir’s Secret\ Sharing scheme to distribute the root key across multiple nodes, ensuring that compromising\ any individual node (or even up to *t-1* nodes, where *t* is the configured threshold) does not\ expose the root key. This approach also enables advanced features like key rotation without\ requiring application reconfiguration. **Key Derivation** dstack-kms implements a comprehensive key derivation framework to generate application-specific keys from the root key: * **Application CA Key**: Derived from the root CA key using the application’s unique identifier\ registered on-chain. * **Disk Encryption Key**: Derived using a combination of the application identifier and instance\ identifier, enabling secure storage with portability. * **Environment Encryption Key**: Derived using the application identifier alone, allowing secure\ environment variable management across instances. * **ECDSA Key**: Derived from the root ECDSA key, providing applications with consistent cryptographic identity for signing operations. Each derived key maintains cryptographic isolation while ensuring consistency across different TEE instances running the same application. **Key Rotation** A critical security feature of dstack-kms is its support for key rotation, which significantly limits the exposure window during potential attacks while ensuring both forward and backward secrecy for sealed data. Key rotation procedures are initiated exclusively through the governance smart\ contract, ensuring transparency and auditability. We discuss two distinct rotation mechanisms: * **Root Key Share Rotation**: Our MPC-based implementation enables rotation of individual\ key shares without modifying the root key itself. This process enhances root key security by\ regularly updating the distribution mechanism while remaining transparent to applications, as\ all derived application keys remain unchanged. * **Root Key Rotation**: In scenarios where the root key may have been compromised, we support\ complete root key rotation. This process generates an entirely new root key and implements a\ controlled handover period during which both old and new keys remain valid. This transition\ window allows applications to re-encrypt their data with keys derived from the new root key\ before the old root key is permanently destroyed, minimizing service disruption while enhancing security. Through these mechanisms, dstack-kms provides the foundation for truly portable confidential containers by decoupling encryption from hardware dependencies while maintaining robust security guarantees. #### dstack-os: Hardware Abstraction Layer dstack-os provides a comprehensive hardware abstraction layer with a minimized operating system image that bridges the gap between application containers and VM-level TEE implementations. This layer enables applications to deploy across diverse TEE environments without code modifications while maintaining security integrity. As a shared component across all TEE applications, dstack-os must be intrinsically secure and verifiable, with no administrative backdoors that could compromise program code or user data. VM-level TEE solutions typically require developers to provide complete system images encompassing bootloader, kernel, operating system, and application components. This requirement not only increases development complexity but also introduces potential security vulnerabilities through misconfigured system components. A secure and properly configured operating system environment fundamentally determines the security baseline for all TEE applications built upon it. This figure illustrates how dstack-os establishes a secure and verifiable startup chain from the TEE hardware foundation through to user-provided container images. Our modular design ensures that the majority of the system can be reused across different TEE hardware implementations, substantially reducing the effort required for code review and security auditing. Combined with reproducible build processes, this approach enables comprehensive verification of dstack-os with minimal overhead while supporting multiple TEE hardware platforms. Using Intel TDX as an exemplar (with similar procedures applicable to other TEE implementations like AMD SEV), dstack-os consists of the following components, presented in boot sequence\ order. During this process, each component measures the next one and records these measurements in hardware registers, which are ultimately incorporated into the TEE attestation report: * **Hypervisor**: When application deployment initiates, the underlying hypervisor (the TDX module) loads the Open Virtual Machine Firmware (OVMF), enabling UEFI support for confidential virtual machines. The TDX module measures the OVMF code and records its cryptographic digest in the MRTD register, with the hypervisor’s integrity guaranteed by the TEE hardware\ itself. * **Open Virtual Machine Firmware (OVMF)**: The OVMF configures virtual machine hardware specifications (CPU, memory, and device configurations) and records the configuration measurement to the RTMR0 register. It then loads the Linux kernel and records the kernel\ image measurement to the RTMR1 register. * **Kernel**: The kernel first loads an initial ramdisk (initrd) — a temporary in-memory filesystem\ containing minimal command-line tools and disk encryption libraries. The initrd mounts the\ root filesystem and then the kernel will call the OVMF to record a full measurement (including\ kernel, initrd and RootFs to load) in the RTMR2 register. * **Root Filesystem**: The root filesystem (RootFs) is a read-only environment containing essential system libraries and tools. It manages the entire lifecycle of deployed applications, measuring application images along with their associated key management system and storing these measurements in the RTMR3 register. The RootFs serves two primary functions: (1) providing the runtime environment for user-level container images, and (2) managing application data encryption. It interfaces with dstack-kms to acquire encryption secrets and implements Linux Unified Key Setup (LUKS) to encrypt application data in isolated volumes. Its data integrity is ensured through dm-verity, a Merkle-tree-based block-level verification tool. Finally, the RootFs configures network ingress for the application and initiates execution. Rather than adapting existing system images, we built dstack-os from scratch to ensure only essential components are included in both the initrd and RootFs. Generally, dstack-os includes only basic utilities (busybox), filesystem support, and container runtime components. By implementing Docker support while delegating orchestration to the application level, we maintain a minimal base image footprint. To ensure comprehensive verifiability, dstack-os is fully open-sourced and supports reproducible\ builds, allowing anyone to review the code and generate identical artifacts for verification. This\ approach provides significant benefits for applications requiring security reviews, as evaluators need only review the application code rather than the entire system image. **Data Backup and Defense against Rollback Attacks** A critical challenge in confidential computing is ensuring data durability and integrity, particularly against rollback attacks where adversaries attempt to revert applications to previous states to bypass security checks or reuse expired secrets. Our architecture provides comprehensive secure data backup support and implements robust anti-rollback mechanisms that application developers can integrate: * **Secure Backup**: Application data is encrypted using keys derived from dstack-kms and can be backed up to external storage providers. Since encryption keys are not bound to specific TEE instances, data can be restored on any authorized TEE, supporting true portability and disaster recovery. * **Monotonic Counters**: To prevent rollback attacks, developers can implement monotonic counters that increment with each critical data update, incorporating the counter value into the encrypted data. During restoration, applications verify that the counter value equals or exceeds the last known value, preventing replay of outdated snapshots. By combining these techniques, dstack ensures both the recoverability and integrity of confidential data across portable container migrations and backup scenarios. Together, dstack-kms and dstack-os provide the foundational infrastructure for truly portable confidential containers that maintain security and integrity across heterogeneous TEE environments while eliminating vendor lock-in and supporting comprehensive verifiability. ### Decentralized Code Management A core principle of Zero Trust platforms is the enforcement of “Code is Law”, ensuring that application code operates as intended without unexpected modifications. While TEE hardware provides code integrity guarantees during execution, it cannot independently prevent administrators from deploying malicious code or making unauthorized modifications. Conversely, smart contracts have established robust methodologies for implementing programmable governance over code lifecycle management — such as multi-signature requirements that prevent single-actor manipulation. Our decentralized code management framework bridges these two worlds by placing TEE application governance under smart contract control, creating a transparent and auditable system for application deployment and updates that aligns with Web3 principles. Our architecture consists of two complementary components: on-chain governance smart contracts and off-chain TEE execution environments connected through dstack-kms. This design establishes the on-chain contracts as the definitive root-of-trust for the entire system. Every\ operation — from initial deployment to code updates — must first be initiated and authorized through the governance contracts before off-chain TEE instances can execute these operations. The governance framework implements a two-tier structure: * **KmsAuth Contract**: Serves as the global authority controlling dstack-kms operations. This\ contract maintains the registry of authorized dstack-os images, applications and their governance parameters, enforcing system-wide policies for TEE deployment. By controlling whether dstack-kms shares application secrets with specific TEE instances, the KmsAuth contract effectively determines whether applications can decrypt their data, thus controlling their complete lifecycle. * **AppAuth Contracts**: Individual governance contracts deployed for each application that define application-specific management rules. These contracts specify permissible code versions (through cryptographic hashes), authorized TEE instance identities, and upgrade approval requirements. This modular approach enables customized governance models ranging from traditional multi-signature schemes to complex DAO voting mechanisms, allowing each application to implement governance appropriate to its requirements. The enforcement mechanism operates through dstack-kms, which will only provide application\ secrets to TEE instances running code versions explicitly authorized by the governance contracts. This creates a cryptographically enforced governance system where unauthorized code versions cannot access application data, regardless of administrative privileges. To illustrate this mechanism, consider a code update process for an application using a multi-\ signature AppAuth contract: 1. Developers publish the new code version and submit its cryptographic hash to the AppAuth\ contract. 2. The contract initiates the approval process, requiring signatures from designated key holders\ (e.g., core developers, security auditors, and community representatives). 3. Each signature is recorded on-chain, creating an immutable audit trail of the approval process. 4. Once the required signature threshold is reached, the AppAuth contract updates its registry of authorized code versions. 5. The KmsAuth contract calls the corresponding AppAuth contract, and updates its authorization records accordingly. 6. dstack-kms, which continuously synchronizes with the KmsAuth contract, begins providing application secrets to TEE instances running the newly approved code version. 7. TEE instances can now deploy the new code with full access to encrypted application data. This process ensures that every code change follows predefined governance rules with complete transparency. The on-chain approval records create an immutable audit trail that allows users to verify the entire history of application changes, while the cryptographic enforcement through dstack-kms guarantees that only properly authorized code can access application data. By combining the code integrity guarantees of TEEs with the transparent governance capabilities\ of smart contracts, our decentralized code management framework ensures that applications truly embody the ”Code is Law” principle. Every aspect of the application lifecycle — from initial deployment to version updates and potential retirement — follows predefined, transparent rules that cannot be circumvented by any single entity, including the platform operators themselves. ### Verifiable Domain Management A complete Zero Trust platform must enable seamless verification for users across both Web3 and traditional Web2 environments. Our Verifiable Domain Management system completes the chain-of-trust by allowing standard Web browsers to cryptographically verify TEE applications without requiring any client-side modifications or specialized knowledge. This capability is critical for mainstream adoption, as it allows users to interact with confidential\ applications using familiar tools while maintaining the same security guarantees provided by the underlying TEE infrastructure. We implement this through two complementary components — dstack-ingress and dstack-gateway — both leveraging our novel **Zero Trust TLS** (ZT-TLS) protocol. The fundamental innovation in ZT-TLS is establishing cryptographic proof that a specific TLS\ certificate (and therefore a domain) is exclusively controlled by a verified TEE application. This\ creates a verification chain extending from the blockchain to standard HTTPS connections, enabling users to confirm application authenticity through familiar security indicators in their browsers. #### Zero Trust TLS Protocol The ZT-TLS protocol addresses a critical challenge: how to cryptographically bind a domain to a\ specific TEE application in a way that prevents unauthorized certificate issuance or domain hijacking. Our solution integrates three key components: * **TEE-Generated Certificates**: The TEE application (specifically dstack-gateway) setups all\ the information for the certificate, including Automated Certificate Management Environment\ (ACME), the secret key and Certificate Signing Request (CSR), and request the CA (e.g. letsencrypt) to issue its TLS certificate using application secrets derived from dstack-kms. This ensures the private key never exists outside the TEE environment and creates a cryptographic link between the on-chain governance and the TLS certificate. * **Certificate Authority Authorization (CAA)**: We leverage DNS CAA records to restrict\ that only TEE-controlled ACME is valid, and which certificate authorities can issue certificates\ for managed domains. By configuring CAA records to only authorize certificates from specific\ issuers (which verify TEE attestation), we prevent unauthorized certificate issuance through\ alternative validation methods. * **Certificate Transparency (CT) Monitoring**: To protect against unauthorized DNS record\ modifications, we implement continuous monitoring of Certificate Transparency logs. This allows detection of any unauthorized certificates and provides an immutable audit trail of certificate issuance history. It is worth noting that such monitoring service can be implemented by anyone to be truly decentralized. #### Implementation Components We provide two complementary implementations to accommodate different application requirements: * **dstack-gateway**: A fully managed reverse proxy running within a TEE that provides immediate domain verification with zero application code changes. Applications register with dstack-gateway through a simple API call, during which dstack-gateway verifies the application’s remote attestation report. Once verified, applications receive a subdomain under a pre-registered wildcard domain (e.g., [app-id.dstack.com](http://app-id.dstack.com/)), with all TLS termination and certificate management handled automatically. * **dstack-ingress**: A more flexible solution for applications requiring custom domain names.\ dstack-ingress provides libraries and services that enable applications to manage their own TLS certificates while maintaining the same verification guarantees. This approach requires minimal integration work while supporting application-specific domain requirements. When the customized domains are enabled, our gateway will fallback to TLS passthrough mode, where the TLS connection is directly established between the client and the application. Both implementations ensure that certificates are cryptographically bound to verified TEE environments, with the verification chain extending from the blockchain governance contracts through to the TLS certificates presented to users’ browsers. #### Security Guarantees and Protections Our Verifiable Domain Management system provides several critical security guarantees: * **Certificate Provenance Verification**: Users can verify that the TLS certificate presented by\ an application was issued to a legitimate TEE environment with the ZT-TLS protocol above. * **Domain Binding Integrity**: The combination of CAA records and Certificate Transparency\ monitoring ensures that domains remain bound to their designated TEE applications, preventing unauthorized certificate issuance. * **Tamper-Evident Certificate Changes**: Any unauthorized attempts to modify domain configurations or issue alternative certificates are detected through Certificate Transparency monitoring, with alerts propagated to clients. While we cannot technically prevent all attacks against centralized DNS infrastructure, our multi-layered approach creates a tamper-evident system where unauthorized changes are quickly detected and rejected by client verification processes. This approach aligns with the ”Assume Breach” principle by implementing defense-in-depth strategies that maintain security even when individual components are compromised. By extending the verification chain from blockchain governance through TEE attestation to standard TLS certificates, our Verifiable Domain Management system enables seamless integration of Zero Trust applications into existing web infrastructure. This creates a practical path for mainstream adoption without requiring specialized knowledge or tools from end users, while maintaining the comprehensive security guarantees that define true Zero Trust platforms. # FAQs Source: https://docs.phala.com/dstack/faqs Deploy confidential applications with dstack — a TEE-based infrastructure platform. ## Does dstack support GCP / Azure dstack is open for PRs to add that support. Currently it focuses on bare metal because it offers the most fine-grained control and access to vanilla TDX. ## How does KMS key generation work and how can it be modified? The key generation logic is implemented in the onboard service:\ Reference: [https://github.com/Dstack-TEE/dstack/blob/master/kms/src/onboard\\\_service.rs#L50](https://github.com/Dstack-TEE/dstack/blob/master/kms/src/onboard\\_service.rs#L50) ## How does on-chain KMS work and how can I customize its governance? The KMS contract allows for customizable ownership and governance: * During deployment, you can specify an owner * After deployment, ownership can be transferred using transferOwnership function\ Reference: [https://github.com/Dstack-TEE/dstack/blob/master/kms/auth-eth/hardhat.config.ts#L96](https://github.com/Dstack-TEE/dstack/blob/master/kms/auth-eth/hardhat.config.ts#L96) ## Where can I find KMS deployment instructions? Complete deployment documentation is available here: Reference: [https://github.com/Dstack-TEE/dstack/blob/master/docs/deployment.md](https://github.com/Dstack-TEE/dstack/blob/master/docs/deployment.md) ## How does the current data encryption system work? The system uses Linux's built-in LUKS (Linux Unified Key Setup) for disk encryption:\ Reference: [https://github.com/Dstack-TEE/dstack/blob/master/tdxctl/src/fde\\\_setup.rs#L437-L442](https://github.com/Dstack-TEE/dstack/blob/master/tdxctl/src/fde\\_setup.rs#L437-L442) ## Where is the deployment function located in the Dstack-TEE codebase, and can it be integrated with custom tools? The deployment logic for Dstack-TEE is part of its Rust-based implementation, there are multiple services involved in the deployment pipeline. The [Dstack-TEE GitHub repository](https://github.com/Dstack-TEE/dstack) contains the core services (e.g., vmm, gateway, kms, meta-dstack). ## Can I run a Docker image directly in a TEE without setting up all dstack components? Yes, you can run a Docker image in a TEE using the dstack SDK without setting up all components, but with limitations. The SDK provides a base VM image that runs a minimal VM containing your Docker container. The essential components are kms (key management service) and vmm (TEE runtime), which must be included. Optional components like dstack-gateway (for TLS support) can be skipped if you don’t need features like encrypted communication. Configure the SDK with your Docker image and the required components as per the documentation. # Getting Started Source: https://docs.phala.com/dstack/getting-started Set up and deploy your first confidential application using dstack TEE infrastructure. **WARNING** **This tutorial assumes your have access to a compatible Intel TDX Server ready to be configured with dstack. Check the** [**Hardware Requirements**](/dstack/hardware-requirements) **for more information or skip setting everything up from scratch and** [**Sign-up for Phala Cloud Account**](/phala-cloud/getting-started/sign-up-for-cloud-account)**.** For the latest information, please refer to the [dstack README](https://github.com/Dstack-TEE/dstack/blob/master/README.mdx). ## dstack dstack is a **developer friendly** and **security first** SDK to simplify the deployment of arbitrary Docker-based apps into TEE. Main features: * 🔒 Deploy containerized apps securely in TEE in minutes * 🛠️ Use familiar tools - just write a docker-compose.yaml * 🔑 Safely manage secrets and sensitive data * 📡 Expose services via built-in TLS termination dstack is community driven. Open sourced and built by [Kevin Wang](https://github.com/kvinwang) and many others from [Phala Network](https://github.com/Phala-Network), inspired by [Andrew Miller](https://github.com/amiller) (Flashbots & Teleport), and contributed by [Nethermind](https://github.com/NethermindEth/nethermind) and [many others](#contributors). ## Overview Components in dstack: * `dstack-vmm`: A service running in bare TDX host to manage CVMs * `dstack-gateway`: A reverse proxy to forward TLS connections to CVMs * `dstack-kms`: A KMS server to generate keys for CVMs * `dstack-guest-agent`: A service running in CVM to serve containers' key derivation and attestation requests * `meta-dstack`: A Yocto meta layer to build CVM guest images The overall architecture is shown below: ## Directory structure ``` dstack/ kms/ A prototype KMS server guest-agent/ A service running in CVM to serve containers' key derivation and attestation requests. dstack-util/ A CLI tool in the guest setting up the system, getting TDX quote, extending RTMR, generating cert for RA-TLS, etc. vmm/ A service running in bare TDX host to manage CVMs gateway/ A reverse proxy to forward TLS connections to CVMs certbot/ A tool to automatically obtain and renew TLS certificates for dstack-gateway ra-rpc/ RA-TLS support for pRPC ra-tls/ RA-TLS support library tdx-attest/ Guest library for getting TDX quote and extending RTMR ``` ## Build and play locally ### Prerequisites * A bare metal TDX server setup following [canonical/tdx](https://github.com/canonical/tdx) * Public IPv4 address assigned to the machine * At least 16GB RAM, 100GB free disk space. * A domain with DNS access if you want to set up `dstack-gateway` for Zero Trust HTTPS ### Install dependencies ```bash theme={"system"} # for Ubuntu 24.04 sudo apt install -y build-essential chrpath diffstat lz4 wireguard-tools xorriso # install rust curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` ### Build and run ```bash theme={"system"} git clone https://github.com/Dstack-TEE/meta-dstack.git --recursive cd meta-dstack/ mkdir build cd build ../build.sh hostcfg # This outputs the following message: # Config file ../build-config.sh created, please edit it to configure the build vim ./build-config.sh ``` Review and customize the `build-config.sh` configuration file according to your environment requirements. The file contains network ports, domain settings, and other important parameters. Once configured, run the build script again to generate the necessary artifacts: ```bash theme={"system"} ../build.sh hostcfg # If everything is okay, you should see the built artifacts in the `build` directory. $ ls certs images dstack-kms kms.toml run dstack-vmm vmm.toml dstack-gateway gateway.toml ``` ### Download guest image or build guest image from source ```bash theme={"system"} # This will download the guest image from the release page. ../build.sh dl 0.5.2 ``` Or build guest image from source: ```bash theme={"system"} # This will build the guest image from source using the yocto meta layer. This will take a while. ../build.sh guest ``` Now you can open 3 terminals to start the components: 1. Run `./dstack-kms -c kms.toml` 2. Run `sudo ./dstack-gateway -c gateway.toml` 3. Run `./dstack-vmm -c vmm.toml` ### Deploy an App Open the dstack-vmm webpage [http://localhost:9080](http://localhost:9080)(change the port according to your configuration) on your local machine to deploy a `docker-compose.yaml` file: dstack-vmm web interface After the container deployed, it should need some time to start the CVM and the containers. Time would be vary depending on your workload. * Click the \[Logs] button to see the logs of the CVM, you can see if the container is finished starting there. * Once the container is running, you can click the \[Dashboard] button to see some information of the container. And the logs of the containers can be seen in the \[Dashboard] page. dstack guest agent status * You can open dstack-gateway's dashboard at [https://localhost:9070](https://localhost:9070) to see the CVM's wireguard ip address, as shown below: dstack gateway dashboard ### Pass Secrets to Apps When deploying a new App, you can pass private data via Encrypted Environment Variables. These variables can be referenced in the docker-compose.yaml file as shown below: dstack secret management The environment variables will be encrypted in the client-side and decrypted in the CVM before being passed to the containers. ### Access the App Once the app is deployed and listening on an HTTP port, you can access the HTTP service via tproxy's public domain. The ingress mapping rules are: * `[s].` maps to port `80` or `443` if with `s` in the CVM. * `-[s].` maps to port `` in the CVM. For example, `3327603e03f5bd1f830812ca4a789277fc31f577-8080.app.kvin.wang` maps to port `8080` in the CVM. Where the `` can be either the app id or the instance id. If the app id is used, one of the instances will be selected by the load balancer.\ If the `id-port` part ends with `s`, it means the TLS connection will be passthrough to the app rather than terminating at dstack-gateway. You can also ssh into the CVM to inspect more information, if your deployment uses the image `dstack-x.x.x-dev`: ```bash theme={"system"} # The IP address of the CVM can be found in the gateway dashboard. ssh root@10.0.3.2 ``` ### Getting TDX quote in docker container To get a TDX quote within app containers: 1. Mount `/var/run/dstack.sock` to the target container in `docker-compose.yaml` ```yaml theme={"system"} version: "3" services: nginx: image: nginx:latest volumes: - /var/run/dstack.sock:/var/run/dstack.sock ports: - "8080:80" restart: always ``` 2. Execute the quote request command in the container. ```bash theme={"system"} # The report_data must be max 64 bytes. For larger data, hash it first. # Example using a hash of the data: curl -X POST --unix-socket /var/run/dstack.sock \ -H 'Content-Type: application/json' \ -d '{"reportData": "0x1234deadbeef00000000000000000000000000000000000000000000000000"}' \ http://dstack/GetQuote | jq . ``` For automated verification of your application's attestation, see the [Trust Center Technical Documentation](/dstack/trust-center-technical) which explains how to use these quotes with the Trust Center verification platform. ### Container logs Container logs can be obtained from the CVM's `dashboard` page or by curl: ```bash theme={"system"} curl 'http://.app.kvin.wang:9090/logs/?since=0&until=0&follow=true&text=true×tamps=true&bare=true' ``` Replace `` and `` with actual values. Available parameters: * since=0: Starting Unix timestamp for log retrieval * until=0: Ending Unix timestamp for log retrieval * follow: Enables continuous log streaming * text: Returns human-readable text instead of base64 encoding * timestamps: Adds timestamps to each log line * bare: Returns the raw log lines without json format The response of the RPC looks like: ``` $ curl 'http://0.0.0.0:9190/logs/zk-provider-server?text×tamps' {"channel":"stdout","message":"2024-09-29T03:05:45.209507046Z Initializing Rust backend...\n"} {"channel":"stdout","message":"2024-09-29T03:05:45.209543047Z Calling Rust function: init\n"} {"channel":"stdout","message":"2024-09-29T03:05:45.209544957Z [2024-09-29T03:05:44Z INFO rust_prover] Initializing...\n"} {"channel":"stdout","message":"2024-09-29T03:05:45.209546381Z [2024-09-29T03:05:44Z INFO rust_prover::groth16] Starting setup process\n"} ``` ## TLS Passthrough with Custom Domain dstack-gateway supports TLS passthrough for custom domains. See the example [here](https://github.com/Dstack-TEE/dstack-examples/tree/main/custom-domain/dstack-ingress) for more details. ### Upgrade an App Got to the dstack-vmm webpage, click the \[Upgrade] button, select or paste the compose file you want to upgrade to, and click the \[Upgrade] button again.\ The app id does not change after the upgrade. Stop and start the app to apply the upgrade. ### HTTPS Certificate Transparency In the tutorial above, we used a TLS certificate with a private key external to the TEE. To establish trust, we need to generate and maintain the certificate's private key within the TEE and provide evidence that all TLS certificates for the domain were originate solely from dstack-gateway CVM. By combining Certificate Transparency Logs and CAA DNS records, we can make best effort to minimize security risks. Here's our approach: * Set CAA records to allow only the account created in dstack-gateway CVM to request Certificates. * Launch a program to monitor Certificate Transparency Log and give alarm once any certificate issued to a pubkey that isn’t generated by dstack-gateway CVM. #### Configurations To launch Certbot, you need to own a domain hosted on Cloudflare. Obtain an API token with DNS operation permissions from the Cloudflare dashboard. Configure it in the `build-config.sh`: ```bash theme={"system"} # The directory to store the auto obtained TLS certificate and key GATEWAY_CERT=${CERBOT_WORKDIR}/live/cert.pem GATEWAY_KEY=${CERBOT_WORKDIR}/live/key.pem # for certbot CF_ZONE_ID=cc0a40... CF_API_TOKEN=g-DwMH... # ACME_URL=https://acme-v02.api.letsencrypt.org/directory ACME_URL=https://acme-staging-v02.api.letsencrypt.org/directory ``` Then re-run the ../build.sh: ```bash theme={"system"} ../build.sh ``` #### Launch certbot Then run the certbot in the `build/` and you will see the following log: ``` $ RUST_LOG=info,certbot=debug ./certbot renew -c certbot.toml 2024-10-25T07:41:00.682990Z INFO certbot::bot: creating new ACME account 2024-10-25T07:41:00.869246Z INFO certbot::bot: created new ACME account: https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853 2024-10-25T07:41:00.869270Z INFO certbot::bot: setting CAA records 2024-10-25T07:41:00.869276Z DEBUG certbot::acme_client: setting guard CAA records for app.kvin.wang 2024-10-25T07:41:01.740767Z DEBUG certbot::acme_client: removing existing CAA record app.kvin.wang 0 issuewild "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168578683" 2024-10-25T07:41:01.991298Z DEBUG certbot::acme_client: removing existing CAA record app.kvin.wang 0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168578683" 2024-10-25T07:41:02.216751Z DEBUG certbot::acme_client: setting CAA records for app.kvin.wang, 0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853" 2024-10-25T07:41:02.424217Z DEBUG certbot::acme_client: setting CAA records for app.kvin.wang, 0 issuewild "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853" 2024-10-25T07:41:02.663824Z DEBUG certbot::acme_client: removing guard CAA records for app.kvin.wang 2024-10-25T07:41:03.095564Z DEBUG certbot::acme_client: generating new cert key pair 2024-10-25T07:41:03.095678Z DEBUG certbot::acme_client: requesting new certificates for *.app.kvin.wang 2024-10-25T07:41:03.095699Z DEBUG certbot::acme_client: creating new order 2024-10-25T07:41:03.250382Z DEBUG certbot::acme_client: order is pending, waiting for authorization 2024-10-25T07:41:03.283600Z DEBUG certbot::acme_client: creating dns record for app.kvin.wang 2024-10-25T07:41:04.027882Z DEBUG certbot::acme_client: challenge not found, waiting 500ms tries=2 domain="_acme-challenge.app.kvin.wang" 2024-10-25T07:41:04.600711Z DEBUG certbot::acme_client: challenge not found, waiting 1s tries=3 domain="_acme-challenge.app.kvin.wang" 2024-10-25T07:41:05.642300Z DEBUG certbot::acme_client: challenge not found, waiting 2s tries=4 domain="_acme-challenge.app.kvin.wang" 2024-10-25T07:41:07.715947Z DEBUG certbot::acme_client: challenge not found, waiting 4s tries=5 domain="_acme-challenge.app.kvin.wang" 2024-10-25T07:41:11.724831Z DEBUG certbot::acme_client: challenge not found, waiting 8s tries=6 domain="_acme-challenge.app.kvin.wang" 2024-10-25T07:41:19.815990Z DEBUG certbot::acme_client: challenge not found, waiting 16s tries=7 domain="_acme-challenge.app.kvin.wang" 2024-10-25T07:41:35.852790Z DEBUG certbot::acme_client: setting challenge ready for https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/14584884443/mQ-I2A 2024-10-25T07:41:35.934425Z DEBUG certbot::acme_client: challenges are ready, waiting for order to be ready 2024-10-25T07:41:37.972434Z DEBUG certbot::acme_client: order is ready, uploading csr 2024-10-25T07:41:38.052901Z DEBUG certbot::acme_client: order is processing, waiting for challenge to be accepted 2024-10-25T07:41:40.088190Z DEBUG certbot::acme_client: order is valid, getting certificate 2024-10-25T07:41:40.125988Z DEBUG certbot::acme_client: removing dns record 6ab5724e8fa7e3e8f14e93333a98866a 2024-10-25T07:41:40.377379Z DEBUG certbot::acme_client: stored new cert in /home/kvin/codes/meta-dstack/dstack/build/run/certbot/backup/2024-10-25T07:41:40.377174477Z 2024-10-25T07:41:40.377472Z INFO certbot::bot: checking if certificate needs to be renewed 2024-10-25T07:41:40.377719Z DEBUG certbot::acme_client: will expire in Duration { seconds: 7772486, nanoseconds: 622281542 } 2024-10-25T07:41:40.377752Z INFO certbot::bot: certificate /home/kvin/codes/meta-dstack/dstack/build/run/certbot/live/cert.pem is up to date ``` Where the command did are: * Registered to letsencrypt and got a new account `https://acme-staging-v02.api.letsencrypt.org/acme/acct/168601853` * Auto set CAA records for the domain on cloudflare, you can open the CF dashboard to see the record: certbot CAA record configuration * Auto requested a new certificate from Let's Encrypt. Automatically renews the certificate to maintain its validity #### Launch dstack-gateway Execute dstack-gateway with `sudo ./dstack-gateway -c gateway.toml`, then access the web portal to check the dstack-gateway CVM managed Let's Encrypt account. The account's private key remains securely sealed within the TEE. dstack gateway account ID ### Certificate transparency log monitor To enhance security, we've limited TLS certificate issuance to dstack-gateway via CAA records. However, since these records can be modified through Cloudflare's domain management, we need to implement global CA certificate monitoring to maintain security oversight. `ct_monitor` tracks Certificate Transparency logs via [https://crt.sh](https://crt.sh/?q=app.kvin.wang), comparing their public key with the ones got from dstack-gateway RPC. It immediately alerts when detecting unauthorized certificates not issued through dstack-gateway: ``` $ ./ct_monitor -t https://localhost:9010/prpc -d app.kvin.wang 2024-10-25T08:12:11.366463Z INFO ct_monitor: monitoring app.kvin.wang... 2024-10-25T08:12:11.366488Z INFO ct_monitor: fetching known public keys from https://localhost:9010/prpc 2024-10-25T08:12:11.566222Z INFO ct_monitor: got 2 known public keys 2024-10-25T08:12:13.142122Z INFO ct_monitor: ✅ checked log id=14705660685 2024-10-25T08:12:13.802573Z INFO ct_monitor: ✅ checked log id=14705656674 2024-10-25T08:12:14.494944Z ERROR ct_monitor: ❌ error in CTLog { id: 14666084839, issuer_ca_id: 295815, issuer_name: "C=US, O=Let's Encrypt, CN=R11", common_name: "kvin.wang", name_value: "*.app.kvin.wang", not_before: "2024-09-24T02:23:15", not_after: "2024-12-23T02:23:14", serial_number: "03ae796f56a933c8ff7e32c7c0d662a253d4", result_count: 1, entry_timestamp: "2024-09-24T03:21:45.825" } 2024-10-25T08:12:14.494998Z ERROR ct_monitor: error: certificate has issued to unknown pubkey: 30820122300d06092a864886f70d01010105000382010f003082010a02820101009de65c767caf117880626d1acc1ee78f3c6a992e3fe458f34066f92812ac550190a67e49ebf4f537003c393c000a8ec3e114da088c0cb02ffd0881fd39a2b32cc60d2e9989f0efab3345bee418262e0179d307d8d361fd0837f85d17eab92ec6f4126247e614aa01f4efcc05bc6303a8be68230f04326c9e85406fc4d234e9ce92089253b11d002cdf325582df45d5da42981cd546cbd2e9e49f0fa6636e747a345aaf8cefa02556aa258e1f7f90906be8fe51567ac9626f35bc46837e4f3203387fee59c71cea400000007c24e7537debc1941b36ff1612990233e4c219632e35858b1771f17a71944adf6c657dd7303583e3aeed199bd36a3152f49980f4f30203010001 ``` ## Troubleshooting #### Error from dstack-vmm: qemu-system-x86\_64: -device vhost-vsock-pci,guest-cid=: vhost-vsock: unable to set guest cid: Address already in use `dstack-vmm` may throw this error when creating a new VM if the [Unix Socket CID](https://man7.org/linux/man-pages/man7/vsock.7.html) is occupied. To solve the problem, first, you should list the occupied CID: ```bash theme={"system"} ps aux | grep 'guest-cid=' ``` Then choose a new range of the CID not conflicting with the CID in use. You can change `build/vmm.toml` file and restart `dstack-vmm`. This error should disappear. For example, you may find 33000-34000 free to use: ```toml theme={"system"} [cvm] cid_start = 33000 cid_pool_size = 1000 ``` When building the dstack from scratch, you should change the CID configs in `build-config.sh` instead, because `vmm.toml` file is generated by `build.sh`. Its content is derived from `build-config.sh`. You may encounter this problem when upgrading from an older version of dstack, because CID was introduced in `build-config.sh` in later versions. In such case, please follow the docs to add the missing entries in `build-config.sh` and rebuild dstack. #### Error: Operation not permitted when building guest image When running `../build.sh guest`, you might encounter this error: ``` Traceback (most recent call last): File "/meta-dstack/poky/bitbake/bin/bitbake-worker", line 278, in child bb.utils.disable_network(uid, gid) File "/meta-dstack/poky/bitbake/lib/bb/utils.py", line 1696, in disable_network with open("/proc/self/uid_map", "w") as f: PermissionError: [Errno 1] Operation not permitted ``` This error occurs because Ubuntu 23.10 and later versions restrict unprivileged user namespaces by default. To fix this, run: ```bash theme={"system"} sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0 ``` Then try building again. For more information about this restriction, see the [Ubuntu discourse post](https://discourse.ubuntu.com/t/spec-unprivileged-user-namespace-restrictions-via-apparmor-in-ubuntu-23-10/37626). ## Contributors dstack is proudly built by open source and Pi-rateship contributors: * Phala Network: [Kevin Wang](https://github.com/kvinwang), [Shelven Zhou](https://github.com/shelvenzhou), [Leechael](https://github.com/leechael) * Teleport: [Andrew Miller](https://github.com/amiller), [Sxy Sun](https://github.com/sxysun) * Flashbots: [Tina](https://github.com/CarboClanC), [Mateusz](https://github.com/Ruteri), [Dmarz](https://github.com/dmarzzz), [Moe](https://github.com/MoeMahhouk) * Ithaca: [Georgios](https://github.com/gakonst) * Fabric: [@gaoist](https://x.com/gaoist) * And many more... The inspiration for this work stems from [Andrew Miller](https://github.com/amiller)’s pioneering concept of a [Docker-based P2P TEE SDK](https://collective.flashbots.net/t/dstack-speedrunning-a-p2p-confidential-vm/3876). This project cannot be built without standing on the shoulders of giants: * [konvera/meta-confidential-compute](https://github.com/konvera/meta-confidential-compute) Special acknowledgment to [Flashbots](https://github.com/flashbots) for building a community around TEE. The TEE Hacker House initiative, organized by [Flashbots](https://github.com/flashbots) and led by [Tina](https://github.com/CarboClanC), has brought together TEE builders to develop tools for TEE-Web3 integration. This collaborative journey has generated invaluable insights for advancing secure, confidential environments within Web3. Together, we’re shaping the future of TEE in Web3, paving the way for more secure and developer-accessible confidential computing! For a full list of the direct contributors to this repo, see [Contributors](https://github.com/Dstack-TEE/dstack/contributors) on GitHub. # Hardware Requirements Source: https://docs.phala.com/dstack/hardware-requirements Hardware requirements and specifications for running dstack TEE infrastructure. When running dstack from a hardware perspective, users have three primary options. * **Physical Server:** The first option is to utilize a physical server, also known as a bare metal server, by purchasing and configuring it themselves, which gives complete control over the hardware environment. * **Rent Server from OVH:** Alternatively, users may choose to rent a server from OVH, a reputable hosting provider, and handle the necessary configurations to run dstack. * **Host on Phala Cloud:** For a more streamlined approach that bypasses configuring the system entirely, getting a Phala Cloud account is a viable option, allowing users to leverage cloud infrastructure, pre-configured for dstack's operations without the need for initial server setup. Let's dive into the the options. ### Physical Server Minimum requirement is to buy an Intel Xeon 5th/6th Generation CPU (TDX Support) and 8 RAM slots filled. The lowest starting cost for a server will be around \$3763 USD. Here is a recommended server if you would like to buy your own and set up dstack. * [https://store.supermicro.com/us\_en/iot-edge-superserver-sys-111e-fwtr.html](https://store.supermicro.com/us_en/iot-edge-superserver-sys-111e-fwtr.html) ### Rent Server from OVH Renting bare metal from OVH has the same requirements as a Physical Server. Typical starting prices for renting a server will run about \$854.50/mo. Here is a link to some OVH servers for rent. * [https://www.ovhcloud.com/en/bare-metal/prices/?display=list\&use\_cases=confidential-computing\&cpu\_brand=Intel](https://www.ovhcloud.com/en/bare-metal/prices/?display=list\&use_cases=confidential-computing\&cpu_brand=Intel) ### Host on Phala Cloud For those that want to get straight to hosting and testing out dstack, [sign-up for your Phala Cloud](/phala-cloud/getting-started/sign-up-for-cloud-account) account and head over to the [getting started](/phala-cloud/getting-started/start-from-cloud-ui) section in Phala Cloud to deploy your first CVM in minutes. # Local Development Guide Source: https://docs.phala.com/dstack/local-development Deploy confidential applications with dstack - a TEE-based infrastructure platform. dstack enables running containerized applications in Trusted Execution Environments (TEE). For development purposes, you can use our TEE simulator to develop and test your applications locally without TEE hardware. This guide covers setting up the simulator and using our SDKs to interact with TEE functionalities. Refer to the [**Hardware Requirements**](/dstack/hardware-requirements) for release. ## Prerequisites * Rust toolchain (for building the simulator) * Git * One of the following development environments: * Node.js 16+ (for JavaScript SDK) * Python 3.7+ (for Python SDK) * Go 1.16+ (for Go SDK) * Rust 1.70+ (for Rust SDK) ## Simulator The latest TEE simulator is available in [dstack](https://github.com/Dstack-TEE/dstack/tree/master/sdk/simulator) code repository. Use the following commands to build it: ### Build Simulator ```bash theme={"system"} # Install rust curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Clone the repository git clone https://github.com/Dstack-TEE/dstack.git # Build the simulator cd dstack/sdk/simulator ./build.sh # Run the simulator ./dstack-simulator # The simulator will create the following socket files: # dstack.sock // Current TEE service interface (v0.5+) # tappd.sock // Legacy TEE service interface (v0.3) # external.sock # guest.sock ``` To use the simulator with any SDK, set the following environment variable with the absolute path: ```bash theme={"system"} export DSTACK_SIMULATOR_ENDPOINT=/path/to/dstack.sock ``` ### Verifying Simulator Setup After starting the simulator, verify the setup by checking: 1. The socket files exist in the simulator directory: ```bash theme={"system"} ls *.sock ``` 2. The environment variable is set correctly: ```bash theme={"system"} echo $DSTACK_SIMULATOR_ENDPOINT ``` ## dstack API The dstack API provides the basic abstraction of the TEE functionalities. In general, it provides the following functions: * **Get Key**: Generate a cryptographic key from the specified key path and returns it along with its certificate chain. * **Get TDX Quote**: Generate a TDX quote with report data. This can be further verified in our [TEE Attestation Explorer](https://proof.t16z.com/) to prove your application and data are in TEE. * **Get TDX Information**: Retrieve worker information, one example response is like ```json theme={"system"} { "app_id": "", "instance_id": "", "app_cert": "", "tcb_info": "", "app_name": "my-app", "public_logs": true, "public_sysinfo": true, "device_id": "", "mr_aggregated": "", "os_image_hash": "", "key_provider_info": "", "compose_hash": "" } ``` The `dstack.sock` is the current TEE RPC API (v0.5+). For more detailed documents, check the [SDK documentation](https://github.com/Dstack-TEE/dstack/tree/master/sdk). > **Note**: If you're upgrading from dstack v0.3, see the [Migration Guide](/phala-cloud/references/migration-from-dstack-v03) for breaking changes. ## Programming Language Support ### JavaScript Check the latest version in [dstack](https://github.com/Dstack-TEE/dstack/tree/master/sdk/js) code repository. #### Installation ```bash theme={"system"} npm install @phala/dstack-sdk ``` #### Basic Usage ```js theme={"system"} import { DstackClient } from '@phala/dstack-sdk'; import crypto from 'crypto'; const client = new DstackClient(); // Check if service is reachable (500ms timeout, never throws) const isReachable = await client.isReachable(); if (!isReachable) { console.log('dstack service is not available'); return; } // Get the information of the Base Image. await client.info(); // Get a deterministic key with path const keyResult = await client.getKey('my-app/encryption/v1'); console.log(keyResult.key); // Uint8Array(32) - already a Uint8Array console.log(keyResult.signature_chain); // Signature chain // Generate TDX quote (manually hash data if > 64 bytes) const data = 'some-data'; const hash = crypto.createHash('sha256').update(data).digest(); const quoteResult = await client.getQuote(hash.slice(0, 32)); console.log(quoteResult.quote); // TDX quote in hex format console.log(quoteResult.event_log); // Event log const rtmrs = quoteResult.replayRtmrs(); // Replay RTMRs ``` ### Python Check the latest version in [dstack](https://github.com/Dstack-TEE/dstack/tree/master/sdk/python) code repository. #### Installation ```bash theme={"system"} pip install dstack-sdk ``` #### Basic Usage ```python theme={"system"} from dstack_sdk import DstackClient import hashlib # Synchronous client client = DstackClient() # Caution: You don't need to do this most of the time. http_client = DstackClient('http://localhost:8000') # Get the information of the Base Image. info = client.info() print(info.app_id) # Application ID print(info.tcb_info.mrtd) # Access TCB info directly print(info.tcb_info.event_log[0].event) # Access event log entries # Get a deterministic key with path key_result = client.get_key('my-app/encryption/v1') print(key_result.key) # Hex string print(key_result.signature_chain) # Signature chain key_bytes = key_result.decode_key() # Decode to bytes (32 bytes) # Generate TDX quote (manually hash data if > 64 bytes) data = b'some-data' hash_value = hashlib.sha256(data).digest() quote_result = client.get_quote(hash_value[:32]) print(quote_result.quote) # TDX quote in hex format print(quote_result.event_log) # Event log rtmrs = quote_result.replay_rtmrs() # Replay RTMRs ``` ### Golang Check the latest version in [dstack](https://github.com/Dstack-TEE/dstack/tree/master/sdk/go) code repository. #### Installation ```bash theme={"system"} go get github.com/Dstack-TEE/dstack/sdk/go ``` #### Basic Usage ```go theme={"system"} package main import ( "context" "fmt" "log/slog" "github.com/Dstack-TEE/dstack/sdk/go/dstack" ) func main() { client := dstack.NewDstackClient( // dstack.WithEndpoint("http://localhost"), // dstack.WithLogger(slog.Default()), ) // Get information about the dstack client instance info, err := client.Info(context.Background()) if err != nil { fmt.Println(err) return } fmt.Println(info.AppID) // Application ID fmt.Println(info.TcbInfo.Mrtd) // Access TCB info directly fmt.Println(info.TcbInfo.EventLog[0].Event) // Access event log entries path := "/test" purpose := "test" // or leave empty // Derive a key with optional path and purpose deriveKeyResp, err := client.GetKey(context.Background(), path, purpose) if err != nil { fmt.Println(err) return } fmt.Println(deriveKeyResp.Key) // Generate TDX quote tdxQuoteResp, err := client.GetQuote(context.Background(), []byte("test")) if err != nil { fmt.Println(err) return } fmt.Println(tdxQuoteResp.Quote) // 0x0000000000000000000 ... rtmrs, err := tdxQuoteResp.ReplayRTMRs() if err != nil { fmt.Println(err) return } fmt.Println(rtmrs) // map[0:00000000000000000 ... } ``` #### Rust Check the latest version in [dstack](https://github.com/Dstack-TEE/dstack/tree/master/sdk/rust) code repository. #### Installation ```toml theme={"system"} [dependencies] dstack-rust = { git = "https://github.com/Dstack-TEE/dstack.git", package = "dstack-rust" } ``` #### Basic Usage ```rust theme={"system"} use dstack_sdk::DstackClient; #[tokio::main] async fn main() -> Result<(), Box> { let client = DstackClient::new(None); // Uses env var or default to Unix socket // Get system info let info = client.info().await?; println!("Instance ID: {}", info.instance_id); // Derive a key let key_resp = client.get_key(Some("my-app".to_string()), None).await?; println!("Key: {}", key_resp.key); println!("Signature Chain: {:?}", key_resp.signature_chain); // Generate TDX quote let quote_resp = client.get_quote(b"test-data".to_vec()).await?; println!("Quote: {}", quote_resp.quote); let rtmrs = quote_resp.replay_rtmrs()?; println!("Replayed RTMRs: {:?}", rtmrs); // Emit an event client.emit_event("BootComplete".to_string(), b"payload-data".to_vec()).await?; Ok(()) } ``` # Media Kit Source: https://docs.phala.com/dstack/media-kit Download official dstack logos, brand assets, and media resources # dstack Media Kit Welcome to the official dstack media kit. Here you'll find all the brand assets, logos, and resources you need to represent dstack in your projects, presentations, and communications. ## Logo Variants ### Horizontal Logos Dstack Horizontal Primary Logo * **Use case**: Light backgrounds, presentations, headers * **Format**: SVG, PNG * Download SVG * Download PNG Dstack Horizontal Dark Logo * **Use case**: Dark backgrounds, dark mode interfaces * **Format**: SVG, PNG * Download SVG * Download PNG ### Vertical Logos Dstack Vertical Primary Logo * **Use case**: Square layouts, social media profiles, app icons * **Format**: SVG, PNG * Download SVG * Download PNG Dstack Vertical Dark Logo * **Use case**: Square layouts on dark backgrounds * **Format**: SVG, PNG * Download SVG * Download PNG ### Icon/Symbol Only Dstack Icon Primary * **Use case**: Favicon, small spaces, minimal branding * **Format**: SVG, PNG * Download SVG * Download PNG Dstack Icon Dark * **Use case**: Favicon on dark themes, minimal dark branding * **Format**: SVG, PNG * Download SVG * Download PNG ## Usage Guidelines ### Do's ✅ * Use the appropriate logo variant for your background (primary on light, dark on dark backgrounds) * Maintain adequate clear space around the logo * Use vector formats (SVG) when possible for crisp scaling * Keep the logo proportions intact when resizing ### Don'ts ❌ * Don't modify the logo colors, fonts, or proportions * Don't place the logo on busy backgrounds that reduce readability * Don't use low-resolution versions when high-quality options are available * Don't combine with other brand elements in confusing ways ## Brand Colors While specific brand color guidelines aren't provided in this media kit, ensure consistency with dstack's visual identity when using these logos in your designs. ## Questions? If you need additional formats, have questions about usage, or require custom variations of these assets, please reach out to the dstack team through our [community channels](/dstack/acknowledgement#contributors). *** *This media kit is provided to help partners, developers, and community members properly represent dstack in their communications and projects.* # Overview Source: https://docs.phala.com/dstack/overview Deploy confidential applications with dstack — a TEE-based infrastructure platform. > [Phala Cloud](/) is the managed version of dstack deployment that enables developers to deploy programs to CVM (Confidential VM), and to follow the security best practices by default. The dstack SDK simplifies deploying programs to CVMs (Confidential VMs) with security best practices built in. Explore [dstack's blockchain-verified computing](https://phala.com/dstack) capabilities and architecture. The main features include: * Deploy any Docker container as a CVM on supported TEEs * Generate remote attestation reports and visualize the chain of trust via Web UI * Wrap HTTP services with automatic RA-HTTPS and content-addressed domains (`0xABCD.dstack.host`) * Keep applications portable across hardware via decentralized Root-of-Trust key management The following example shows how dstack works with a typical multi-container application configured with Docker Compose. As the architecture shown below, multiple docker containers can run inside one single CVM. The underlying infrastructure we provide is to make sure the application is secure and verifiable. Your containers use the `dstack` component to communicate with the underlying `tappd`. `dstack` sets up the CVM environment, handles remote attestation, and manages the lifecycle of all Docker containers running inside the CVM. `tappd` communicates with a decentralized Key Management Service (KMS) that derives deterministic encryption keys for the application. These keys encrypt application-specific storage and protect data integrity. Because the KMS operates independently from any specific TEE instance, your applications avoid vendor lock-in and can be securely migrated between different hardware environments without data loss. dstack CVM architecture diagram showing the relationship between docker containers, dstack, tappd, and decentralized KMS ## Verify If An Application is Running Inside a TEE When the application launches, dstack exports a Remote Attestation (RA) Report that cryptographically binds the application's runtime information — Docker image hash, startup arguments, and environment variables. The TEE hardware signs this report, and the application's own derived key co-signs it. Anyone can verify the report using standard TEE RA verification tools. For applications deployed on Phala Intel TDX workers, RA reports are exported and verified by default — use the [TEE Attestation Explorer](https://proof.t16z.com/) to inspect them. ## Conclusion Ready to build? See [Getting Started](/phala-cloud/getting-started/overview) for deployment options. For managed hosting without running dstack on your own hardware, use [Phala Cloud](/). # dstack Security Audit Source: https://docs.phala.com/dstack/security-audit Third-party security audit of dstack by zkSecurity covering system architecture, TEE integration, and production hardening # dstack Security Audit Phala Network engaged zkSecurity to conduct a comprehensive security audit of the dstack project in May 2025. This audit provides an independent assessment of dstack's security architecture, implementation quality, and production readiness. ## Audit Overview **Auditor:** zkSecurity **Engagement Period:** May 26 - June 13, 2025 **Audit Team:** Two security consultants **Report Date:** May 26, 2025 ### Scope The audit covered two primary areas: #### Low-Level Libraries and Tooling * **ra-tls and ra-rpc**: Remote attestation TLS implementation * **guest-agent**: In-CVM service for key derivation and attestation * **dstack-util**: CLI tools including full-disk encryption functionality #### Image-Related Files * **Yocto BitBake recipes**: Production image build configurations * **Base initialization scripts**: System setup and hardening * **Production vs development images**: Security difference analysis ### Methodology The audit followed a two-phase approach: **Phase 1: Core Security Analysis** * Understanding attacker models and trust boundaries * Reviewing RATLS (Remote Attestation TLS) protocol implementation * Analyzing CVM interfaces and access controls * Evaluating privilege escalation strategies **Phase 2: System Image Analysis** * Reviewing build reproducibility * Validating measurement integrity * Analyzing production image hardening * Assessing dm-verity integration * Evaluating host operator attack vectors ## Key Findings Summary The audit identified **12 findings** across different risk levels: | Risk Level | Count | Examples | | ----------------- | ----- | -------------------------------------------------------------- | | **High** | 1 | VMM currently trusted in OVMF build | | **Medium** | 6 | Terminal binaries in production, symbolic link vulnerabilities | | **Low** | 3 | Incomplete measurement checks, documentation gaps | | **Informational** | 2 | Production deployment guidance, design documentation | ### Critical Finding: OVMF Configuration The highest-severity finding identified that dstack was using OVMF Configuration A, which trusts the Virtual Machine Manager (VMM). The audit recommended moving to Configuration B, which places the VMM outside the Trusted Computing Base (TCB). **Impact:** This configuration choice affects the fundamental trust model of the TEE environment. **Status:** ✅ **Addressed** - The dstack team implemented the recommended OVMF Configuration B. ## Implementation Status The dstack team has been proactive in addressing audit findings: ✅ **Fixed**: OVMF configuration upgraded to secure mode ✅ **Fixed**: Production image hardening improvements ✅ **Fixed**: Symbolic link vulnerability patched ✅ **Fixed**: Terminal binary removal from production ✅ **Enhanced**: Documentation and security guides added ## Access Full Report Access the complete 39-page security audit report with detailed technical findings, recommendations, and implementation guidance. ## Related Documentation Learn about dstack's architecture and core concepts Understand Phala Cloud's security model Deep dive into TEE attestation mechanisms Security checklist for production deployments # Trust Center Technical Documentation Source: https://docs.phala.com/dstack/trust-center-technical Technical architecture and API reference for Trust Center verification system ## Overview The Trust Center is an open-source verification platform that automates the validation of TEE attestations for dstack-based applications. It performs multi-dimensional verification across hardware, operating system, source code, and domain ownership to provide cryptographic proof of deployment integrity. **Architecture**: Monorepo with TypeScript verifier library, background worker queue, and web dashboard. **Repository**: [https://github.com/Phala-Network/trust-center](https://github.com/Phala-Network/trust-center) ## System Architecture ### Components 1. **Verifier Package** (`@phala/dstack-verifier`) * Core TypeScript library for attestation verification * Modular verifier classes with configurable flags * Supports Intel TDX quote validation * Platform-specific verifiers (PhalaCloud, Redpill.ai) 2. **Background Worker** * Built on BullMQ for reliable job processing * PostgreSQL for task persistence * Redis for queue management * Asynchronous verification execution 3. **Web Application** * Next.js dashboard for report visualization * Direct database access for read operations * Real-time verification status updates * Public report URLs for transparency 4. **Storage Layer** * PostgreSQL: Verification tasks and results * Redis: Job queue state * S3-compatible: Raw attestation data and quotes ## Verification Phases The Trust Center performs verification in four distinct phases: ### Phase 1: Hardware Attestation Validates the TEE quote to prove execution on genuine Intel hardware. **What's verified:** * **Quote signature**: ECDSA-P256 signature validation using Intel certificates * **Certificate chain**: Validates back to Intel root CA * **TCB status**: Checks Trusted Computing Base security version * **Platform identity**: Confirms genuine Intel TDX processor * **Revocation status**: Verifies no certificates or platforms are revoked **Tools used:** * Quote verification with [DCAP-QVL](https://github.com/Phala-Network/dcap-qvl) **Implementation:** ```typescript theme={"system"} // Note: verifyQuote is an internal utility in @phala/dstack-verifier // For hardware verification, use VerificationService with hardware flag import { VerificationService } from '@phala/dstack-verifier'; const service = new VerificationService(); // For Redpill apps (with GPU models) const result = await service.verify({ contractAddress: '0x...', model: 'phala/llama-3.1-8b' }, { hardware: true, os: false, sourceCode: false }); // Or for Phala Cloud apps const result2 = await service.verify({ contractAddress: '0x...', domain: 'myapp.phala.network' }, { hardware: true, os: false, sourceCode: false }); console.log('Hardware verified:', result.success); ``` **Failure modes:** * Invalid signature: Quote not signed by genuine TEE * Revoked certificates: Platform or attestation key compromised * Outdated TCB: Security patches not applied ### Phase 2: Operating System Integrity Confirms the OS image matches a known trusted dstack version. **What's verified:** * **MRTD**: Measurement Root of Trust Domain (TD initial state) * **RTMR0**: Virtual hardware environment * **RTMR1**: Linux kernel measurement * **RTMR2**: Kernel command line and initrd **Tools used:** * dstack image verification with [dstack-mr-cli](https://github.com/Dstack-TEE/dstack/tree/master/dstack-mr/cli) **Measurement calculation:** ```bash theme={"system"} # Calculate OS measurements using dstack-mr-cli dstack-mr-cli --kernel bzImage --initrd initrd.img --cmdline "console=htyS0" # Output: MRTD, RTMR0, RTMR1, RTMR2 values ``` **Verification logic:** ```typescript theme={"system"} // OS verification is performed by VerificationService // The service automatically downloads trusted dstack images and compares measurements import { VerificationService } from '@phala/dstack-verifier'; const service = new VerificationService(); const result = await service.verify({ contractAddress: '0x...', model: 'phala/llama-3.1-8b', // For Redpill // OR domain: 'myapp.phala.network' // For Phala Cloud metadata: { osSource: { github_repo: 'https://github.com/Dstack-TEE/meta-dstack', git_commit: 'abc123', version: 'v0.5.2' } } }, { hardware: false, os: true, // Enable OS verification sourceCode: false }); console.log('OS verified:', result.success); ``` **Why it matters**: Ensures the OS hasn't been modified, backdoored, or replaced with an untrusted image. ### Phase 3: Source Code Verification Validates the application code matches the deployed version. **What's verified:** * **compose-hash**: SHA256 hash of `docker-compose.yaml` * **app-id**: Application identifier (used for key derivation) * **instance-id**: Specific deployment instance * **key-provider**: KMS service address * **RTMR3**: Application measurement register containing above values **RTMR3 structure:** ```typescript theme={"system"} interface RTMR3Events { 'app-id': string; // Application identifier 'compose-hash': string; // SHA256 of docker-compose.yaml 'instance-id': string; // Deployment instance UUID 'key-provider': string; // KMS endpoint URL } ``` **RTMR3 calculation:** ```typescript theme={"system"} // RTMR3 is computed as a hash chain of events function computeRTMR3(events: RTMR3Events): string { let rtmr3 = '0'.repeat(96); // Initial value (all zeros) for (const [key, value] of Object.entries(events)) { const event = JSON.stringify({ [key]: value }); const eventHash = sha384(event); rtmr3 = sha384(rtmr3 + eventHash); } return rtmr3; } ``` **Verification against blockchain:** ```typescript theme={"system"} import { createPublicClient, http } from 'viem'; import { mainnet } from 'viem/chains'; const client = createPublicClient({ chain: mainnet, transport: http() }); // Read compose-hash from blockchain registry const onChainHash = await client.readContract({ address: '0x...', abi: registryABI, functionName: 'getAppComposeHash', args: [appId] }); const match = onChainHash === quote.body.rtmr3.composeHash; console.log('Source code verified:', match); ``` **Tools:** * [dstack attestation docs](https://github.com/Dstack-TEE/dstack/blob/master/attestation.md) ### Phase 4: Domain Verification (Gateway Only) For applications with custom domains, verifies zero-trust HTTPS. **What's verified:** * **TLS certificate**: Private key generated in TEE * **CAA records**: DNS locked to TEE-controlled Let's Encrypt account * **Certificate Transparency**: All certs logged to public CT logs * **Domain control**: Cryptographic proof of domain ownership **Verification process:** 1. Extract TLS public key from `reportData` field 2. Query Certificate Transparency logs (via crt.sh) 3. Verify all certificates for domain match the TEE public key 4. Check CAA records point to TEE Let's Encrypt account 5. Confirm no unauthorized certificates exist **CAA record format:** ``` example.com CAA 0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/XXXXXX" ``` **CT Log monitoring:** ```typescript theme={"system"} // Domain verification (including CT logs) is performed by VerificationService // The service checks Certificate Transparency logs when ctLog flag is enabled import { VerificationService } from '@phala/dstack-verifier'; const service = new VerificationService(); const result = await service.verify({ contractAddress: '0x...', domain: 'example.com' }, { hardware: false, os: false, sourceCode: false, teeControlledKey: true, // Verify TEE controls TLS private key certificateKey: true, // Verify certificate matches dnsCAA: true, // Verify DNS CAA records ctLog: true // Check Certificate Transparency logs (can be slow) }); if (!result.success) { console.error('❌ Domain verification failed:', result.errors); } ``` **Why it matters**: Traditional cloud providers can intercept HTTPS traffic because they control certificate private keys. Zero-trust HTTPS proves even the cloud provider cannot decrypt your traffic. ## Using the Verifier Package ### Basic Usage ```typescript theme={"system"} import { VerificationService } from '@phala/dstack-verifier'; const service = new VerificationService(); // For Redpill AI models const result = await service.verify({ contractAddress: '0x7e0817205044eb202a590a0d236ccb4d66140197', model: 'phala/llama-3.1-8b', metadata: { osSource: { github_repo: 'https://github.com/Dstack-TEE/meta-dstack', git_commit: 'abc123', version: 'v0.5.2' } } }, { hardware: true, // Verify Intel TDX quote os: true, // Verify OS integrity sourceCode: true, // Verify compose-hash teeControlledKey: false, certificateKey: false, dnsCAA: false, ctLog: false }); // For Phala Cloud apps with custom domains const result2 = await service.verify({ contractAddress: '0x...', domain: 'myapp.phala.network' }, { hardware: true, os: true, sourceCode: true, teeControlledKey: true, // Verify TEE controls TLS key certificateKey: true, // Verify certificate dnsCAA: false, // Skip DNS CAA check (can be slow) ctLog: false // Skip CT log check (can be very slow) }); console.log('Verification success:', result.success); console.log('Verification errors:', result.errors); console.log('Data objects:', result.dataObjects); ``` ### Configurable Verification Flags ```typescript theme={"system"} import type { VerificationFlags } from '@phala/dstack-verifier'; // Available verification flags (all optional, defaults shown) const flags: VerificationFlags = { hardware: true, // Verify Intel TDX quote os: true, // Verify OS integrity (MRTD, RTMR0-2) sourceCode: true, // Verify source code (compose-hash in RTMR3) teeControlledKey: true, // Verify TEE controls private key certificateKey: true, // Verify certificate matches attestation dnsCAA: true, // Verify DNS CAA records (can be slow) ctLog: false // Verify Certificate Transparency logs (can be very slow) }; // Use with VerificationService const service = new VerificationService(); const result = await service.verify(config, flags); ``` ### Platform-Specific Configuration ```typescript theme={"system"} import { VerificationService } from '@phala/dstack-verifier'; import type { PhalaCloudConfig, RedpillConfig } from '@phala/dstack-verifier'; const service = new VerificationService(); // Phala Cloud configuration const phalaConfig: PhalaCloudConfig = { contractAddress: '0x...', domain: 'myapp.phala.network', metadata: { osSource: { github_repo: 'https://github.com/Dstack-TEE/meta-dstack', git_commit: 'abc123', version: 'v0.5.2' } } }; // Redpill.ai configuration const redpillConfig: RedpillConfig = { contractAddress: '0x...', model: 'phala/llama-3.1-8b', metadata: { osSource: { github_repo: 'https://github.com/Dstack-TEE/meta-dstack', git_commit: 'def456', version: 'v0.5.3' } } }; // Use the same service for both platforms const phalaResult = await service.verify(phalaConfig); const redpillResult = await service.verify(redpillConfig); ``` ### Error Handling ```typescript theme={"system"} import { VerificationService } from '@phala/dstack-verifier'; import type { VerificationError } from '@phala/dstack-verifier'; const service = new VerificationService(); try { const result = await service.verify(config, flags); if (!result.success) { console.error('Verification failed'); // Errors are objects with { message: string } for (const error of result.errors) { console.error('Error:', error.message); // Check error messages for specific issues if (error.message.includes('signature')) { console.error('→ Hardware signature invalid - not genuine TEE'); } else if (error.message.includes('compose')) { console.error('→ Deployed code does not match source'); } else if (error.message.includes('MRTD') || error.message.includes('RTMR')) { console.error('→ OS measurements do not match trusted version'); } } } else { console.log('✓ All verification checks passed'); console.log('Data objects:', result.dataObjects); } } catch (err) { // Handle unexpected errors during verification console.error('Verification process error:', err); } ``` ## Self-Hosting Trust Center Organizations can run their own Trust Center instance for private verification. ### Prerequisites * Node.js 18+ / Bun * PostgreSQL 14+ * Redis 6+ * S3-compatible storage (MinIO, AWS S3, etc.) ### Setup ```bash theme={"system"} # Clone repository git clone https://github.com/Phala-Network/trust-center.git cd trust-center # Install dependencies bun install # Configure environment cp .env.example .env # Edit .env with your database, redis, and S3 credentials # Run database migrations bun run db:migrate # Start services bun run worker # Background verification worker bun run webapp # Web dashboard (localhost:3000) ``` ### Configuration **`.env` file:** ```bash theme={"system"} # Database DATABASE_URL=postgresql://user:pass@localhost:5432/trust_center # Redis REDIS_URL=redis://localhost:6379 # S3 Storage S3_ENDPOINT=https://s3.amazonaws.com S3_BUCKET=trust-center-quotes S3_ACCESS_KEY=... S3_SECRET_KEY=... # PCCS (for quote verification) PCCS_URL=https://pccs.phala.network/sgx/certification/v4/ # Optional: Blockchain RPC for source code verification ETH_RPC_URL=https://eth-mainnet.g.alchemy.com/v2/... ``` ### Docker Deployment ```yaml theme={"system"} version: '3.8' services: postgres: image: postgres:14 environment: POSTGRES_DB: trust_center POSTGRES_USER: trust_center POSTGRES_PASSWORD: secure_password volumes: - postgres_data:/var/lib/postgresql/data redis: image: redis:6 volumes: - redis_data:/data worker: build: . command: bun run worker depends_on: - postgres - redis env_file: .env webapp: build: . command: bun run webapp ports: - "3000:3000" depends_on: - postgres env_file: .env volumes: postgres_data: redis_data: ``` ## Next Steps * **User Documentation**: [Trust Center Verification Reports](/phala-cloud/attestation/trust-center-verification) * **Attestation Basics**: [Understanding TEE Attestation](/phala-cloud/attestation/verifying-attestation) * **API Reference**: [Attestation API](/phala-cloud/phala-cloud-api/attestations) * **dstack SDK**: [Getting Started with dstack](/dstack/getting-started) ## Additional Resources * **GitHub Repository**: [trust-center](https://github.com/Phala-Network/trust-center) * **DeepWiki Analysis**: [Technical Deep Dive](https://deepwiki.com/Phala-Network/trust-center) * **Live Platform**: [trust.phala.com](https://trust.phala.com) * **dstack Attestation Docs**: [attestation.md](https://github.com/Dstack-TEE/dstack/blob/master/attestation.md) # Overview Source: https://docs.phala.com/index Official documentation for Phala Cloud - deploy AI applications with hardware-level privacy, TEE security, and verifiable computation. Get started with confidential computing today. Phala Documentation - Confidential AI and TEE Computing Platform Welcome to the official documentation for **Phala** - the secure cloud platform that enables AI developers to deploy applications with hardware-level privacy and verifiable computation. ## What is Phala Cloud Phala Cloud is a Confidential AI native Neocloud solution that provides you with a secure, user-friendly environment for running AI applications. You can deploy any standard Docker application directly into a **Trusted Execution Environment (TEE)**, ensuring your AI models and data remain confidential and tamper-proof. ### Key Features for AI Developers * **Confidential AI Computing**: Run LLMs and AI models in GPU TEE with hardware-level security * **Verifiable Results**: Cryptographic proof that your AI computations are genuine and unmodified * **Privacy Protection**: Complete confidentiality for your models, data, and algorithms * **Easy Deployment**: Deploy existing Docker applications with zero code changes * **Cost Effective**: Web2 pricing model with transparent, second-level billing ## Getting Started New to Phala Cloud? You can start here: Deploy your first secure application in minutes Explore the complete cloud platform features Run LLMs in GPU TEE environments ## Core Concepts Understanding Phala Cloud's architecture: Learn how TEE provides hardware-level security Secure cryptographic key generation and management Developer tools for building on Phala Cloud ## Explore More * **Platform**: Learn more about [Phala Cloud](https://phala.com) * **Blog & Tutorials**: Read the latest [blog posts and guides](https://phala.com/blog) * **About Us**: Discover [Phala's mission and team](https://phala.com/about) ## Support Need help? We're here to assist: * **Documentation Issues**: Report problems with our docs on [GitHub](https://github.com/Phala-Network/phala-docs) * **Technical Support**: Join our [Telegram community](https://t.me/+nbhjx1ADG9EyYmI9) for real-time help * **Enterprise**: [Contact us](https://phala.com/contact) for enterprise solutions and partnerships *** Ready to build secure AI? [Get started with Phala Cloud →](https://cloud.phala.com) # Introduction Source: https://docs.phala.com/network/compute-providers/introduction Essential information for becoming a compute provider on Phala Network(ETH L2). After [Sunsetting Phala Parachain and Migrating to Ethereum L2](https://phala.subsquare.io/democracy/referenda/77), and upgrading the network infrastructure to Intel TDX and GPU Confidential Computing, computing power providers will migrate from the parachain-based environment to the new Ethereum L2 ecosystem. But Phala’s core tokenomics and incentive structure remain largely consistent and predictable. ## 1. Total Supply & High-Level Distribution Phala maintains a fixed total token supply of: * **1,000,000,000 PHA (1 billion PHA)** Out of the total supply, **70% is allocated to mining rewards**. This portion is designed to: * Incentivize long-term network participation * Reward resource providers (especially GPU miners) * Support staking-based security and governance * Fund ecosystem growth via the treasury The **70% mining reward pool** is further split into three parts: | Allocation Target | Share of Mining Rewards | Description | | ---------------------------- | ----------------------- | --------------------------------------------------------- | | **Treasury** | 20% | Ecosystem, development, and long-term strategic funding | | **Ethereum Staking Rewards** | 40% | Rewards for PHA staked into the Ethereum staking contract | | **GPU Miners** | 40% | Rewards for contributing TDX-secured GPU compute | ## 2. PHA → vPHA: The L2 Native & Governance Token After the migration to Ethereum and an L2-based architecture, Phala introduced a **staking contract on Ethereum** that allows users to convert: PHA → vPHA ### 2.1 What is vPHA? **vPHA** is the core token for the **Phala L2 Network**, and is designed to be the primary token used within the L2 ecosystem. Key roles of **vPHA** include: * **Governance**\ Used to participate in community governance and protocol decision-making. * **GPU Worker Staking**\ Used as stake/collateral for GPU workers to participate in the mining set. * **L2 Application Token Use Cases**\ Intended to be the core token for future GPU-related L2 applications and usage scenarios. ### 2.2 Why Stake PHA into vPHA? By staking PHA into the Ethereum staking contract and receiving vPHA, users: * Help secure and govern the Phala L2 ecosystem * Gain access to **staking rewards** (from the 40% mining reward share allocated to Ethereum staking) * Obtain the core utility token for future L2 GPU-related use cases Therefore, the community is **encouraged to convert PHA into vPHA**, and the protocol offers **corresponding PHA staking rewards** to incentivize this behavior. ## 3. GPU Miners: Core of the TDX-Backed GPU Compute Network **GPU miners** are the backbone of Phala’s future **TDX-secured GPU computing network**. By contributing GPU resources to the network, miners receive a portion of the mining rewards based primarily on: * **GPU type** (hardware class and performance) * **Online time** (availability and stability) Rewards are essentially a function of **hardware weight × uptime**. ### 3.1 Reward Weights by GPU Type Different GPU models receive different reward weights to reflect their relative performance and compute value. Currently supported GPU models and their reward weights: | GPU Model | Reward Weight | | --------- | ------------- | | **H100** | 1.0 | | **H200** | 1.5 | | **B200** | 1.8 | A higher weight implies a higher share of GPU mining rewards, assuming comparable uptime and reliability. ### 3.2 GPU Reward Distribution Mechanism GPU mining rewards are distributed in the form of **vPHA**. * Rewards are calculated continuously based on validated GPU uptime and hardware weight. * **Payouts occur every hour**, with rewards sent at the **end of each hour**. ## 4. vPHA Collateral Requirements for GPU Miners To ensure the **security and reliability** of GPU compute in the network, Phala requires GPU miners to provide additional **vPHA collateral** for each GPU they operate. > This collateral is **not** the same as staking into the Ethereum staking contract.\ > It is **non-yield-bearing** and exists purely as a **security guarantee** for GPU compute. ### 4.1 Required vPHA per GPU The collateral requirements per GPU are: | GPU Model | Required vPHA Stake per GPU | | --------- | --------------------------- | | **H100** | 1250 vPHA | | **H200** | 1875 vPHA | | **B200** | 2250 vPHA | Characteristics of this collateral: * It does **not** accrue staking rewards or interest. * It is used strictly as a security mechanism. * It links the operator’s economic interest to the quality and honesty of the GPU service they provide. This design improves: * **Sybil resistance** – prevents low-cost spam or malicious GPU registration * **Reliability** – encourages miners to maintain high-quality nodes and configurations * **Accountability** – provides economic basis for punitive or exclusion mechanisms if needed ## 5. Quality Control & Governance Over GPU Capacity To uphold network reliability, the Phala team will **periodically verify the quality of GPU compute** available in the network. These verifications may include: * Performance and benchmark validation * TDX integrity and security checks * Uptime and stability metrics * Other trust and safety standards defined by the protocol and community If a GPU is found to be **non-compliant** (e.g., underperforming, misconfigured, or malicious), the process is: 1. The issue is documented and shared with the community. 2. Upon announced, the corresponding GPU(s) can be **removed from the mining set**, and associated rewards can be stopped. This ensures the GPU mining ecosystem remains: * **Secure** * **Reliable** * **Aligned with community standards** # Phala GPU Mining Deployment Guide (H100 / H200 / B200) Source: https://docs.phala.com/network/compute-providers/run-GPU-on-phala/gpu-deployment-guide Deploy and manage worker nodes to provide computing power on Phala Network. This document is a step-by-step guide for community members to deploy **Phala GPU Mining** on their own servers. Currently supported GPU models: * **NVIDIA H100** * **NVIDIA H200** * **NVIDIA B200** Deployment is divided into **two major stages**: 1. **Enable TDX in BIOS, configure the system environment, and install Dstack (with GPU stack enabled)** 2. **Run the Phala GPU Mining service inside the server via Docker and connect to Phala Cloud** ## 1. Enable TDX & Install Dstack ### 1.1 Enable TDX in BIOS Before installing or booting the OS, enter the server BIOS and enable: * Intel TDX * Intel VT-x / VT-d * SR-IOV (if available) * Any required CPU virtualization / security features (names may vary by vendor) > Different hardware vendors may use different BIOS names. Please refer to your server’s manual. ### 1.2 Configure TDX System Environment Follow Intel’s / Dstack’s official instructions to configure the TDX system environment: * Dstack repo & basic setup:\ **[https://github.com/Dstack-TEE/dstack](https://github.com/Dstack-TEE/dstack)** If you encounter errors related to **Intel modules** (for example, TDX module installation failures), refer to Intel’s enabling guide: * Intel TDX hardware setup guide:\ **[https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/04/hardware\_setup/](https://cc-enabling.trustedservices.intel.com/intel-tdx-enabling-guide/04/hardware_setup/)** Make sure all required kernel modules and firmware settings for TDX are correctly configured before continuing. ### 1.3 Upgrade NVIDIA Driver to Latest Version Phala GPU Mining relies on NVIDIA Confidential Computing–capable GPUs and drivers.\ Please upgrade your NVIDIA driver to the latest supported version and verify compatibility: * NVIDIA Confidential Computing compatibility matrix:\ **[https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/secure-ai-compatibility-matrix/](https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/secure-ai-compatibility-matrix/)** After installation, verify that the GPU is correctly recognized: ```bash theme={"system"} nvidia-smi ``` ### 1.4 Install and Run Dstack The base deployment of Dstack follows the official documentation: * Dstack repo:\ **[https://github.com/Dstack-TEE/dstack](https://github.com/Dstack-TEE/dstack)** > By default, the Dstack GitHub documentation focuses on **Confidential CPU** only.\ > To run **Phala GPU Mining**, you must additionally enable and configure the **GPU Stack**. #### 1.4.1 Update `vmm.toml` for GPU Stack Locate your `vmm.toml` configuration file and update (or add) the following fields. **Global paths & KMS configuration:** ```toml theme={"system"} image_path = "" run_path = "" kms_url = "" [cvm] kms_urls = [""] gateway_urls = [""] cid_start = 672000 cid_pool_size = 1000 max_disk_size = 2000 qemu_single_pass_add_pages = false qemu_pic = false qemu_pci_hole64_size = 1125899906842624 qemu_hotplug_off = true ``` **GPU configuration:** ```toml theme={"system"} [cvm.gpu] enabled = true # To expose all available GPU devices to the dashboard and VMs: include = [] # Alternatively, you can explicitly list GPU PCI IDs, e.g.: # include = ["0000:17:00.0"] ``` **Supervisor configuration:** ```toml theme={"system"} [supervisor] detached = true auto_start = false ``` **Notes:** * `cvm.gpu.include` * `[]` = use **all** available GPUs * Or specify specific GPU device IDs as needed. * `kms_url` / `kms_urls` * Replace with your actual KMS installation address or domain. * `gateway_urls` * Replace with your actual Dstack gateway URL(s). * `qemu_path` * In some environments, `qemu` is **not** located at `/opt/qemu-xxx`.\ Check the actual QEMU location and ensure your Dstack configuration points to the correct path. * `image_path` / `run_path` * Use paths where your VM images and runtime files will be stored. #### 1.4.2 Start Dstack Services Once configuration is complete, start the core Dstack components in the following order: 1. **Start KMS** ```bash theme={"system"} ./dstack-kms -c kms.toml ``` 2. **Start Gateway** ```bash theme={"system"} sudo ./dstack-gateway -c gateway.toml ``` 3. **Start VMM** ```bash theme={"system"} ./dstack-vmm -c vmm.toml ``` #### 1.4.3 Verify Dstack Setup Dstack is considered correctly deployed if: * All three processes (**KMS**, **Gateway**, **VMM**) run without critical errors. * You can access the Dstack web UI via its URL (dashboard loads successfully). * GPU devices are visible/usable through Dstack after enabling `[cvm.gpu]`. If these are met, your **Confidential CPU + GPU Stack environment is ready**. ## 2. Run the Phala GPU Mining Service After Dstack and the GPU stack are running correctly, you can start deploying the Phala GPU Mining service. The mining backend is a Docker-based service that: * Connects to your local Dstack instance * Registers the node with **Phala Cloud** * Exposes your GPU resources for Phala GPU mining > It is recommended to run the backend as **root** (or with equivalent privileges). ### 2.1 Use the Official Mining Backend Docker Setup Phala provides an official Docker Compose configuration: * Docker Compose file:\ **[https://github.com/Phala-Network/dstack-mining-backend/blob/main/docker-compose.yaml](https://github.com/Phala-Network/dstack-mining-backend/blob/main/docker-compose.yaml)** * Full backend repo:\ **[https://github.com/Phala-Network/dstack-mining-backend](https://github.com/Phala-Network/dstack-mining-backend)** ### 2.2 Clone the Repository and Configure ```bash theme={"system"} git clone https://github.com/Phala-Network/dstack-mining-backend.git cd dstack-mining-backend ``` Edit `docker-compose.yaml` and update necessary fields: * Dstack gateway and KMS endpoints * Phala Cloud API credentials * Optional: volume mounts, logging, etc. ### 2.3 Start the Mining Backend ```bash theme={"system"} docker compose up -d ``` This will: * Launch the backend container * Connect it to your local Dstack instance * Register your GPU node with Phala Cloud ### 2.4 Confirm Mining Is Active Check Docker containers: ```bash theme={"system"} docker ps ``` View logs: ```bash theme={"system"} docker compose logs -f ``` Your setup is successful when: * Backend connects to Dstack gateway & KMS * GPUs (H100 / H200 / B200) are detected * Your node registers on Phala Cloud * Mining jobs begin to run ## 3. Troubleshooting Common issues: * **Intel TDX module errors**\ → Re-check BIOS and Intel’s TDX guide. * **GPU not detected**\ → Ensure `cvm.gpu.enabled = true` and correct NVIDIA drivers. * **QEMU path issue**\ → Verify QEMU location on your OS. * **Backend cannot reach Dstack**\ → Confirm gateway/KMS URLs and firewall settings. * **Docker device permission issues**\ → Use root or ensure access to `/dev/nvidia*`. # Register Your GPU Worker On-Chain and Start Earning Rewards Source: https://docs.phala.com/network/compute-providers/run-GPU-on-phala/register-work-onchain How to Register Your GPU Worker On-Chain and Start Earning Rewards: After successfully deploying Dstack, enabling TDX + GPU stack support, your GPU worker is technically online — but **not yet registered on-chain**.\ To actually start earning GPU mining rewards, each GPU must: 1. Pass environment verification 2. Receive an official **GPU NFT** 3. Create a **stakepool** 4. Lock the GPU NFT into the stakepool 5. Deposit required **vPHA collateral** 6. Activate mining This guide walks you through each step. ## 1. Manual Environment Verification (Required Before On-Chain Registration) Because the automated GPU verification pipeline is still under development, GPU miners must currently undergo **manual verification**. After your machine: * appears online in Phala Cloud * shows healthy TDX + GPU stack operation * completes all configuration steps Please contact the Phala Delegation Team: [delegation@phala.network](mailto:delegation@phala.network) Include the following: * Your GPU hardware model (H100 / H200 / B200) * Your wallet address (for receiving the NFT) * Any logs or additional information if requested The Phala team will manually verify: * TDX environment readiness * GPU compatibility and stability * Proper Dstack configuration * Worker reliability Once approved, you may proceed to on-chain registration. ## 2. Receive Your GPU NFT After the manual verification is completed, the Phala team will help mint a **GPU Worker NFT** for your device. This NFT: * Represents your GPU on-chain * Stores metadata such as GPU model & reward weight * Is mandatory for earning mining rewards * Must be locked into your stakepool Without the NFT, your GPU **cannot** receive vPHA mining rewards, even if it is online. ## 3. Create Your StakePool A **stakepool** is a container that: * Holds your GPU NFTs * Stores your required vPHA collateral * Tracks your mining activity and earnings * Acts as your mining identity on-chain Each miner typically only needs **one stakepool**, unless managing GPUs under different ownership structures. Using your wallet interface (e.g., MetaMask): 1. Click **Create StakePool** 2. Sign the transaction 3. Wait for on-chain confirmation Your stakepool is now ready. ## 4. Lock Your GPU NFT Into the StakePool Once your stakepool is created: 1. Open the stakepool interface 2. Choose **“Lock NFT”** 3. Select the GPU Worker NFT you just received 4. Confirm the transaction Your GPU is now officially registered on-chain. ## 5. Deposit Required vPHA Collateral Each GPU requires a certain amount of **vPHA collateral** to ensure network security. | GPU Model | Required vPHA | | ---------- | ------------- | | **H100** | 1250 vPHA | | **H100x8** | 10000 vPHA | | **H200** | 1875 vPHA | | **H200x8** | 15000 vPHA | | **B200** | 2250 vPHA | | **B200x8** | 18000 vPHA | To deposit collateral: 1. Convert PHA → vPHA through the staking contract if needed 2. Bridge the vPHA to L2 network via "bridge.phala.network" 3. Open your stakepool 4. Choose **Deposit Collateral** 5. Deposit the required amount (based on GPU type) This collateral: * Does **not** earn rewards * Is used purely as a **security guarantee** * Is fully withdrawable when you retire your GPU and exit properly ## 6. Start Mining and Earn vPHA Rewards After completing: * NFT lock * vPHA collateral deposit * Worker online status in Phala Cloud Your GPU will automatically start the mining process. ### Reward Distribution Details * Rewards are paid in **vPHA** * Rewards are calculated hourly and **Payout happens every hour** at the **end of each hour** ### Reward Weights by GPU Type | GPU Model | Weight | | --------- | ------ | | H100 | 1.0 | | H200 | 1.5 | | B200 | 1.8 | You can monitor these metrics on app.phala.network * Worker status * On-chain status (stakepool interface) * Earnings flow (vPHA rewards updated hourly) # Phala Network Overview Source: https://docs.phala.com/network/index Build confidential, verifiable, and decentralized applications with Phala Network's TEE technology. # 👾 Phala Network Docs Welcome to the official documentation for Phala Network, the decentralized cloud computing protocol that enables confidential smart contracts and secure off-chain computation. ## Quick Start * **[Overview](/overview/phala-network)** - Learn about Phala Network's architecture and core concepts * **[Phala Cloud](/phala-cloud/getting-started/overview)** - Deploy confidential applications in Trusted Execution Environments ## What is Phala Network? Phala Network is a decentralized cloud computing protocol that: * **Provides Privacy**: Execute smart contracts and computations in secure enclaves * **Ensures Verifiability**: All computations are cryptographically verifiable * **Scales Efficiently**: Off-chain execution with on-chain verification * **Maintains Decentralization**: Distributed network of compute providers ## Key Products * **[Phala Cloud](/phala-cloud/getting-started/overview)**: Deploy TEE applications easily * **[Dstack](/dstack/overview)**: Decentralized cloud infrastructure * **[GPU TEE](/phala-cloud/confidential-ai/overview)**: Run AI models in secure environments ## Community Join our growing community of developers and users: * [Discord](https://discord.gg/phala-network) * [GitHub](https://github.com/Phala-Network/) * [Twitter](https://twitter.com/PhalaNetwork) # Block Explorers Source: https://docs.phala.com/network/overview/block-explorers Support resources, endpoints, and compatibility information for Phala Network. | Chain | URL | | --------------------------- | ------------------------------------------------------------------ | | Ethereum Mainnet | [https://etherscan.io/](https://etherscan.io/) | | Phala network (Ethereum L2) | [https://explorer.phala.network/](https://explorer.phala.network/) | # Acquiring PHA Source: https://docs.phala.com/network/overview/get-pha-and-transfer Reference documentation and resources for Phala Network development. Phala tokens exist in multiple environments, and it is important for community members to correctly recognize and verify the token contracts they interact with. **Migrating from Phala/Khala Parachain?** The Phala parachain has been sunset. If you have assets on Khala, Phala parachain, Hydration, or stuck in XCM bridges, see the [Migration Guide](#migrating-from-phala-parachain) section below. ## PHA as an ERC-20 Token **PHA** is an ERC-20 token and can be displayed, transferred, and managed in any **EVM-compatible wallet**, including: * MetaMask * Rabby * Safe (Gnosis Safe) * WalletConnect-compatible wallets This means PHA can be used across the broader Ethereum ecosystem without special configuration. ## What Is vPHA? **vPHA** is the token generated from the **Phala staking contract** on Ethereum.\ Users stake PHA → receive vPHA. vPHA serves several important roles: * **Community governance** * **Staking on Phala L2** * **Required collateral for GPU mining** * **Utility token for future Phala L2 applications** Unlike PHA, vPHA is generally not used for transfers or payments — it represents staked value and governance power. ## How to Add Phala L2 Network to Your Wallet You can add **Phala L2** to any EVM wallet (such as MetaMask or Rabby) by manually entering the official network details. Simply open “Add Network,” fill in: RPC `https://rpc.phala.network`, Chain ID `2035`, Symbol `ETH`, and Explorer `https://explorer.phala.network`, then save. After adding the network, you may also import PHA and vPHA tokens using the correct contract addresses if needed. Once done, your wallet is ready to interact with Phala L2. ## Always Verify Token Contract Addresses For security reasons, community members must always ensure they are interacting with the **correct token contracts**.\ Below are the *official and only valid* contract addresses: ### **ERC-20 Contracts (Ethereum Mainnet)** | Token | Contract Address | | ---------------------- | -------------------------------------------- | | **PHA (ERC20)** | `0x6c5bA91642F10282b576d91922Ae6448C9d52f4E` | | **vPHA (ERC20)** | `0x21d6eC8fc14CaAcc55aFA23cBa66798DAB3a0ec0` | | **PHA Claim Contract** | `0x8b6d7AB0BE8d1f6AE40738C4CFF75134110a777F` | ### **Phala L2 Contracts** | Token | Contract Address | | ------------------- | -------------------------------------------- | | **PHA (Phala L2)** | `0xd63dD43678124DEb6bF38b0d7f8cf0674e85c46E` | | **vPHA (Phala L2)** | `0xFdCEBD2E111D64Fc8a682703Eac62918093ec856` | ## Transfer PHA/vPHA between Ethereum and Phala L2 You can transfer PHA and vPHA between Ethereum and Phala L2 using [Superbridge](https://bridge.phala.network). ### Bridge from Ethereum to Phala L2 1. Go to [bridge.phala.network](https://bridge.phala.network) 2. Connect your wallet (MetaMask, Rabby, etc.) 3. Select **Ethereum → Phala L2** 4. Enter the amount of PHA or vPHA to bridge 5. Confirm the transaction and pay gas fees ### Bridge from Phala L2 to Ethereum 1. Go to [bridge.phala.network](https://bridge.phala.network) 2. Connect your wallet and switch to **Phala L2** network 3. Select **Phala L2 → Ethereum** 4. Enter the amount to bridge 5. Confirm the transaction **Gas fees required:** When bridging from Phala L2 to Ethereum, you need ETH on Phala L2 for gas. Bridge a small amount of ETH (\~0.001 ETH is sufficient) from Ethereum to Phala L2 first. ## Get your Phala Tokens (PHA) If you do not hold any PHA, you can purchase PHA from any of the following exchanges: Binance Trade PHA on Binance → Kraken Trade PHA on Kraken → OKX Trade PHA on OKX → Huobi Trade PHA on Huobi → KuCoin Trade PHA on KuCoin → Gate.io Trade PHA on Gate.io → Not all brokers may be available in your jurisdiction. Note that some brokers require you to have an account with them for a certain period before you can transfer your funds to another wallet, such as the [Phala Wallet](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fphala.api.onfinality.io%2Fpublic-ws#/accounts). *** ## Migrating from Phala Parachain The Phala/Khala parachain has been sunset. PHA tokens now exist primarily on Ethereum and Phala L2. If you have assets on the old parachain or related networks, follow the guides below to migrate them. ### Claim Assets from Khala/Phala Parachain If you had PHA staked, delegated in pools/vaults, or in your wallet on the original Khala or Phala parachain: 1. Go to the **Khala Assets Portal**: [app.phala.network/khala-assets](https://app.phala.network/khala-assets) 2. Connect your Ethereum wallet that corresponds to your old Substrate address 3. Follow the claim process to receive your assets on Phala L2 **Requirements for claiming:** * An Ethereum wallet with ETH for gas fees * Assets will be claimed to Phala L2 * Add Phala L2 to your wallet first (see [How to Add Phala L2](#how-to-add-phala-l2-network-to-your-wallet)) **Resources:** * [Claim System Tutorial](https://forum.phala.network/t/phala-assets-claim-system-now-alive/4002) * [Sunset Announcement](https://forum.phala.network/t/proposal-sunsetting-phala-parachain-and-migrating-to-ethereum-l2/3999) ### Migrate PHA from Hydration If your PHA is on Hydration (formerly Basilisk), a Polkadot parachain with EVM-style addresses (0x...): Hydration addresses look like Ethereum addresses but are NOT Ethereum accounts. The Phala snapshot did not include Hydration balances, so the claim UI will show "0 PHA". **Manual Claim Process:** Since the Phala network is sunset, the team processes Hydration claims manually: 1. **Transfer tokens to the burning address:** ``` 13UVJyLnbVp9RBZYFwFGyDvVd1y27Tt8tkntv6Q7JVPhFsTB ``` 2. **Provide verification:** * Your Hydration address (the one that held PHA) * Your Ethereum address as recipient * Signature data for ownership verification * Link to the burning transaction 3. **Alternative verification method:** Send three batches to the burning address to prove ownership: * First: 1 PHA * Second: 2 PHA * Third: All remaining PHA 4. **Contact support** via the [Phala Forum](https://forum.phala.network/t/manual-claim-guide-for-hydration-users/4003) or support channels After verification, the team will execute the claim to your Ethereum address. ### Stuck XCM Transfers (Moonbeam, etc.) If you have PHA stuck in an XCM transfer from Phala to Moonbeam or other parachains: Since the Phala parachain is sunset, XCM bridges are no longer operational. The team can help claim stuck assets manually. **Process:** 1. Verify ownership of your Phala address: [Verification Guide](https://www.notion.so/phalanetwork/How-to-verify-your-Phala-address-2bc0317e04a1809291c7eb6f3e33ad6b) 2. Provide: * Signature proving address ownership * Your Ethereum address as recipient 3. Contact support with verification proof After verification, the team will issue the transfer to your Ethereum address. ### Bridge PHA from BSC (Binance Smart Chain) For PHA on Binance Smart Chain (BEP-20), there is no direct bridge to Ethereum ERC-20. Use one of these options: **Option 1: Via Binance Exchange** 1. Deposit your BSC PHA to Binance (the centralized exchange) 2. Withdraw as ERC-20 PHA to your Ethereum wallet **Option 2: Swap on DEX** 1. Go to [PancakeSwap](https://pancakeswap.finance/?chain=bsc) 2. Swap your BSC PHA for another token (e.g., USDT, BNB) 3. Bridge that token to Ethereum using a cross-chain bridge 4. Swap back to PHA on Ethereum ### Already Have PHA on Ethereum? If your PHA tokens are already on Ethereum (ERC-20), there's nothing you need to do. Your tokens are in the correct format. You can: * [Stake PHA to receive vPHA](/network/overview/phala-staking) * Use PHA with any DeFi protocol on Ethereum * Bridge to Phala L2 for lower gas fees # Phala Governance Source: https://docs.phala.com/network/overview/phala-governance Participate in Phala Network governance using PHA tokens and democratic processes. Phala uses **Snapshot** as its off-chain governance platform, allowing community members to participate in proposals and voting without paying gas fees. Anyone holding **vPHA** (the governance token) can take part in decision-making. ## How Voting Works Snapshot takes periodic “snapshots” of wallet balances to determine voting power.\ Your **vPHA balance at the snapshot block** decides how many votes you have.\ No tokens are locked, and no transactions are sent—governance is **gas-free**. ## How to Participate 1. Go to the Phala Snapshot space:\ **[https://snapshot.box/#/s:phala-network.eth](https://snapshot.box/#/s:phala-network.eth)** 2. Connect your wallet (MetaMask, Rabby, OKX Wallet, etc.). 3. Browse active proposals:\ Each proposal includes: * Title & description * Voting options * Voting period * Impact on the network 4. Cast your vote:\ Choose an option and sign a message.\ **No gas fee** is involved, as Snapshot uses off-chain signatures. ## Creating a Proposal (for eligible members) 1. Click **New Proposal** 2. Fill in: * Title * Summary and motivation * Voting options (Yes/No/Multiple Choice) * Voting duration 3. Submit and sign—your proposal will go live instantly. ## Best Practices for Governance * Hold **vPHA** to participate effectively. * Read proposal details carefully before voting. * Participate early in case you need to switch networks or adjust your wallet. * Discuss proposals on the Discord before they go live. Phala’s Snapshot governance is designed to be open, gas-free, and accessible.\ If you hold vPHA, no matter it's on Ethereum or Phala L2, you are an active part of the network—your vote directly shapes Phala’s future. # Phala Network Source: https://docs.phala.com/network/overview/phala-network Overview of Phala Network's decentralized computing infrastructure and features. Confidential AI Inference stands as a cornerstone for safeguarding sensitive data and ensuring secure AI model execution in Web3. Through the implementation of LLM models within Trusted Execution Environments (TEE), Phala Network delivers private and verifiable AI computations. This robust approach addresses fundamental challenges in the Web3 ecosystem: data privacy protection, secure execution guarantees, and computational verifiability. Such capabilities are essential for applications where the protection of user data and model integrity is paramount. ## **What Sets Phala Apart:** * **We Don't Trust Any Provider**: Our **security model** goes beyond traditional cloud solutions (e.g., AWS, Azure, GCP). **Phala does not trust any cloud platform, hardware provider, or even its users,** ensuring true zero-trust. * **User-Friendly & Affordable**: With **easy integration**, developers can migrate their **Web2 software** into a zero-trust environment. By using TEE as part of our hybrid infrastructure, developers can select the **level of proof** they need based on their use case, making the system both flexible and affordable. * **Web3 Standard Compatibility**: Phala is fully **decentralized, privacy-focused**, and auditable. Our SDK enables your programs to **easily integrate with blockchains**, providing seamless co-processing and interaction with Web3 systems. ## **How It Works** * Phala introduces a **new root-of-trust** beyond traditional hardware-based models. * We implement a combination of **TEE, MPC, ZKP (FHE)** and **blockchain game theory** to build this **new root-of-trust**. * Our system offers **auditable computation**, ensuring that anyone can verify the integrity of execution results, creating a **tamper-proof** environment. ## **The Solution** Phala provides a comprehensive suite of tools and infrastructure to make zero-trust computing **easy to access**, **build**, and **verify**: * **Easy Access to Computers**: [Phala Cloud](https://cloud.phala.com/register?invite=PHALAWIKI) provides access to TEE hardware, including **Intel TDX**, **Intel SGX**, **AMD SEV**, and **NVIDIA H100/H200 (TEE)**, offering secure and verifiable computation at scale. Phala Cloud supports deployment of any Docker application into TEE environments, making it easy to migrate existing workloads to confidential computing. * **Easy to Build**: [Dstack](/dstack/overview) is the TEE SDK developed by Phala and Flashbots jointly * **Docker / VM migration into TEE**, allowing developers to move existing workloads into a zero-trust environment. * **Serverless edge functions**: Leverage pre-built templates to create **serverless, privacy-preserving functions** that run in secure TEEs. * **Easy to Prove**: Phala offers **on-chain attestation** for auditable logs of off-chain computations, ensuring integrity and transparency. Developers can prove the correctness of their computations in a verifiable, decentralized way. ## **The Future is Here (Use Cases)** * **Web2 to Web3**: Easily transition Web2 software to **Web3 standards**, with seamless connectivity to smart contracts across blockchains. * **Intelligence-Based Economy**: Solutions like **MEV, Intent-centric applications**. Flashbots uses TEE for decentralized MEV-boost on Ethereum. Uniswap is using TEE to build a DeFi-native Ethereum L2, becoming a hub for liquidity across chains. * **Defense in Depth**: Projects like **Lit Protocol (MPC)**, **Zama (FHE)**, and **Scroll (ZKP)** use TEE to enhance their security and privacy, strengthening their defenses. * **Decentralized AI**: With **NVIDIA's GPU TEEs**, decentralized AI platforms like **NEAR AI, Sentient, Zero Gravity, Ritual, Morpheus**, and **Autonolas** use TEEs to run LLMs in secure, verifiable, and privacy-focused environments as part of their infrastructure. # phala staking Source: https://docs.phala.com/network/overview/phala-staking Stake PHA tokens to earn rewards and support network governance. ## What Is the Phala Staking Contract? The Phala Staking Contract lives on **Ethereum Mainnet** and allows users to convert **PHA to vPHA**.\ In simple terms: * You deposit **PHA** into the staking contract * The contract mints and sends you **vPHA** (an ERC-20 token) * **vPHA** is used for: * Community governance (Snapshot voting) * Staking & collateral on Phala L2 * GPU mining (required collateral + reward token) In short:\ **PHA = liquid token**,\ **vPHA = governance & staking token** for the Phala ecosystem. ## Official Contract Addresses (Verify Before Use) Always double-check that you are interacting with the correct contracts: * **PHA (ERC20)**: `0x6c5bA91642F10282b576d91922Ae6448C9d52f4E` * **vPHA (ERC20)**: `0x21d6eC8fc14CaAcc55aFA23cBa66798DAB3a0ec0` > ⚠️ Only use the addresses above.\ > Do not stake or transfer tokens to any unknown or unverified contract. ## How to Stake PHA and Receive vPHA > The process works with any EVM wallet such as MetaMask, Rabby, Safe, etc.\ > official Phala staking page: [https://app.phala.network/staking](https://app.phala.network/staking) . ### **Step 1 — Prepare Your Wallet** 1. Switch wallet to **Ethereum Mainnet** 2. Make sure your wallet has: * **PHA (ERC20)** — the tokens you want to stake * **ETH** — to pay Ethereum gas fees ### **Step 2 — Connect to the Phala Staking Page** 1. Open the official Phala staking page 2. Click **“Connect Wallet”** 3. Approve the connection in your wallet ### **Step 3 — Stake PHA → Receive vPHA** 1. Enter the amount of **PHA** you want to stake 2. First-time users must click **Approve** to allow the contract to use PHA 3. Once approved, click **Stake** 4. Confirm the transaction in your wallet After the transaction is confirmed: * Your PHA is deposited into the staking contract * You receive vPHA equal in value to the PHA you staked, based on the real-time vPHA-PHA exchange rate * As the staking contract accumulates rewards, the exchange rate increases over time, meaning vPHA represents a growing amount of PHA when you unstake. To display vPHA, manually add the ERC-20 contract address in your wallet. *** ## Unstaking vPHA to PHA Unstaking converts your vPHA back to PHA. The process takes **21 days** to complete. ### Step 1: Initiate Unstaking 1. Go to [app.phala.network/staking](https://app.phala.network/staking) 2. Connect your wallet (must be on **Ethereum Mainnet**) 3. Click **Unstake** 4. Enter the amount of vPHA to unstake 5. Confirm the transaction and pay gas fees After the transaction confirms, your vPHA enters a 21-day unlock period. ### Step 2: Wait for Unlock Period * Your tokens are locked for **21 days** * You can track the unlock countdown in the staking dashboard * You cannot cancel an unstaking request once initiated ### Step 3: Claim Your PHA After the 21-day period ends: 1. Return to [app.phala.network/staking](https://app.phala.network/staking) 2. Connect your wallet 3. You'll see your unlocked PHA available to claim 4. Click **Claim** to receive your PHA 5. Confirm the transaction **You must manually claim your PHA** after the unlock period. Tokens do not automatically transfer to your wallet. ### Unstaking from Phala L2 If your vPHA is on Phala L2 (not Ethereum Mainnet): 1. First, bridge your vPHA to Ethereum using [bridge.phala.network](https://bridge.phala.network) 2. You need ETH on Phala L2 for gas fees (\~0.001 ETH is sufficient) 3. Once on Ethereum, follow the unstaking steps above **Need ETH on Phala L2?** Bridge a small amount from Ethereum first using [bridge.phala.network](https://bridge.phala.network). ### Troubleshooting **"Insufficient ETH" error when bridging from L2:** * You need ETH on Phala L2 for gas, not just on Ethereum * Bridge \~0.001 ETH from Ethereum to Phala L2 first **Can't see vPHA in portfolio:** * Check if your vPHA is on Phala L2 or Ethereum * Add the vPHA token contract to your wallet: `0x21d6eC8fc14CaAcc55aFA23cBa66798DAB3a0ec0` **vPHA balance seems lower than expected:** * vPHA represents staked value at a conversion rate that increases over time * Check your deposit history at the bottom of the staking page * If you provided liquidity on Uniswap, your vPHA won't show in the Phala app *** ## Safety Notes Always verify: * Contract addresses before any transaction * Transaction prompts match your intended action * You are connected to **Ethereum Mainnet** Never send tokens to unknown addresses. Only use the official contract addresses listed in this documentation. # Phala Treasury Source: https://docs.phala.com/network/overview/phala-treasury Participate in Phala Network governance using PHA tokens and democratic processes. The Treasury is a pot of funds collected through transaction fees, slashing, Secure Worker mining rate (20%) , etc. The funds held in the Treasury can be spent by making a spending proposal that, if approved by the community, will be issued to the receipents. The Treasury attempts to spend as many proposals in the queue as it can without running out of funds. Proposals may consist of (but are not limited to): * Infrastructure deployment and continued operation. * Network security operations (monitoring services, continuous auditing). * Ecosystem provisions (collaborations with friendly chains). * Marketing activities (advertising, paid features, collaborations). * Community events and outreach (meetups, pizza parties, hackerspaces). * Software development (wallets and wallet integration, clients and client upgrades). The Treasury comes from: 70% of the PHA will be used for Secure Worker Mining, and following the progress of the Phala network, 20% of the mining rewards generated by each block will be used to fund treasury. # Attestation Fields Reference Source: https://docs.phala.com/phala-cloud/attestation/attestation-fields Reference guide for Intel TDX attestation quote fields and their meanings. This reference explains each field in an Intel TDX attestation quote. Use this when you need to understand or verify specific measurements. ## Example Quote Here's a typical attestation quote from Phala Cloud: ```json theme={"system"} { "tee_tcb_svn": "06010300000000000000000000000000", "mr_seam": "5b38e33a6487958b72c3c12a938eaa5e3fd4510c51aeeab58c7d5ecee41d7c436489d6c8e4f92f160b7cad34207b00c1", "mr_signer_seam": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "seam_attributes": "0000000000000000", "td_attributes": "0000001000000000", "xfam": "e702060000000000", "mr_td": "c68518a0ebb42136c12b2275164f8c72f25fa9a34392228687ed6e9caeb9c0f1dbd895e9cf475121c029dc47e70e91fd", "mr_config_id": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "mr_owner": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "mr_owner_config": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "rt_mr0": "85e0855a6384fa1c8a6ab36d0dcbfaa11a5753e5a070c08218ae5fe872fcb86967fd2449c29e22e59dc9fec998cb6547", "rt_mr1": "9b43f9f34a64bc7191352585be0da1774a1499e698ba77cbf6184547d53d1770d6524c1cfa00b86352f273fc272a8cfe", "rt_mr2": "7cc2dadd5849bad220ab122c4fbf25a74dc91cc12702447d3b5cac0f49b2b139994f5cd936b293e5f0f14dea4262d668", "rt_mr3": "2c482b5b34f6902293bc203696f407241bfa319d2410a04c604d1021888d6028bf4bd280ff859ee270a0429aac5f0d82", "report_data": "afab9790acb13c4c651c1933a22b5f0663ef22927120dd08cc8291d7e0912d8b1c36eb75cf661a64735042f8e81bbe42cb9ab310ca95bf8d36c44cb8835c901f" } ``` ## Hardware Fields ### tee\_tcb\_svn **Trusted Computing Base Security Version Number** Security patch level for TEE hardware components. Verify this matches the latest version to ensure no known vulnerabilities. ### mr\_seam (MRSEAM) **Measurement of TDX-Module** Cryptographic hash of the SEAM (Secure Arbitration Mode) firmware. Confirms the TDX firmware hasn't been tampered with. ### mr\_td (MRTD) **Measurement Register for Trust Domain** Hash of the initial Trust Domain memory and configuration. Proves the TD was initialized with expected contents. ### seam\_attributes **SEAM Firmware Attributes** Configuration flags for SEAM firmware. Always zeros for Intel TDX. ### td\_attributes **Trust Domain Attributes** Configuration flags for the TD. Verifies the TD has expected security settings. ### xfam **eXtended Feature Activation Mask** Controls which CPU extended features are accessible to the TD. Ensures proper security boundaries. ## Runtime Measurement Registers (RTMRs) These registers contain hash chains of boot components. ### RTMR0 **Virtual Hardware Measurement** Measures the virtual hardware environment provided to the CVM. ### RTMR1 **Linux Kernel Measurement** Hash of the Linux kernel loaded during boot. ### RTMR2 **Kernel Parameters Measurement** Measurement of kernel command line and initrd. ### RTMR3 **Application Measurement** dstack-specific measurements including: * **app-id**: Relevant for KMS key derivation * **compose-hash**: Hash of your Docker Compose configuration * **instance-id**: Unique CVM instance identifier * **key-provider**: Who distributed the root filesystem encryption keys ## RTMR3 Hash Chain RTMR3 uses a hash chain mechanism: ``` RTMR3_new = SHA384(RTMR3_old || SHA384(event)) ``` Each application component (compose-hash, instance-id, key-provider) is hashed individually, then chained together. Verify the final RTMR3 value using the [rtmr3-calculator](https://rtmr3-calculator.vercel.app/) tool. ## Application-Specific Fields ### report\_data (reportData) **User-Provided Data (64 bytes)** Custom data you attach to the quote. Common uses: * **Public keys**: Bind a public key to prove you control the private key * **Nonces**: Challenge-response protocols * **Application state**: Hash of configuration or state ### mr\_config\_id, mr\_owner, mr\_owner\_config **Configuration and Ownership** Usually all zeros for dstack deployments. Reserved for advanced attestation scenarios. ## Which Fields Should I Verify? Different verification levels require checking different fields: **Quick Verification:** * `report_data` - Your challenge or public key * Use Cloud API for hardware verification See [Quickstart](/phala-cloud/attestation/quickstart) for step-by-step guide. **Application Verification:** * `rt_mr3` (RTMR3) - Your compose-hash * `report_data` - Custom data binding * Use Cloud API for hardware and OS See [Verify Your Application](/phala-cloud/attestation/verify-your-application) for complete guide. **Complete Platform Verification:** * `mr_td` (MRTD) - OS initial state * `rt_mr0` through `rt_mr2` - OS boot chain * `rt_mr3` (RTMR3) - Application and KMS binding * `tee_tcb_svn` - Security patch level * `mr_seam` - TDX firmware measurement See [Verify the Platform](/phala-cloud/attestation/verify-the-platform) for complete guide. **Security Audit:** * All measurements above * On-chain governance verification * Reproducible OS builds * KMS attestation See [Complete Chain of Trust](/phala-cloud/attestation/chain-of-trust) for checklist. ## Resources * **[dstack Attestation Guide](https://github.com/Dstack-TEE/dstack/blob/master/attestation.md)** - How dstack implements attestation * **[Intel TDX Specification](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html)** - Official Intel documentation * **[RTMR3 Calculator](https://rtmr3-calculator.vercel.app/)** - Verify RTMR3 values * **[Verification Script](https://github.com/Dstack-TEE/dstack-examples/blob/main/attestation/rtmr3-based/verify.py)** - Example verification code ## Next Steps Verify your first quote Prove your exact code is running Verify OS, KMS, and infrastructure Complete verification with no trust assumptions # Complete Chain of Trust Source: https://docs.phala.com/phala-cloud/attestation/chain-of-trust Complete security checklist ensuring no verification gaps exist. Hardware attestation alone isn't enough — anyone in the supply chain can still compromise your security. This guide covers every verification item needed for trustless execution and proves they're sufficient together. ## Why Complete Verification Matters Hardware attestation alone proves your code runs in genuine TEE hardware. But anyone in the supply chain could still compromise your security: **Cloud provider** could substitute OS images with backdoored versions. **Docker registry** could serve malicious container images. **Malicious KMS** could leak encryption keys and TLS certificates. **Network attacker** could impersonate your domain with fraudulent certificates. **Unauthorized developer** could deploy malicious code updates. Each of these attacks bypasses hardware attestation. You need to verify the **complete chain of trust** from source code to running CVM. ## The Complete Trust Chain ```mermaid theme={"system"} flowchart TB subgraph L1[What You Control] B[Application: compose-hash in RTMR3] G[Custom Data: reportData field] end subgraph L2[Platform Infrastructure] A[OS: MRTD, RTMR0-2] C[Keys: key-provider in RTMR3] D[Network: TLS certificate binding] end subgraph L3[Authorization] E[Governance: Smart contracts] end L1 --> Q[TDX Quote: Cryptographically signed package] L2 --> Q L3 --> Q Q --> V[Verification: Compare measurements against expected values] ``` Six independent trust chains are recorded in one cryptographically signed attestation quote. Verification confirms each measurement matches your expected values. ## Application Verification Checklist These items prove your specific application code runs unmodified: ### ☐ reportData Binding **What it prevents:** Replay attacks where old attestation quotes are reused. **How it works:** The 64-byte `reportData` field contains your custom challenge (nonce) or public key. The TEE includes this in the signed quote. **Why it's necessary:** Without reportData verification, an attacker could replay an old valid quote instead of generating a fresh one. Challenge-response proves the quote was generated specifically for you. **How to verify:** See [Quickstart: Step 2](/phala-cloud/attestation/quickstart#step-2-verify-the-challenge) ```javascript theme={"system"} // Verify reportData contains your challenge const reportData = Buffer.from(quote.quote.reportData, 'hex'); const receivedChallenge = reportData.subarray(0, 32); assert(receivedChallenge.equals(challenge)); ``` ### ☐ compose-hash Matching **What it prevents:** Code substitution attacks where different Docker images are deployed. **How it works:** Your entire docker-compose.yaml (including all image SHA256 digests) is hashed into RTMR3. The CVM verifies image digests before pulling from the registry. **Why it's necessary:** Without compose-hash verification, a compromised registry could serve malicious images. The TEE prevents this by verifying digests match the compose-hash recorded in RTMR3. **How to verify:** See [Verify Your Application: Complete Verification](/phala-cloud/attestation/verify-your-application#complete-verification-trustless) ```javascript theme={"system"} import { getComposeHash } from '@phala/dstack-sdk'; // Calculate expected hash const expectedHash = getComposeHash(JSON.parse(appCompose)); // Extract from RTMR3 event log const events = JSON.parse(quote.event_log); const attestedHash = events.find(e => e.event === 'compose-hash').event_payload; assert(expectedHash === attestedHash); ``` ### ☐ Docker Digest Pinning **What it prevents:** Image tampering where registry serves different images for the same tag. **How it works:** All images must use SHA256 digests (not tags) in docker-compose.yaml. Tags like `nginx:latest` are mutable. Digests are immutable. **Why it's necessary:** Without digest pinning, the compose-hash would be meaningless. A tag could point to different images over time, making verification impossible. **How to verify:** See [Verify Your Application: Understanding Image Digests](/phala-cloud/attestation/verify-your-application#understanding-image-digests) ```yaml theme={"system"} # Good: Immutable digest image: nginx@sha256:eee5eae48e79b2e75178328c7c585b89d676eaae616f03f9a1813aaed820745a # Bad: Mutable tag image: nginx:latest ``` ### ☐ RTMR3 Event Log Replay **What it prevents:** Event log tampering where attacker modifies compose-hash or key-provider events. **How it works:** RTMR3 uses a hash chain (RTMR3 = SHA384(RTMR3 || event)). You replay all events to recalculate RTMR3 and verify it matches the quote. **Why it's necessary:** Without replay verification, you can't trust the event payloads. An attacker could modify events and claim different compose-hash values. **How to verify:** See [Verify Your Application: Step 4](/phala-cloud/attestation/verify-your-application#step-4-replay-rtmr3-event-log) ```javascript theme={"system"} // Replay events to verify RTMR3 const rtmrs = quoteResponse.replayRtmrs(); assert(rtmrs.rtmr3 === quoteResponse.quote.rt_mr3); ``` ### ☐ Image Provenance **What it prevents:** Supply chain attacks where build infrastructure is compromised. **How it works:** Link Docker image digests back to source code using Sigstore (GitHub-endorsed builds) or reproducible builds. **Why it's necessary:** The compose-hash proves which images ran, but not whether those images came from your source code. Provenance links images to specific git commits. **How to verify:** See [Verify Your Application: Linking Images to Source Code](/phala-cloud/attestation/verify-your-application#linking-images-to-source-code) ### ☐ Application Governance **What it prevents:** Unauthorized updates where malicious developers deploy unapproved versions. **How it works:** Only compose hashes whitelisted in the `DstackApp` smart contract can boot. **Why it's necessary:** Without on-chain governance, any developer with deploy access could push malicious code. The contract enforces authorization. **How to verify:** See [Verify Your Application: Step 6](/phala-cloud/attestation/verify-your-application#step-6-verify-on-chain-governance-optional) ```bash theme={"system"} cast call "allowedComposeHashes(bytes32)" ``` ## Platform Verification Checklist These items prove Phala's infrastructure is secure: ### ☐ Hardware Measurements **What it prevents:** Fake TEE hardware or compromised firmware. **How it works:** Intel signs the TDX quote with hardware root of trust. The signature proves genuine Intel TDX hardware generated the quote. **Why it's necessary:** Without hardware verification, an attacker could fake a quote using software simulation. The cryptographic signature prevents this. **How to verify:** Use Phala Cloud API (verifies Intel signature automatically) or see [Verify the Platform: Hardware & Firmware](/phala-cloud/attestation/verify-the-platform#1-hardware--firmware-verification) ```bash theme={"system"} curl -X POST "https://cloud-api.phala.com/api/v1/attestations/verify" \ -H "Content-Type: application/json" \ -d '{"hex": "YOUR_QUOTE_HEX"}' ``` ### ☐ OS Measurements **What it prevents:** Backdoored OS images with malware or data exfiltration code. **How it works:** MRTD and RTMR0-2 contain hashes of virtual firmware, hardware config, kernel, and boot parameters. These measurements are recorded during boot. **Why it's necessary:** Without OS verification, a compromised cloud provider could substitute the OS with a backdoored version. Measurements make OS tampering detectable. **How to verify:** See [Verify the Platform: Operating System Verification](/phala-cloud/attestation/verify-the-platform#2-operating-system-verification) ```bash theme={"system"} # Calculate expected measurements cargo run --bin dstack-mr measure -c 4 -m 4G ../dstack-0.5.4/metadata.json # Compare against quote ``` ### ☐ OS Reproducibility **What it prevents:** Compromised build infrastructure injecting malicious code. **How it works:** The dstack OS is built from [meta-dstack](https://github.com/Dstack-TEE/meta-dstack) using Yocto. Anyone can rebuild from a specific git commit and get identical measurements. **Why it's necessary:** Without reproducibility, you must trust Phala's pre-built releases. Reproducible builds let you verify independently. **How to verify:** See [Verify the Platform: Step 2 Option B](/phala-cloud/attestation/verify-the-platform#step-2-download-or-build-os-image) ```bash theme={"system"} git clone https://github.com/Dstack-TEE/meta-dstack.git cd meta-dstack git checkout # Follow reproducible build instructions ``` ### ☐ OS Whitelist **What it prevents:** Unauthorized OS versions with unknown security properties. **How it works:** Only OS image hashes whitelisted in `DstackKms.allowedOsImages` can boot. **Why it's necessary:** Without on-chain governance, any OS version could run. The whitelist ensures only audited OS versions are used. **How to verify:** See [Verify the Platform: Step 5](/phala-cloud/attestation/verify-the-platform#step-5-verify-on-chain-os-whitelist) ```bash theme={"system"} cast call "allowedOsImages(bytes32)" ``` ### ☐ KMS Identity **What it prevents:** Malicious KMS instances leaking encryption keys and TLS certificates. **How it works:** The KMS root CA public key hash is recorded in RTMR3 as the `key-provider` event. This binds your app to a specific KMS instance. **Why it's necessary:** All your keys (disk encryption, TLS, signing keys) derive from the KMS root keys. A malicious KMS compromises everything. RTMR3 binding prevents silent KMS substitution. **How to verify:** See [Verify the Platform: Key Management Verification](/phala-cloud/attestation/verify-the-platform#3-key-management-verification) ```javascript theme={"system"} const events = JSON.parse(quote.event_log); const keyProvider = events.find(e => e.event === 'key-provider'); console.log('KMS ID:', JSON.parse(keyProvider.event_payload).id); ``` ### ☐ KMS Attestation **What it prevents:** Compromised KMS running malicious code. **How it works:** The KMS itself is a TEE instance with its own attestation quote. You verify the KMS's hardware, OS, and application code independently. **Why it's necessary:** Binding to a KMS in RTMR3 is meaningless if that KMS is compromised. You must verify the KMS's complete chain of trust. **How to verify:** See [Verify the Platform: Step 2](/phala-cloud/attestation/verify-the-platform#step-2-verify-the-kms-itself) ### ☐ KMS Governance **What it prevents:** Unauthorized KMS instances with unknown security properties. **How it works:** Only KMS instances with whitelisted "aggregated MR" (combined hash of all measurements) can operate. **Why it's necessary:** Without governance, any KMS instance could be used. The whitelist ensures only audited KMS instances are trusted. **How to verify:** See [Verify the Platform: Step 3](/phala-cloud/attestation/verify-the-platform#step-3-verify-kms-governance) ```bash theme={"system"} cast call "kmsAllowedAggregatedMrs(bytes32)" ``` ### ☐ TLS Certificate Binding **What it prevents:** Certificate impersonation attacks where attacker issues fraudulent certificates. **How it works:** The TEE generates TLS certificates and publishes evidence files at `/evidences/`. The quote's `reportData` contains a hash of the evidence, cryptographically binding certificate to quote. **Why it's necessary:** Without binding, anyone (domain owner, CA, cloud provider) could issue valid certificates for your domain. Binding proves the TEE controls the certificate. **How to verify:** See [Domain Attestation](/phala-cloud/networking/domain-attestation) ```javascript theme={"system"} // Verify evidence hash in reportData const evidenceHash = crypto.createHash('sha256').update(evidenceFiles).digest(); assert(quote.reportData.includes(evidenceHash)); ``` ### ☐ CAA Records **What it prevents:** Unauthorized certificate issuance even if DNS is compromised. **How it works:** CAA DNS records restrict which Certificate Authorities can issue certificates for your domain. **Why it's necessary:** Without CAA records, any CA could issue certificates. CAA records add defense-in-depth against compromised CAs or DNS attacks. **How to verify:** See [Domain Attestation: CAA Records](/phala-cloud/networking/domain-attestation) ```bash theme={"system"} dig CAA yourdomain.com ``` ## Why This Is Sufficient The verification checklist covers the complete Trusted Computing Base (TCB). Here's why no gaps exist: **Hardware root of trust:** Intel's signature proves genuine TDX hardware. All measurements are recorded by this hardware during boot before any software runs. **Measurement immutability:** Once recorded in the signed quote, measurements cannot be modified. Change one byte anywhere in the TCB, and measurements won't match. **Cryptographic binding:** Each layer binds to the next through measurements. OS binds to hardware (MRTD, RTMR0-2). Application binds to OS (RTMR3). Keys bind to KMS (key-provider). Certificates bind to quote (reportData). **Complete coverage:** The checklist covers all TCB components. Nothing runs in the CVM that isn't measured or verified. **Governance enforcement:** Smart contracts ensure only authorized versions can boot. On-chain authorization prevents malicious updates even with compromised infrastructure. **No trust assumptions:** With complete verification, you don't need to trust Phala Cloud, Docker registries, build infrastructure, or Certificate Authorities. Everything is cryptographically verified. ## Security Proof **Theorem:** If all checklist items pass, the CVM runs your exact code in genuine TEE hardware with no security gaps. **Proof by contradiction:** Assume an attacker compromises the CVM despite passing all checks. The attacker must modify at least one component (hardware, firmware, OS, application, keys, or certificates). But: 1. **Hardware/firmware tampering** changes MRTD, mr\_seam, or tee\_tcb\_svn, failing hardware verification. 2. **OS tampering** changes RTMR0-2, failing OS measurement verification. 3. **Application tampering** changes compose-hash in RTMR3, failing compose-hash verification. 4. **Image substitution** changes image digests, failing the TEE's digest check before pulling. 5. **KMS substitution** changes key-provider in RTMR3, failing KMS identity verification. 6. **Certificate impersonation** fails because evidence hash won't match reportData. 7. **Unauthorized updates** fail on-chain governance checks. Any modification causes verification to fail. Therefore, if all checks pass, no compromise is possible. QED. ## Complete Verification Checklist Use this checklist for security audits. All items must pass for complete trustless verification: **Application Layer (What You Control):** * [ ] reportData contains expected challenge/public key * [ ] compose-hash matches calculated hash * [ ] All Docker images use SHA256 digests (not tags) * [ ] RTMR3 event log replays to quoted RTMR3 * [ ] Image digests link to audited source code (Sigstore or reproducible builds) * [ ] compose-hash is whitelisted in DstackApp contract (if using governance) **Platform Layer (Infrastructure):** * [ ] TDX quote signature is valid (Intel root certificates) * [ ] `tee_tcb_svn` matches latest security patches * [ ] MRTD and RTMR0-2 match calculated OS measurements * [ ] VM config (CPU, memory, GPU) matches deployment * [ ] OS image hash is whitelisted in DstackKms contract * [ ] (Optional) OS rebuilt reproducibly from source * [ ] KMS ID from `key-provider` event is known and trusted * [ ] KMS's own attestation quote is valid * [ ] KMS aggregated MR is whitelisted in DstackKms contract **Network Layer (Custom Domains Only):** * [ ] TLS certificate fingerprint matches served certificate * [ ] Evidence files at `/evidences/` cryptographically bind to quote * [ ] CAA DNS records restrict certificate issuance **Governance Layer:** * [ ] Smart contract addresses are verified and trusted * [ ] Contract permissions match security policy * [ ] Contract ownership and upgrade mechanisms are understood ## Verification Workflow ```mermaid theme={"system"} graph TD A[Start] --> B{Need trustless execution?} B -->|No| C[Quick Verification] B -->|Yes| D[Complete Verification] C --> C1[Verify reportData] C --> C2[Verify compose-hash] C --> C3[Use Cloud API for hardware] C1 --> E[Done - Trust Phala] C2 --> E C3 --> E D --> D1[Application Verification] D --> D2[Platform Verification] D1 --> D1A[reportData + compose-hash] D1 --> D1B[RTMR3 event log replay] D1 --> D1C[Image provenance] D1 --> D1D[On-chain governance] D2 --> D2A[Hardware signature] D2 --> D2B[OS measurements] D2 --> D2C[KMS attestation] D2 --> D2D[TLS binding] D1A --> F[All checks pass?] D1B --> F D1C --> F D1D --> F D2A --> F D2B --> F D2C --> F D2D --> F F -->|Yes| G[Done - Trustless verified] F -->|No| H[Investigate failure] ``` ## Tools and Resources * **[Quickstart](/phala-cloud/attestation/quickstart)** - Quick verification (reportData + Cloud API) * **[Verify Your Application](/phala-cloud/attestation/verify-your-application)** - Complete application verification * **[Verify the Platform](/phala-cloud/attestation/verify-the-platform)** - Complete platform verification * **[Attestation Fields](/phala-cloud/attestation/attestation-fields)** - Field reference * **[Domain Attestation](/phala-cloud/networking/domain-attestation)** - TLS certificate verification * **[trust-center](https://github.com/Phala-Network/trust-center)** - Reference implementation * **[dstack SDK](https://www.npmjs.com/package/@phala/dstack-sdk)** - `getComposeHash()` and `replayRtmrs()` * **[meta-dstack](https://github.com/Dstack-TEE/meta-dstack)** - Reproducible OS builds * **[Smart Contracts](https://github.com/Dstack-TEE/dstack/tree/master/kms/auth-eth/contracts)** - DstackKms and DstackApp ## Real-World Example For a complete implementation of the full chain of trust, see how Confidential AI verifies: * Hardware stack (NVIDIA GPUs + Intel TDX) * OS integrity from reproducible builds * Application code via compose-hash * KMS trust for key derivation * Request/response integrity signatures Learn more: **[Confidential AI Verification](/phala-cloud/confidential-ai/verify/overview)** ## Next Steps Quick verification guide Prove your exact code is running Verify OS, KMS, and infrastructure Understand all quote fields # Become a Featured Builder Source: https://docs.phala.com/phala-cloud/attestation/feature-builder How to become a Featured Builder on Trust Center and showcase your verified application ## What is Featured Builder? Featured Builders are highlighted applications on [trust.phala.com](https://trust.phala.com) that demonstrate excellent use cases of Phala's trusted execution environment. Being featured gives your application increased visibility and credibility within the Phala ecosystem. Featured Builder section on trust.phala.com homepage ## Requirements Before applying to become a Featured Builder, ensure your application: * Is deployed on Phala Cloud with passing Trust Center verification * Has a complete workspace profile with proper branding * Has enabled public listing for the application ## Step 1: Set Up Your Workspace Profile A complete workspace profile helps users understand who built the application and establishes trust. Navigate to your workspace settings in the Phala Cloud dashboard and fill in your profile information: * **Workspace Name**: Your organization or project name * **Description**: A brief description of what you do * **Logo**: Upload a clear, recognizable logo Workspace profile configuration page ## Step 2: Enable Public Listing Public listing makes your application discoverable on the Trust Center platform. This is required for Featured Builder consideration. In your application settings, locate and enable the **Public Listing** toggle: Public listing toggle in application settings Enabling public listing makes your application's Trust Center verification report publicly accessible. This is designed for transparency and allows users to verify your deployment. ## Step 3: Apply for Featured Builder Once your workspace profile is complete and public listing is enabled, contact the Phala team to apply for Featured Builder status. Send an email to **[cloud@phala.network](mailto:cloud@phala.network)** with: * Your application name and App ID * A brief description of your application's use case * Why your application would be a good showcase for Phala Cloud The team will review your application and respond within a few business days. ## Benefits of Being Featured * **Increased Visibility**: Your application appears on the Trust Center homepage * **Credibility**: Featured status signals quality and trust to potential users * **Community Recognition**: Join the showcase of leading builders on Phala ## Next Steps * **View Trust Center**: [trust.phala.com](https://trust.phala.com) * **Understand Verification**: [Trust Center Verification](/phala-cloud/attestation/trust-center-verification) * **Deploy Your App**: [Getting Started with Phala Cloud](/phala-cloud/getting-started/overview) # Get Attestation Source: https://docs.phala.com/phala-cloud/attestation/get-attestation Learn how to get CVM attestation on Phala Cloud using dashboard or dstack SDK. Get attestation reports from your CVM to prove it's running in genuine TEE hardware. You can check the dashboard for quick verification, or use the dstack SDK to generate quotes programmatically with custom data. ## View Attestation Report on Dashboard Open the **Attestation** tab in your CVM dashboard. Click **Check Attestation** to verify the default report generated during bootstrap. ## Generate Attestation Programmatically ### Prerequisites: Configure Docker Compose Your application needs access to the dstack service to generate attestation quotes. Mount the dstack socket into your container: ```yaml theme={"system"} version: '3' services: app: image: your-app-image ports: - 8080:8080 volumes: # Mount dstack socket for TEE operations - /var/run/dstack.sock:/var/run/dstack.sock ``` ### Using dstack SDK The dstack SDK connects to `/var/run/dstack.sock` automatically. Install for your language: * **JavaScript/TypeScript:** [@phala/dstack-sdk](https://www.npmjs.com/package/@phala/dstack-sdk) * **Python:** [dstack-sdk](https://pypi.org/project/dstack-sdk/) * **Go:** [github.com/Dstack-TEE/dstack](https://github.com/Dstack-TEE/dstack/tree/master/sdk/go) ### Attaching Custom Data (reportData) The attestation quote includes a 64-byte `reportData` field for your custom data. **Important:** The SDK throws an error if you exceed 64 bytes—it does not auto-hash. **Two patterns:** 1. **Short data (≤64 bytes)**: Nonces, small challenges, or hashes—pass directly 2. **Long data (>64 bytes)**: Any arbitrary data—hash it first with SHA256 (produces 32 bytes) ```javascript TypeScript theme={"system"} import { DstackClient } from '@phala/dstack-sdk'; import crypto from 'crypto'; const client = new DstackClient(); // Pattern 1: Short data (≤64 bytes) - pass directly // Example: 32-byte nonce for challenge-response const nonce = crypto.randomBytes(32); const quote1 = await client.getQuote(nonce); // Pattern 2: Long data (>64 bytes) - hash it first // Example: JSON with arbitrary data const userData = JSON.stringify({ version: '1.0.0', timestamp: Date.now(), user_id: 'alice', public_key: '0x1234...' }); // Hash to fit in 64 bytes (SHA256 produces 32 bytes) const hash = crypto.createHash('sha256').update(userData).digest(); const quote2 = await client.getQuote(hash); console.log('Quote:', quote2.quote); console.log('Event Log:', quote2.event_log); ``` ```python Python theme={"system"} import json import time import hashlib import secrets from dstack_sdk import DstackClient client = DstackClient() # Pattern 1: Short data (≤64 bytes) - pass directly # Example: 32-byte nonce for challenge-response nonce = secrets.token_bytes(32) quote1 = client.get_quote(nonce) # Pattern 2: Long data (>64 bytes) - hash it first # Example: JSON with arbitrary data user_data = json.dumps({ "version": "1.0.0", "timestamp": time.time(), "user_id": "alice", "public_key": "0x1234..." }) # Hash to fit in 64 bytes (SHA256 produces 32 bytes) data_hash = hashlib.sha256(user_data.encode()).digest() quote2 = client.get_quote(data_hash) print('Quote:', quote2.quote) print('Event Log:', quote2.event_log) ``` **reportData Parameter Required** The `getQuote()` method requires a reportData parameter. If you don't need custom data, pass an empty value: `''` in TypeScript or `b''` in Python. Calling `getQuote()` without any parameter will fail. ### Exposing Attestation via API Expose attestation endpoints so external verifiers can validate your CVM. The `/attestation` endpoint provides the quote for hardware verification, while `/info` provides the application configuration for code verification: ```javascript TypeScript theme={"system"} import express from 'express'; import { DstackClient } from '@phala/dstack-sdk'; const app = express(); const client = new DstackClient(); app.get('/attestation', async (req, res) => { const result = await client.getQuote(''); res.json({ quote: result.quote, event_log: result.event_log, vm_config: result.vm_config // Required by dstack-verifier }); }); app.get('/info', async (req, res) => { const info = await client.info(); res.json(info); }); app.listen(8080); ``` ```python Python theme={"system"} from flask import Flask, jsonify from dstack_sdk import DstackClient app = Flask(__name__) client = DstackClient() @app.route('/attestation') def get_attestation(): result = client.get_quote(b'') return jsonify({ 'quote': result.quote, 'event_log': result.event_log, 'vm_config': result.vm_config # Required by dstack-verifier }) @app.route('/info') def get_info(): info = client.info() return jsonify(info) if __name__ == '__main__': app.run(host='0.0.0.0', port=8080) ``` These endpoints allow external verifiers to fetch attestation data and verify your CVM. See [Verify Your Application](/phala-cloud/attestation/verify-your-application) for how verifiers use these endpoints. ## Next Steps Prove your exact code is running Verify OS, KMS, and infrastructure Understand what each field in the quote means Complete verification with no trust assumptions # Overview Source: https://docs.phala.com/phala-cloud/attestation/overview Learn how attestation proves your CVM runs unmodified code in genuine TEE hardware. ## Why Attestation Matters Attestation proves your CVM runs in genuine TEE hardware with the exact software you expect. No one can fake it. It gives you cryptographic proof covering three layers: Intel TDX hardware, the operating system, and your application code. Anyone can independently verify this proof to confirm your CVM's security. You can verify what you control (your application code) or verify everything (including Phala's infrastructure). The choice depends on your security requirements. ## How Attestation Works Your CVM generates a cryptographic quote that includes measurements of the Intel TDX CPU, OS images, your application configuration, and optional custom data like public keys or challenge responses. The quote is signed by Intel hardware. Verifiers check this signature against Intel's root certificates to confirm authenticity. ## What You Can Verify **Your Application**: Prove your specific Docker containers and configuration run unmodified. This verifies no one substituted your code. **The Platform**: Prove the OS, key management system, and network infrastructure are secure. This verifies Phala's infrastructure integrity. **Complete Chain**: Verify everything end-to-end with no trust assumptions. This provides complete trustless execution. ## Get Started Generate your first quote and verify it works Generate quotes using dashboard or SDK Prove your exact code is running Verify OS, KMS, and infrastructure Complete verification with no trust assumptions Understand what each field in the quote means ## Next Steps New to attestation? Start with the [Quickstart](/phala-cloud/attestation/quickstart) to generate and verify your first quote. # Quickstart Source: https://docs.phala.com/phala-cloud/attestation/quickstart Get your first attestation proof using the dashboard. Get your first attestation proof using just the dashboard. No code required. ## Step 1: Open Attestation Tab Open your CVM in the [Phala Cloud dashboard](https://cloud.phala.com). Click **View Details** on your CVM card. Navigate to the **Attestations** tab. You'll see the default attestation quote that was generated during CVM bootstrap. ## Step 2: Check Attestation Click the **Check Attestation** button in the certificate chain section. This automatically opens the [TEE Attestation Explorer](https://proof.t16z.com/) in a new tab with your quote already loaded and verified. You'll see verification results showing: * ✓ Hardware signature is valid (genuine Intel TDX) * ✓ Quote measurements (MRTD, RTMR0-3) * ✓ Security status (TCB level, no known vulnerabilities) ## You're Done! You just verified your CVM runs in genuine TEE hardware. The cryptographic proof shows Intel TDX signed your quote, making it impossible to fake. You can share the proof URL with anyone who needs to verify your CVM's security. ## What You Verified **Hardware authenticity**: Intel's signature proves genuine TDX hardware generated this quote. **Measurement integrity**: The quote includes cryptographic hashes (measurements) of your OS, application, and hardware configuration. Change one byte anywhere, and the measurements won't match. **Unforgeable proof**: The signature binds all measurements together. No one can modify the quote without invalidating Intel's signature. ## What's Next? Now that you understand attestation, choose your path: ### Want Programmatic Attestation? **Generate quotes from code** instead of the dashboard. Useful for APIs, services, or automated verification. See [Get Attestation](/phala-cloud/attestation/get-attestation) for SDK examples. ### Want to Bind Custom Data? **Attach challenges, public keys, or nonces** to your quotes. This proves freshness and prevents replay attacks. See [Get Attestation: Attaching Custom Data](/phala-cloud/attestation/get-attestation#attaching-custom-data-reportdata) for reportData patterns. ### Want to Verify Your Application Code? **Prove your exact Docker images are running** unmodified. This verifies no one substituted your code. See [Verify Your Application](/phala-cloud/attestation/verify-your-application) for compose-hash verification. ### Want to Verify Platform Security? **Verify the OS, KMS, and infrastructure** end-to-end. This proves Phala's platform integrity without trust assumptions. See [Verify the Platform](/phala-cloud/attestation/verify-the-platform) for complete platform verification. ### Want Complete Security Audit? **Verify everything with no trust assumptions.** Security researchers and auditors need the complete checklist. See [Complete Chain of Trust](/phala-cloud/attestation/chain-of-trust) for necessary and sufficient verification. ### Want to Understand Quote Fields? **Learn what each measurement means** (MRTD, RTMR0-3, reportData, etc.). See [Attestation Fields Reference](/phala-cloud/attestation/attestation-fields) for field-by-field explanation. # Trust Center Verification Source: https://docs.phala.com/phala-cloud/attestation/trust-center-verification Automated verification reports for every Phala Cloud deployment with the Trust Center platform ## What is the Trust Center? The [Trust Center](https://trust.phala.com/) is an automated verification platform that generates comprehensive security reports for every application deployed on Phala Cloud. It provides cryptographic proof that your application is running in a genuine TEE with the exact code you deployed, giving you and your users confidence in the integrity and security of your deployment. **Automatic Verification**: Every application deployed to Phala Cloud automatically receives a Trust Center verification report. No additional configuration required. ## Why Trust Center Verification Matters Traditional cloud platforms require you to trust the provider. With Phala Cloud and Trust Center: * **Verify Hardware Authenticity**: Cryptographically prove your app runs on genuine Intel TDX and Nvidia CC hardware * **Confirm Code Integrity**: Verify the exact Docker compose configuration deployed matches your source code * **Validate OS Security**: Ensure the operating system hasn't been tampered with * **Prove Zero Trust**: For gateway deployments, verify domain certificates are managed entirely within the TEE This matters for: * **Regulatory Compliance**: Meet GDPR, HIPAA, and other privacy requirements with verifiable evidence * **Customer Trust**: Share public verification reports with users to prove data privacy * **Security Audits**: Provide auditors with cryptographic proof of deployment integrity ## Accessing Your Verification Report Every Phala Cloud application receives a unique verification report URL: ``` https://trust.phala.com/app/{your-app-id} ``` You can find your app ID in: 1. The Phala Cloud dashboard 2. The API response when creating a CVM **Example Report**: [https://trust.phala.com/app/7e0817205044eb202a590a0d236ccb4d66140197](https://trust.phala.com/app/7e0817205044eb202a590a0d236ccb4d66140197) ## Understanding Your Verification Report The Trust Center report displays comprehensive verification results across multiple security dimensions. Let's walk through each section: ### Overview Section The left column of the report shows your application's basic information and overall verification status. Trust Center application overview section **What you see:** * **App Name**: Your application's identifier (e.g., "account-demo") * **Verification Status**: Overall pass/fail indicator with green checkmark when all checks succeed * **Hardware Verification**: Confirms TEE runs on genuine Intel TDX and Nvidia CC hardware * **Source Code Verification**: Validates deployed Docker Compose matches your source code repository * **Network Verification**: Checks domain certificates and network infrastructure security * **Operating System Verification**: Ensures OS image matches trusted dstack release * **Key Management Verification**: Validates cryptographic key derivation ### Hardware Verification This section proves your application runs on genuine Intel and Nvidia TEE hardware. Hardware attestation verification status **What's verified:** * **Intel TDX Quote**: Cryptographically signed attestation from Intel hardware * **(Optional) Nvidia CC Attestation Report**: Cryptographically signed attestation from Nvidia hardware * **TCB Status**: Trusted Computing Base security version (checks for known vulnerabilities) * **Platform Certificates**: Validates certificate chain back to Intel/Nvidia root CA * **Hardware Identity**: Confirms genuine Intel processor with TDX enabled and Nvidia GPU with CC enabled **Why it matters**: Ensures your application runs on real security hardware, not a simulated environment or compromised system. ### Source Code Verification This is the critical verification that your deployed application contains exactly the code you intended. Source code verification showing compose hash match **What's verified:** * **Compose Hash**: SHA256 hash of your configuration file containing the raw Docker compose file and other metadata * **RTMR3**: Application-specific measurement register containing: * `app-id`: Application identifier * `compose-hash`: Hash of deployment manifest * `instance-id`: Specific deployment instance * `os-image-hash`: Hash of operating system image **Why it matters**: This is the core of code integrity. Anyone can recompute the compose hash from your source code and verify it matches the deployed version. No hidden modifications possible. **How to verify yourself:** ```bash theme={"system"} # Compute the compose hash of your compose_file.json sha256sum compose_file.json # Compare with the hash shown in compose-hash Event Logs # They must match exactly ``` ### Operating System Integrity This verification confirms the OS running in your TEE matches a known trusted version. Operating system integrity verification **What's verified:** * **MRTD (Measurement Root of Trust Domain)**: Hash of the initial TD state * **RTMR0**: Virtual hardware environment measurement * **RTMR1**: Linux kernel measurement * **RTMR2**: Kernel command line and initrd measurement * **dstack Version**: Specific OS image version hash **Why it matters**: Proves the operating system hasn't been modified or backdoored. The OS matches publicly auditable dstack releases. ### Network Verification (Gateway Deployments) For applications exposed via custom domains with dstack-gateway, this verifies zero-trust HTTPS and secure traffic forwarding. Domain verification for zero-trust HTTPS **What's verified:** * **Gateway TEE Status**: Confirms the dstack-gateway itself runs in a TEE environment * **TLS Certificate**: Certificate private key generated entirely within TEE * **CAA Records**: DNS Certification Authority Authorization locked to TEE-controlled account * **Certificate Transparency**: All certificates logged to public CT logs * **Domain Ownership**: Cryptographic proof the domain is controlled by the TEE **Why it matters**: The gateway runs entirely within a TEE, ensuring it can safely decrypt, inspect, and forward traffic to your application without exposing it to untrusted infrastructure. Traditional HTTPS terminates at cloud load balancers where providers can access your traffic. With Phala's TEE-based gateway, even Phala cannot intercept your TLS traffic—everything stays encrypted until it reaches your application's TEE. ### Key Management Service Verification This section verifies that cryptographic keys are properly managed and derived through Phala's decentralized key management system. Key management service verification status **What's verified:** * **Key Provider Address**: The decentralized KMS endpoint used for key derivation * **KMS Node TEE Status**: All KMS nodes run in TEE environments with program digests matching the smart contract * **App-ID Binding**: Confirms keys are uniquely derived from your application identifier * **Key Derivation Protocol**: Validates the cryptographic protocol used to generate encryption keys * **No Vendor Lock-in**: Verifies keys can be recreated on any Phala worker or dstack node with the same app-id **Why it matters**: Unlike traditional cloud platforms where encryption keys are tied to specific hardware or vendor infrastructure, Phala's decentralized key management ensures your keys are derived from your application identity, not the physical machine. This means you can migrate your application between different compute providers while maintaining access to your encrypted data. The KMS verification proves that your keys are generated securely and consistently, without giving Phala or any single party control over your encryption keys. ## Interpreting Verification Status ### ✅ Passed Verification When all checks pass: * Your app runs on genuine Intel TEE hardware * The OS matches a trusted dstack version * Your source code hash matches the deployed version * (If applicable) Domain certificates are TEE-controlled **What this means**: You have cryptographic proof your application is running securely and exactly as intended. You can share this report with users, auditors, or compliance officers. ### ⚠️ Partial Verification Some checks passed, others failed or skipped: * Check which specific verifications failed * Common causes: outdated TCB version, domain not configured, custom OS modifications **Action**: Review failed checks. Some failures may be expected (e.g., domain verification for apps without custom domains). ### ❌ Failed Verification Verification could not complete: * Hardware attestation signature invalid * Source code hash mismatch * OS image not recognized **Action**: This indicates a potential security issue. Contact Phala support or re-deploy your application. ## Sharing Your Verification Report Trust Center reports are **publicly accessible** by design. This enables: 1. **Customer Transparency**: Share your report URL in documentation or marketing 2. **Auditor Access**: Provide reports to security auditors without exposing credentials 3. **Automated Verification**: Third parties can programmatically verify your deployment Example uses: ```markdown theme={"system"} Security: [View our Trust Center report](https://trust.phala.com/app/YOUR_APP_ID) ``` ```html theme={"system"} TEE Verified ``` ## Programmatic Verification See [Trust Center Technical Documentation](/dstack/trust-center-technical) for API details and integration examples. ## Trust Center vs Manual Verification | Feature | Trust Center | Manual Verification | | -------------------------- | -------------------------- | --------------------------------- | | **Ease of Use** | ✅ Automatic, zero config | ❌ Complex, requires tooling | | **Public Auditability** | ✅ Shareable URL | ⚠️ Must share raw quote data | | **Verification Speed** | ✅ Seconds | ⚠️ Minutes to hours | | **Technical Expertise** | ✅ No expertise needed | ❌ Requires cryptography knowledge | | **Blockchain Integration** | ✅ Verifies onchain records | ⚠️ Manual registry lookup | For most users, Trust Center provides the easiest and most reliable verification. Advanced users can still perform [manual verification](/phala-cloud/attestation/verification-guide) for full control. ## Next Steps * **Learn the Technical Details**: [Trust Center Technical Architecture](/dstack/trust-center-technical) * **Understand Attestation**: [Verifying TEE Attestation](/phala-cloud/attestation/verification-guide) * **Deep Dive on Security**: [Security Architecture](/phala-cloud/security-and-privacy/security-architecture) ## Additional Resources * **Trust Center Platform**: [https://trust.phala.com](https://trust.phala.com) * **GitHub Repository**: [https://github.com/Phala-Network/trust-center](https://github.com/Phala-Network/trust-center) * **Example Report**: [account-demo verification](https://trust.phala.com/app/7e0817205044eb202a590a0d236ccb4d66140197) # End-to-End Attestation Verification Source: https://docs.phala.com/phala-cloud/attestation/verification-guide Fetch, verify, and integrate CVM attestation into your workflow — end-to-end with working code Verify that your CVM runs unmodified code on genuine TEE hardware, from fetching the attestation quote to integrating verification into your CI/CD pipeline. This guide ties together the attestation workflow end-to-end with working code examples. If you're new to attestation, start with the [Quickstart](/phala-cloud/attestation/quickstart) to understand the basics through the dashboard. This guide assumes you want programmatic verification. ## The verification flow Every CVM attestation verification follows three steps: get a quote from your running CVM, verify the quote's hardware signature, and check that the measurements match your expected application. The entire process runs in seconds from anywhere outside the CVM. ## Step 1: Get the attestation quote Your CVM must expose attestation endpoints. If you haven't set this up yet, see [Get Attestation](/phala-cloud/attestation/get-attestation) for the dstack SDK setup. Fetch the quote with a fresh challenge nonce to prevent replay attacks. ```typescript TypeScript theme={"system"} import crypto from 'crypto'; const CVM_URL = 'https://your-app.example.com'; // Generate a fresh 32-byte nonce const nonce = crypto.randomBytes(32).toString('hex'); // Fetch attestation with nonce as reportData const attestResponse = await fetch(`${CVM_URL}/attestation?reportData=${nonce}`); const attestData = await attestResponse.json(); // Fetch application info for compose verification const infoResponse = await fetch(`${CVM_URL}/info`); const appInfo = await infoResponse.json(); console.log('Quote length:', attestData.quote.length); console.log('Event log events:', JSON.parse(attestData.event_log).length); ``` ```python Python theme={"system"} import secrets import requests CVM_URL = "https://your-app.example.com" # Generate a fresh 32-byte nonce nonce = secrets.token_hex(32) # Fetch attestation with nonce as reportData attest_response = requests.get(f"{CVM_URL}/attestation?reportData={nonce}") attest_data = attest_response.json() # Fetch application info for compose verification info_response = requests.get(f"{CVM_URL}/info") app_info = info_response.json() print(f"Quote length: {len(attest_data['quote'])}") ``` ```go Go theme={"system"} package main import ( "crypto/rand" "encoding/hex" "encoding/json" "fmt" "io" "net/http" ) func main() { cvmURL := "https://your-app.example.com" // Generate a fresh 32-byte nonce nonceBytes := make([]byte, 32) rand.Read(nonceBytes) nonce := hex.EncodeToString(nonceBytes) // Fetch attestation resp, _ := http.Get(fmt.Sprintf("%s/attestation?reportData=%s", cvmURL, nonce)) body, _ := io.ReadAll(resp.Body) var attestData map[string]interface{} json.Unmarshal(body, &attestData) fmt.Printf("Quote length: %d\n", len(attestData["quote"].(string))) } ``` Always generate a fresh nonce per verification request. Without it, an attacker could replay an old valid quote from compromised hardware. ## Step 2: Verify the hardware signature The TDX quote is signed by Intel hardware. Verify this signature to confirm the quote came from a genuine TEE. You have two options: use Phala's verification API for convenience, or verify locally for trustless verification. ### Option A: Phala Cloud verification API The fastest approach. Sends your quote to Phala's service which checks it against Intel's root certificates. ```typescript TypeScript theme={"system"} const verifyResponse = await fetch( 'https://cloud-api.phala.com/api/v1/attestations/verify', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ hex: attestData.quote }), } ); const result = await verifyResponse.json(); if (!result.quote.verified) { throw new Error('Hardware verification failed'); } console.log('Hardware signature: valid'); console.log('TDX version:', result.quote.body.tee_tcb_svn); ``` ```python Python theme={"system"} verify_response = requests.post( "https://cloud-api.phala.com/api/v1/attestations/verify", json={"hex": attest_data["quote"]} ) result = verify_response.json() assert result["quote"]["verified"], "Hardware verification failed" print(f"Hardware signature: valid") print(f"TDX version: {result['quote']['body']['tee_tcb_svn']}") ``` ```bash curl theme={"system"} curl -X POST "https://cloud-api.phala.com/api/v1/attestations/verify" \ -H "Content-Type: application/json" \ -d '{"hex": "YOUR_QUOTE_HEX"}' ``` ### Option B: Local verification with dcap-qvl For trustless verification without relying on Phala's API, use the open-source [dcap-qvl](https://github.com/Phala-Network/dcap-qvl) verifier. This checks Intel's signature chain locally. ```bash theme={"system"} # Install dcap-qvl cargo install dcap-qvl # Verify a quote locally dcap-qvl verify --quote-hex "YOUR_QUOTE_HEX" ``` You can also paste your quote into the [TEE Attestation Explorer](https://proof.t16z.com/) for an interactive visual breakdown of all quote fields. ## Step 3: Verify your application code Hardware verification proves the TEE is genuine. Now verify that your specific application is running inside it. The compose-hash in RTMR3 is a SHA256 hash of your entire Docker Compose configuration. ```typescript TypeScript theme={"system"} import { createHash } from 'crypto'; // Calculate the compose hash from the app info const appCompose = appInfo.tcb_info.app_compose; const calculatedHash = createHash('sha256') .update(appCompose) .digest('hex'); // Extract the attested hash from the RTMR3 event log const events = JSON.parse(attestData.event_log); const composeEvent = events.find( (e: { event: string }) => e.event === 'compose-hash' ); const attestedHash = composeEvent.event_payload; if (calculatedHash !== attestedHash) { throw new Error( `Compose hash mismatch: expected ${calculatedHash}, got ${attestedHash}` ); } console.log('Application code: verified'); console.log('Compose hash:', calculatedHash); ``` ```python Python theme={"system"} import hashlib import json # Calculate the compose hash from the app info app_compose = app_info["tcb_info"]["app_compose"] calculated_hash = hashlib.sha256(app_compose.encode()).hexdigest() # Extract the attested hash from the RTMR3 event log events = json.loads(attest_data["event_log"]) compose_event = next(e for e in events if e["event"] == "compose-hash") attested_hash = compose_event["event_payload"] assert calculated_hash == attested_hash, ( f"Compose hash mismatch: expected {calculated_hash}, got {attested_hash}" ) print(f"Application code: verified") print(f"Compose hash: {calculated_hash}") ``` ## Step 4: Verify the nonce binding Confirm that the quote contains your original nonce. This proves the attestation was freshly generated for your request. ```typescript TypeScript theme={"system"} // The reportData field in the quote should start with your nonce const reportData = attestData.report_data; if (!reportData.startsWith(nonce)) { throw new Error('Nonce mismatch: possible replay attack'); } console.log('Freshness: verified'); ``` ```python Python theme={"system"} report_data = attest_data["report_data"] assert report_data.startswith(nonce), "Nonce mismatch: possible replay attack" print("Freshness: verified") ``` ## Complete verification script Here's everything combined into a single script you can drop into your project. ```typescript TypeScript theme={"system"} import crypto from 'crypto'; import { createHash } from 'crypto'; async function verifyCVM(cvmUrl: string): Promise { // 1. Generate fresh nonce const nonce = crypto.randomBytes(32).toString('hex'); // 2. Fetch attestation and app info const [attestRes, infoRes] = await Promise.all([ fetch(`${cvmUrl}/attestation?reportData=${nonce}`), fetch(`${cvmUrl}/info`), ]); const attestData = await attestRes.json(); const appInfo = await infoRes.json(); // 3. Verify hardware signature const verifyRes = await fetch( 'https://cloud-api.phala.com/api/v1/attestations/verify', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ hex: attestData.quote }), } ); const verifyResult = await verifyRes.json(); if (!verifyResult.quote.verified) { console.error('Hardware verification failed'); return false; } // 4. Verify compose hash const appCompose = appInfo.tcb_info.app_compose; const calculatedHash = createHash('sha256').update(appCompose).digest('hex'); const events = JSON.parse(attestData.event_log); const composeEvent = events.find( (e: { event: string }) => e.event === 'compose-hash' ); if (calculatedHash !== composeEvent.event_payload) { console.error('Compose hash mismatch'); return false; } // 5. Verify nonce freshness if (!attestData.report_data.startsWith(nonce)) { console.error('Nonce mismatch'); return false; } console.log('All checks passed'); return true; } // Usage verifyCVM('https://your-app.example.com').then(console.log); ``` ```python Python theme={"system"} import hashlib import json import secrets import requests def verify_cvm(cvm_url: str) -> bool: # 1. Generate fresh nonce nonce = secrets.token_hex(32) # 2. Fetch attestation and app info attest_data = requests.get( f"{cvm_url}/attestation?reportData={nonce}" ).json() app_info = requests.get(f"{cvm_url}/info").json() # 3. Verify hardware signature verify_result = requests.post( "https://cloud-api.phala.com/api/v1/attestations/verify", json={"hex": attest_data["quote"]}, ).json() if not verify_result["quote"]["verified"]: print("Hardware verification failed") return False # 4. Verify compose hash app_compose = app_info["tcb_info"]["app_compose"] calculated_hash = hashlib.sha256(app_compose.encode()).hexdigest() events = json.loads(attest_data["event_log"]) compose_event = next(e for e in events if e["event"] == "compose-hash") if calculated_hash != compose_event["event_payload"]: print("Compose hash mismatch") return False # 5. Verify nonce freshness if not attest_data["report_data"].startswith(nonce): print("Nonce mismatch") return False print("All checks passed") return True if __name__ == "__main__": verify_cvm("https://your-app.example.com") ``` ## Integrate verification into CI/CD Attestation verification fits naturally into deployment pipelines. Run it after every deployment to confirm your CVM booted correctly with the expected code. ### GitHub Actions example ```yaml theme={"system"} name: Verify CVM Attestation on: workflow_dispatch: schedule: - cron: '0 */6 * * *' # Every 6 hours jobs: verify: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install dependencies run: pip install requests - name: Verify attestation env: CVM_URL: ${{ secrets.CVM_URL }} run: python scripts/verify_attestation.py "$CVM_URL" - name: Alert on failure if: failure() run: | curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \ -d '{"text": "CVM attestation verification failed!"}' ``` ### Continuous monitoring For production systems, run verification checks on a schedule. Here's a lightweight monitoring approach: ```python theme={"system"} import time import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger("attestation-monitor") CVM_URLS = [ "https://app1.example.com", "https://app2.example.com", ] CHECK_INTERVAL = 3600 # 1 hour while True: for url in CVM_URLS: try: result = verify_cvm(url) if result: logger.info(f"PASS: {url}") else: logger.error(f"FAIL: {url}") # Send alert to your monitoring system except Exception as e: logger.error(f"ERROR: {url} - {e}") time.sleep(CHECK_INTERVAL) ``` Attestation verification must always run **outside** the CVM. Verification running inside the CVM provides no security guarantees because a compromised CVM could fake its own results. ## What each check proves Here's a quick reference for what you're verifying at each step and what attack it prevents. | Check | What it proves | Attack prevented | | ------------------ | ------------------------------------ | -------------------- | | Hardware signature | Quote came from genuine Intel TDX | Fake TEE simulation | | Compose hash | Your exact Docker images are running | Code substitution | | Nonce freshness | Quote was generated for this request | Replay of old quotes | | reportData binding | Custom data is attested by hardware | Data tampering | For advanced verification including RTMR3 event log replay, Docker digest pinning, on-chain governance, and source code provenance, see the [Complete Chain of Trust](/phala-cloud/attestation/chain-of-trust). ## Next steps Advanced verification including RTMR3 replay and image digest checks Verify OS, KMS, and infrastructure without trust assumptions Understand every field in the attestation quote Verify GPU-specific attestation for Confidential AI workloads # Verify the Platform Source: https://docs.phala.com/phala-cloud/attestation/verify-the-platform Verify Phala's infrastructure integrity end-to-end. Verify Phala's infrastructure integrity end-to-end. This proves the OS, key management system, network certificates, and governance are all secure. ## Do You Need This? Prove platform integrity to your end users. Verify both your application and the infrastructure it runs on. Application verification proves your code runs in a TEE. But users still have to trust the OS, key management, network certificates, and governance. Any of these could be compromised. Verify the platform too, and you get end-to-end security with zero trust assumptions. The only thing left to trust is the TEE hardware itself. ## What is TCB (Trusted Computing Base)? The Trusted Computing Base (TCB) represents all components critical for security. In dstack, the TCB consists of cryptographic measurements that prove integrity across three layers. A **measurement** is a cryptographic hash (like a fingerprint) of a component. TEE hardware records these measurements in the remote attestation report during boot. Once recorded in the cryptographically signed quote, measurements cannot be modified. This makes them unforgeable proof of what code is running. Change even one byte in any component, and the measurements won't match your expected values. **Hardware & Firmware** use MRTD for virtual firmware and RTMR0 for hardware configuration like CPU count and memory size. **Operating System** uses RTMR1 for the Linux kernel and RTMR2 for kernel parameters and initrd. **Application** uses RTMR3 for your compose hash and runtime info. All RTMRs (0-3) use the same hash chain structure. Each starts at zero and is "extended" by hashing in events during boot (RTMR = SHA384(RTMR || event)). For verification, only RTMR3 event logs matter. RTMR0-2 event logs contain low-level hardware/firmware details you can ignore. Just use dstack-mr to reproduce the final RTMR0-2 values directly. ## Prerequisites Your application must expose an `/attestation` endpoint that returns the quote, event log, and VM configuration. This endpoint is what dstack-verifier calls to get the data it needs. See [Get Attestation](/phala-cloud/attestation/get-attestation) for how to set this up inside your CVM. ## Using dstack-verifier The dstack-verifier tool automates complete platform verification. It validates hardware, firmware, OS, key management, and event log integrity in one operation. Run it as an HTTP service: ```bash theme={"system"} # Via Docker docker run -p 8080:8080 dstacktee/dstack-verifier:latest # Or via cargo cargo run --bin dstack-verifier ``` Verify a quote: ```bash theme={"system"} # Get quote from your app curl https://your-app.example.com/attestation -o quote.json # Verify it curl -d @quote.json localhost:8080/verify | jq ``` The verifier checks: * **Hardware & firmware**: MRTD, RTMR0, TCB status, debug mode disabled * **Operating system**: RTMR1, RTMR2, OS image hash matches known build * **Key management**: KeyProviderInfo extracted from RTMR3 events * **Event log integrity**: Replays events to verify RTMR values match quote Response shows `is_valid: true` when all checks pass. See [dstack-verifier README](https://github.com/Dstack-TEE/dstack/tree/master/verifier) for API details and configuration. ### What the Verifier Checks Automatically dstack-verifier automates cryptographic verification by fetching reference values and comparing them against your quote. **Automatically fetched and verified**: * OS images downloaded from dstack releases (if not cached locally) * Expected MRTD and RTMR0-2 calculated using dstack-mr from the OS image + VM config * Quote signature validated using Intel root certificates via dcap-qvl * RTMR3 recalculated by replaying the event log * TCB status checked for security patches and debug mode disabled **You provide** (from your app's `/attestation` endpoint): * TDX quote (hex string) * Event log (hex-encoded JSON) * VM configuration (JSON with CPU, memory, os\_image\_hash) **You verify separately**: * Governance policies via smart contracts - see [5. Governance Verification](#5-governance-verification) * KMS instance attestation - see [3. Key Management Verification](#3-key-management-verification) * Network certificate binding - see [4. Network Security Verification](#4-network-security-verification) The verifier handles all cryptographic verification. You handle policy and governance verification. ## 1. Hardware & Firmware Verification ### Attack Vector Modified firmware can compromise the entire boot sequence. An attacker who controls the virtual firmware (OVMF) can load a malicious OS that appears legitimate to your application. The firmware is the trust anchor - if it's compromised, everything that follows is suspect. ### How It's Secured **MRTD** measures the virtual firmware (OVMF). This is the first code executed after CVM startup and serves as your trust anchor. **RTMR0** measures the virtual hardware configuration - CPU count, memory size, and device setup. This ensures the CVM runs with expected hardware resources. **mr\_seam** must be all zeros for TDX TD 1.0. This verifies the SEAM firmware signature is correct. **Debug mode** must be disabled. The TCB validation checks this to prevent debugging interfaces from exposing secrets. Intel signs the quote containing these measurements with their hardware root of trust. ### How to Verify Use dstack-verifier to verify hardware and firmware automatically. It validates the quote signature, checks TCB status, and ensures debug mode is disabled. **Implementation details**: See [`verify_quote` function](https://github.com/Dstack-TEE/dstack/blob/master/verifier/src/verification.rs) in verifier/src/verification.rs for the complete TCB validation logic. ## 2. Operating System Verification ### Attack Vector Anyone in the cloud compute supply chain (cloud provider, network operator, data center staff, or compromised infrastructure) could substitute the dstack OS image with a modified version containing backdoors or data exfiltration code. ### How It's Secured The dstack OS is built from the [meta-dstack](https://github.com/Dstack-TEE/meta-dstack) repository using Yocto. This means you can reproduce it from any specific git commit. The OS produces four measurements (MRTD, RTMR0-2) that cryptographically prove firmware, hardware config, kernel, and boot parameters. Only OS images with approved hashes can boot, enforced by the `DstackKms.allowedOsImages` smart contract. ### How to Verify Use dstack-verifier to verify OS integrity automatically. It downloads the OS image, calculates MRTD and RTMR0-2 using the VM configuration, and compares them against the quote. For reproducible builds, you can independently build the OS from [meta-dstack](https://github.com/Dstack-TEE/meta-dstack) source and verify it produces identical measurements. **Implementation details**: See [`verify_os_image_hash` function](https://github.com/Dstack-TEE/dstack/blob/master/verifier/src/verification.rs) in verifier/src/verification.rs for the OS verification logic. ## 3. Key Management Verification ### Attack Vector A malicious KMS could leak all keys used by your application: disk encryption keys, TLS private keys, and signing keys. This would compromise your application even if everything else was verified. ### How It's Secured Your application uses multiple keys: disk encryption, TLS certificates, and signing keys. All of these derive from two KMS root key pairs using deterministic Key Derivation Functions (KDFs). The Root CA Key (P256) derives your TLS certificates and disk encryption keys. The Root K256 Key (secp256k1) derives Ethereum-compatible signing keys. Each derived key combines your app's unique ID with a purpose string for cryptographic separation. Verify the KMS is trustworthy, and you've automatically verified all derived keys too. The KMS root CA public key is recorded in RTMR3 as the `key-provider` event. This binds your app to a specific KMS instance. You can't silently swap to a different KMS without changing the attestation. ### How to Verify Use dstack-verifier to extract the key provider information from RTMR3 events automatically. The verifier returns the KMS ID and name in the verification response. **Important**: dstack-verifier only extracts which KMS instance your app uses. It does not verify the KMS itself. The KMS is a separate TEE instance that requires its own complete verification: 1. **Hardware verification**: Validate the KMS's TDX quote against Intel root CAs 2. **OS integrity verification**: Verify the KMS's MRTD and RTMR0-2 match expected values for the dstack OS version 3. **Source code verification**: Verify the KMS's compose hash matches known trustworthy KMS configurations 4. **Governance verification**: Check the KMS's aggregated MR is whitelisted in the `DstackKms.kmsAllowedAggregatedMrs` smart contract For production deployments, use [trust-center](https://github.com/Phala-Network/trust-center) which automates complete KMS verification including all reference value comparisons. **Implementation details**: See [`decode_app_info` method](https://github.com/Dstack-TEE/dstack/blob/master/verifier/src/verification.rs) in verifier/src/verification.rs for key provider extraction logic. ## 4. Network Security Verification ### Attack Vector Anyone could issue a valid TLS certificate for your domain and impersonate your TEE. This includes the domain owner, cloud provider, or a compromised Certificate Authority. Without verification, users can't tell the legitimate TEE-controlled certificate from a fraudulent one used for man-in-the-middle attacks. ### How It's Secured Your TLS certificates are generated and controlled entirely within the TEE. For custom domains, the TEE creates its own ACME account and TLS private keys inside encrypted memory. The private keys never leave the TEE. Evidence files published at `/evidences/` prove the TEE controls the certificate through cryptographic binding. The TEE puts a hash of the certificate evidence into the TDX quote's `report_data` field. This proves the TEE created both the quote and the certificate evidence at the same time. CAA (Certification Authority Authorization) DNS records add another layer of protection. They restrict which Certificate Authorities can issue certificates for your domain, preventing unauthorized issuance even if DNS is compromised. ### How to Verify The verification process differs based on your domain type. #### For custom domains Verify the evidence files: 1. Download evidence files from `https://your-domain.com/evidences/` 2. Verify the certificate fingerprint matches what's being served 3. Check that the TDX quote contains the hash of the evidence files 4. Confirm CAA records restrict certificate issuance to the TEE's ACME account See **[Domain Attestation](/phala-cloud/networking/domain-attestation)** for complete step-by-step verification. #### For Phala Cloud domains `*.phala.network` domains use the gateway TEE for TLS termination. The gateway performs mutual attestation with your CVM to establish a secure tunnel. You don't need to verify TLS certificates for these domains. The gateway's attestation is verified separately. ## 5. Governance Verification ### Attack Vector Without governance verification, a malicious developer could update the application at any time to introduce backdoors. This could happen even if the previous version was fully attested, verified, and trusted by users. ### How It's Secured The `DstackApp` and `DstackKms` contracts define which compose hashes (application versions), OS images (system versions), and KMS instances are allowed. Code updates pushed to Phala Cloud must pass contract authorization. The new compose-hash must be whitelisted on-chain before deployment. ### How to Verify #### Application governance Check which application versions are authorized: ```bash theme={"system"} # Check if your app's compose-hash is whitelisted cast call "allowedComposeHashes(bytes32)" # Monitor for new authorized versions cast logs --event "ComposeHashAdded(bytes32,address)" ``` #### Platform governance The `DstackKms` contract controls platform-level security: ```bash theme={"system"} # Check allowed OS images cast call "allowedOsImages(bytes32)" # Check allowed KMS instances cast call "kmsAllowedAggregatedMrs(bytes32)" # Monitor governance changes cast logs ``` Understanding who controls these contracts and how they're governed is critical for assessing platform trustworthiness. ## Complete Platform Verification Checklist Verify the full platform integrity by checking each component: **Hardware & Firmware:** * [ ] TDX quote signature is valid (Intel's root certificates) * [ ] `tee_tcb_svn` matches latest security patches * [ ] `mr_seam` matches known TDX firmware **Operating System:** * [ ] OS version from `appInfo.tcb_info.os_version` is known * [ ] MRTD and RTMR0-2 match calculated values * [ ] VM config (CPU, memory, GPU) matches `appInfo.vm_config` * [ ] OS image hash is whitelisted in `DstackKms.allowedOsImages` * [ ] (Optional) OS built reproducibly from source **Key Management:** * [ ] KMS ID from RTMR3 `key-provider` event is known * [ ] KMS's own attestation quote is valid * [ ] KMS aggregated MR is whitelisted in `DstackKms.kmsAllowedAggregatedMrs` **Network Security:** * [ ] TLS certificate fingerprint matches served certificate * [ ] Evidence files at `/evidences/` are cryptographically bound to quote * [ ] CAA DNS records restrict certificate issuance **Governance:** * [ ] Smart contract addresses are verified * [ ] Contract permissions match security policy * [ ] Contract ownership and upgrade mechanisms are understood ## Attack Scenarios Prevented **Compromised OS:** OS measurements in RTMR0-2 and on-chain whitelist prevent unauthorized OS versions from booting. **Malicious KMS:** KMS binding in RTMR3 and on-chain governance prevent unauthorized KMS instances from providing keys. **Certificate impersonation:** Evidence files and CAA records prevent unauthorized TLS certificates from being issued. **Supply chain attacks:** Reproducible OS builds and governance contracts prevent compromised build infrastructure from injecting malicious code. **Unauthorized updates:** On-chain governance ensures only authorized OS versions, KMS instances, and application updates can run. ## Tools and Resources * **[dstack-verifier](https://github.com/Dstack-TEE/dstack/tree/main/verifier)** - Automated platform verification tool * **[meta-dstack](https://github.com/Dstack-TEE/meta-dstack)** - Reproducible OS builds * **[dstack SDK](https://www.npmjs.com/package/@phala/dstack-sdk)** - Application info and quote retrieval * **[Smart Contracts](https://github.com/Dstack-TEE/dstack/tree/master/kms/auth-eth/contracts)** - DstackKms and DstackApp source * **[trust-center](https://github.com/Phala-Network/trust-center)** - Complete reference implementation ## Real-World Example: Confidential AI For a complete implementation of platform verification, see how Confidential AI verifies: * Hardware stack (NVIDIA GPUs + Intel TDX) * OS integrity from reproducible builds * Application code via compose-hash * KMS trust for key derivation * Request/response integrity signatures Learn more: **[Confidential AI Verification](/phala-cloud/confidential-ai/verify/overview)** ## Next Steps Verify your application code and Docker images Security audit checklist for 100% verification Understand all quote fields and measurements Verify TLS certificate binding for custom domains # Verify Your Application Source: https://docs.phala.com/phala-cloud/attestation/verify-your-application Prove your exact Docker containers run unmodified in the TEE. Prove your exact Docker containers run unmodified in the TEE. Verify no one substituted different code. ## What Should You Verify? **Basic verification:** Confirm your application code (reportData and compose-hash) runs in genuine TEE hardware. Suitable for most use cases. **Advanced verification:** Everything from Basic, plus RTMR3 event log replay, on-chain governance checks, and source code provenance. Required for high-assurance applications (DeFi, confidential AI, sensitive data). ## Basic Verification Verify your application code runs in genuine TEE hardware. This verification confirms: * Your specific data (reportData) is present in the quote * Your application configuration (compose-hash) matches what's deployed * The quote comes from genuine Intel TDX hardware This code runs on the **verifier side** (end users or auditors checking your CVM), not inside the CVM itself. ### Prerequisites Your application must expose attestation endpoints. See [Get Attestation](/phala-cloud/attestation/get-attestation) for how to set this up inside your CVM. Fetch all required data from your CVM: ```javascript TypeScript theme={"system"} // Fetch attestation quote const attestResponse = await fetch('https://your-app.example.com/attestation'); const attestData = await attestResponse.json(); const quote = attestData.quote; const eventLog = attestData.event_log; const reportData = attestData.report_data; // Fetch application configuration const infoResponse = await fetch('https://your-app.example.com/info'); const appInfo = await infoResponse.json(); const appComposeConfig = appInfo.tcb_info.app_compose; ``` ```python Python theme={"system"} import requests # Fetch attestation quote attest_response = requests.get('https://your-app.example.com/attestation') attest_data = attest_response.json() quote = attest_data['quote'] event_log = attest_data['event_log'] report_data = attest_data['report_data'] # Fetch application configuration info_response = requests.get('https://your-app.example.com/info') app_info = info_response.json() app_compose_config = app_info['tcb_info']['app_compose'] ``` ### Step 1: Verify reportData Check that reportData contains the expected challenge or public key: ```javascript TypeScript theme={"system"} import assert from 'assert'; const expectedChallenge = 'your-expected-challenge-hex'; assert(reportData.startsWith(expectedChallenge), 'reportData mismatch'); ``` ```python Python theme={"system"} expected_challenge = 'your-expected-challenge-hex' assert report_data.startswith(expected_challenge), 'reportData mismatch' ``` ### Step 2: Verify compose-hash The compose-hash is a SHA256 hash of your app-compose.json configuration file (which includes your docker-compose.yaml plus metadata). This hash proves which exact Docker images are running. Verify the configuration you fetch from the CVM matches what's attested in RTMR3: ```javascript TypeScript theme={"system"} import assert from 'assert'; import { createHash } from 'crypto'; // Calculate SHA-256 hash of app-compose const calculatedHash = createHash('sha256') .update(appComposeConfig) .digest('hex'); // Extract attested hash from RTMR3 event log const events = JSON.parse(eventLog); const composeEvent = events.find(e => e.event === 'compose-hash'); const attestedHash = composeEvent.event_payload; // Verify hashes match assert.strictEqual(calculatedHash, attestedHash, 'compose-hash mismatch'); ``` ```python Python theme={"system"} import hashlib import json # Calculate SHA-256 hash of app-compose calculated_hash = hashlib.sha256(app_compose_config.encode()).hexdigest() # Extract attested hash from RTMR3 event log events = json.loads(event_log) compose_event = next(e for e in events if e['event'] == 'compose-hash') attested_hash = compose_event['event_payload'] # Verify hashes match assert calculated_hash == attested_hash, 'compose-hash mismatch' ``` ### Step 3: Verify quote signature Verify the TDX quote was signed by genuine Intel hardware. This uses Phala Cloud's verification API (alternatively, you can use [dcap-qvl](https://github.com/Phala-Network/trust-center) locally for trustless verification): ```javascript TypeScript theme={"system"} import assert from 'assert'; const verifyResponse = await fetch('https://cloud-api.phala.com/api/v1/attestations/verify', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ hex: quote }) }); const result = await verifyResponse.json(); assert(result.quote.verified, 'Hardware verification failed'); ``` ```python Python theme={"system"} import requests verify_response = requests.post( 'https://cloud-api.phala.com/api/v1/attestations/verify', json={'hex': quote} ) result = verify_response.json() assert result['quote']['verified'], 'Hardware verification failed' ``` You've verified your application code runs in genuine TEE hardware. ## Advanced Verification For high-assurance applications, verify additional security properties beyond Basic Verification: * **RTMR3 event log replay**: Cryptographically prove the boot sequence * **Docker image digests**: Ensure images are pinned to specific versions * **On-chain governance**: Verify only authorized configs can run * **Source code provenance**: Link images back to audited source code All verification steps **must run outside the CVM**. Only verification that happens outside the TEE provides security guarantees. ### Prerequisites Complete [Basic Verification](#basic-verification) first. You'll need the same data (quote, event\_log, app\_compose, calculated hash). ### Step 1: Verify RTMR3 event log replay RTMR3 uses a hash chain where each event extends the previous value. The event log records three events during boot: 1. `compose-hash` - Your application configuration 2. `instance-id` - Unique CVM instance identifier 3. `key-provider` - KMS that distributed encryption keys The hash chain mechanism ensures no one can modify the event log without breaking the chain. Each new value is computed as: `RTMR3_new = SHA384(RTMR3_old || SHA384(event))`. To verify this: 1. Parse the event log and extract all RTMR3 events (where `imr === 3`) 2. Start with initial RTMR3 value (48 zero bytes) 3. For each event digest in order: * Pad digest to 48 bytes if needed * Compute: `RTMR3 = SHA384(RTMR3 || digest)` 4. Compare final replayed RTMR3 with the `rtmr3` value from the quote **TODO**: The dstack SDK does not currently export a `replayRtmrs()` utility function for external verifiers. For production use, refer to the [Rust verification implementation](https://github.com/Dstack-TEE/dstack/blob/d3275d1e0a0f4e2191273f0d9df62986f3d6acca/verifier/src/verification.rs#L26-L149) or use [trust-center](https://github.com/Phala-Network/trust-center) for complete RTMR3 replay verification. ### Step 2: Verify Docker image digests Docker images must be pinned by SHA256 digest, not mutable tags. Tags like `nginx:latest` can point to different images over time, breaking verification. Verify all images in your app-compose use immutable digests: ```javascript TypeScript theme={"system"} import assert from 'assert'; // Parse app-compose and extract docker-compose const appCompose = JSON.parse(appComposeConfig); const dockerComposeYaml = appCompose.docker_compose_file; // Check all image lines use @sha256 digests const imageLines = dockerComposeYaml.split('\n').filter(line => line.trim().startsWith('image:') ); for (const line of imageLines) { assert( line.includes('@sha256:'), `Image not pinned by digest: ${line.trim()}` ); } ``` ```python Python theme={"system"} import json import yaml # Parse app-compose and extract docker-compose app_compose = json.loads(app_compose_config) docker_compose = yaml.safe_load(app_compose['docker_compose_file']) # Check all services use @sha256 digests for service_name, service in docker_compose.get('services', {}).items(): image = service.get('image', '') assert '@sha256:' in image, f'Image not pinned by digest: {service_name}' ``` Example of proper image pinning: ```yaml theme={"system"} services: app: # BAD: Mutable tag image: nginx:latest # GOOD: Immutable digest image: nginx@sha256:eee5eae48e79b2e75178328c7c585b89d676eaae616f03f9a1813aaed820745a ``` Get the digest for any image: ```bash theme={"system"} docker pull nginx:latest docker inspect nginx:latest | grep -A 1 RepoDigests ``` ### Step 3: Verify on-chain governance (optional) If your application uses on-chain governance, verify the compose-hash is whitelisted. This ensures only authorized application versions can boot: ```bash theme={"system"} # Check if compose-hash is authorized in DstackApp contract cast call "allowedComposeHashes(bytes32)" # Returns true if whitelisted, false otherwise # Monitor for new authorized versions cast logs --event "ComposeHashAdded(bytes32,address)" ``` ### Step 4: Verify source code provenance Link Docker image digests back to audited source code using cryptographic build provenance. **Recommended: Sigstore for GitHub builds** Sigstore cryptographically links container images to specific GitHub commits and workflows. When you build images via GitHub Actions, Sigstore automatically signs the build with GitHub's identity, creating unforgeable proof. Platform components use Sigstore: * [dstack-gateway releases](https://github.com/Dstack-TEE/dstack/releases?q=%22dstack-gateway%22\&expanded=true) * [dstack-kms releases](https://github.com/Dstack-TEE/dstack/releases?q=%22dstack-kms%22\&expanded=true) * [dstack-verifier releases](https://github.com/Dstack-TEE/dstack/releases?q=%22dstack-verifier%22\&expanded=true) Set up Sigstore signing in your GitHub Actions workflow to provide the same cryptographic build provenance for your applications. **Alternative: Reproducible builds** For maximum verifiability, use reproducible builds where anyone can rebuild from source and get identical digests. Publish your Dockerfile and build instructions. This requires careful environment control but removes the need to trust any build service. ## Attack Scenarios Prevented **Compromised Docker registry:** The TEE verifies image digests against compose-hash before pulling. Even if the registry serves malicious images, digest verification fails. **Code substitution:** The compose-hash in RTMR3 is recorded by TEE hardware during boot. No one can modify it without invalidating the attestation quote signature. **Unauthorized updates:** On-chain governance (if enabled) ensures only whitelisted compose hashes can boot. Malicious developers cannot deploy unapproved versions. **Event log tampering:** RTMR3 event log replay cryptographically proves the events are authentic. Modification causes replay to produce different RTMR3 that won't match the quote. ## Verification Checklist Complete application verification requires checking: * [ ] All Docker images use SHA256 digests (not tags) * [ ] Calculated compose-hash matches expected value * [ ] RTMR3 event log replays to quoted RTMR3 value * [ ] Attested compose-hash matches calculated hash * [ ] compose-hash is whitelisted on-chain (if using governance) * [ ] reportData contains expected challenge/public key * [ ] Image digests link to audited source code (Sigstore or reproducible builds) ## Tools and Resources * **[dstack SDK](https://www.npmjs.com/package/@phala/dstack-sdk)** - `getComposeHash()` and `replayRtmrs()` methods * **[RTMR3 Calculator](https://rtmr3-calculator.vercel.app/)** - Web tool for compose-hash calculation * **[Verification Script](https://github.com/Dstack-TEE/dstack-examples/blob/main/attestation/rtmr3-based/verify.py)** - Complete Python example * **[trust-center](https://github.com/Phala-Network/trust-center)** - Reference implementation ## Next Steps Verify OS, KMS, and infrastructure security Security audit checklist for 100% verification Understand all quote fields and measurements Complete verification service documentation # Cases Explorer Source: https://docs.phala.com/phala-cloud/cases/best-practices-and-successful-case Explore real-world use cases showing how Phala Cloud enables secure, verifiable AI and trustless applications using TEE technology. Explore how Phala Cloud enables secure, verifiable AI and trustless applications using TEE technology. [https://cloud.phala.com/cases](https://cloud.phala.com/cases) # Overview Source: https://docs.phala.com/phala-cloud/cases/overview Explore real-world applications and use cases for TEE technology. TEE provide hardware-based isolation for sensitive computations, ensuring data confidentiality and integrity. This document explores various use cases where TEEs can be combined with other technologies to create powerful, secure solutions. ## Industry Success Stories Explore real-world implementations across different sectors: * [Financial Services](https://phala.com/success-stories/financial-services) - Confidential AI for banking and trading * [Healthcare & Research](https://phala.com/success-stories/healthcare-research) - HIPAA-compliant AI and medical data privacy * [B2B SaaS Platforms](https://phala.com/success-stories/ai-saas-platform) - Enterprise AI privacy and multi-tenant security * [Decentralized AI](https://phala.com/success-stories/decentralized-ai) - Web3 and blockchain AI applications ## TEE + AI TEEs can protect AI model training and inference, preserving both model privacy and data confidentiality. This combination enables secure machine learning on sensitive data while preventing unauthorized access to proprietary models. TEE also can be used to build unruggable AI Agents that operate with the same level of trustworthiness as smart contracts, check [the article by Phala CEO Marvin Tong](https://x.com/marvin_tong/status/1866319231350100330) to see why we get there. Check out [Building Confidential AI with TEE](/phala-cloud/cases/tee_with_ai) ## TEE + FHE/MPC TEEs can complement Fully Homomorphic Encryption (FHE) and Multi-Party Computation (MPC) to create hybrid systems with enhanced performance and security guarantees. These combinations allow for secure computation on encrypted data with reduced overhead. Check out [TEE with FHE and MPC](/phala-cloud/cases/tee_with_fhe_and_mpc) ## Private Proving with TEE **Private Proving** runs zero-knowledge provers inside TEE to protect sensitive data during proof generation. While ZK proofs verify computation correctness, they don't prevent the prover from accessing your private data. Running ZK provers in TEE solves this critical problem - your transaction data, API keys, AI models, and business logic stay hardware-encrypted even during proving. Phala Cloud provides production-ready infrastructure for private proving with GPU TEE (H100/H200), enabling developers to generate ZK proofs at hardware speed while keeping sensitive inputs encrypted at the silicon level. Check out [Private Proving with TEE](/phala-cloud/cases/tee_with_zk_and_zkrollup) # Building Confidential AI Source: https://docs.phala.com/phala-cloud/cases/tee_with_ai Build confidential AI applications with Trusted Execution Environment technology. Check Phala Network's [2025 dAGI report](https://phala.com/reports/2025RealCodeForRealdAGI.pdf) to see how TEE is used in AI full stack. ## Introduction Phala Network 2025 dAGI report cover showing AI robot building with data structures and partner ecosystem logos AI has become a transformative force across industries, but its widespread adoption faces significant challenges related to data privacy, model security, and computational trust. As AI systems process increasingly sensitive information and make critical decisions, ensuring the confidentiality and integrity of both the data and the AI models themselves has become paramount. TEE offer a compelling solution to these challenges by providing hardware-enforced isolation for AI workloads. By running AI models and AI agents programs within TEE, organizations can protect sensitive data, prevent unauthorized access to proprietary models, and provide verifiable guarantees about the execution environment. ## The Need for TEE in AI Systems The integration of TEE technology with AI addresses several critical challenges: * **Data Privacy Concerns**: AI training and inference often require access to sensitive personal, financial, or proprietary data. TEEs ensure this data remains encrypted and protected even during processing. * **Model Protection**: Valuable AI models represent significant intellectual property that needs protection from theft, tampering, or unauthorized access. TEEs can safeguard these assets throughout their lifecycle. * **Verifiable Computation**: Users need assurance that AI computations are performed correctly and that results haven't been manipulated. TEE attestation provides cryptographic proof of the execution environment's integrity. * **Preventing Data Leakage**: Even in systems designed for privacy, subtle information leakage can occur through side channels or model outputs. TEEs provide an additional layer of protection against such vulnerabilities. Confidential AI inference architecture diagram showing data flow through TEE-protected environment ## Key Benefits of TEE for AI Applications 1. **Confidential AI Inference**: Run inference on sensitive data without exposing either the data or the model to the host system or service provider. 2. **Secure Model Training**: Protect training data and emerging model parameters during the training process, preventing data poisoning or model extraction attacks. 3. **Trustless AI Marketplaces**: Enable secure sharing and monetization of AI models without risking intellectual property theft. 4. **Verifiable AI Results**: Provide cryptographic proof that AI outputs were generated by specific models running in secure environments, enhancing trust in critical applications. 5. **Regulatory Compliance**: Meet stringent data protection requirements by ensuring data remains encrypted throughout the AI pipeline. 6. **Autonomous AI Agents**: Enable AI agents to securely manage cryptographic keys and digital assets without exposing sensitive credentials. ## Case Studies ### TEE in Agent Frameworks
🤖 Eliza: Secure AI Agent Framework with TEE Integrationtee\_eliza.pngEliza TEE Plugin
🎮 Virtuals: Game Agents Running in Secure TEE Environmenttee\_game\_agents.pngVirtuals Game Agent
### Other TEE-AI Integrations
📡 Vana Network: Decentralized Data Privacy with TEEtee\_vana.pngPhala x Vana
🌐 0G: Confidential AI Computing in Decentralized Networkstee\_0g.pngPhala x 0G
💰 Fairblock: Unruggable AI with MPC-TEE Hybrid Architecturetee\_fairblock.pngPhala x Fairblock
🌱 Spore.fun: Autonomous AI Evolution in Secure Environmentstee\_spore.pngSpore.fun
# 2FA for FHE/MPC Systems Source: https://docs.phala.com/phala-cloud/cases/tee_with_fhe_and_mpc Combine TEE with FHE and MPC for enhanced privacy-preserving computations. ## Introduction Fully Homomorphic Encryption (FHE) and Multi-Party Computation (MPC) are foundational technologies for privacy-preserving computation, enabling operations on encrypted data without decryption. However, their adoption in real-world systems, particularly blockchain and decentralized networks, faces challenges such as computational overhead, key management risks, and trust assumptions. This chapter explores how Phala Network's TEE can act as a 2FA mechanism to enhance the security and practicality of FHE/MPC workflows. Check out earlier thoughts on **[SGX as 2FA for FHE/MPC](https://ethresear.ch/t/sgx-as-2fa-for-fhe-mpc/19780)** and **[Drawbacks in FHE Blockchain and how TEE can help](https://collective.flashbots.net/t/drawbacks-in-fhe-blockchain-and-how-tee-can-help-it/3642)** ## Challenges in FHE/MPC Systems **Key Management Risks** * Secure key generation, storage, and usage are critical vulnerabilities * Key compromise threatens data confidentiality and computation integrity * Software-based solutions remain susceptible to memory attacks and insider threats **Performance Limitations** * FHE introduces significant computational overhead, impractical for time-sensitive applications * MPC reduces individual computation but increases network communication and coordination costs * Both technologies face scalability challenges in high-throughput environments **Trust Vulnerabilities** * Systems rely on honest-majority assumptions that weaken with participant count * Collusion attacks become feasible when economic incentives align for malicious actors * Lack of accountability mechanisms when malicious behavior occurs * Threshold schemes vulnerable to withholding attacks that prevent result finalization ## TEE as a 2FA Mechanism: Architectural Overview TEEs provide hardware-enforced isolation for sensitive operations, combining the benefits of TEE with cryptographic protocols. When integrated with FHE/MPC, TEEs act as a secondary trust layer, ensuring: * **Secure Key Generation/Storage**: Cryptographic keys are generated and stored within the TEE, isolated from the host OS or untrusted applications. * **Computation Integrity**: Critical operations (e.g., decryption of FHE results or MPC coordination) are verified within the TEE. * **Attestation**: Remote parties can cryptographically verify that computations were executed in a genuine TEE. ### Workflow Example * MPC nodes build a docker image and deploy it to [Phala Cloud](https://cloud.phala.com/register?invite=PHALAWIKI), see the [tutorial](/phala-cloud/cvm/create-with-docker-compose). 1. A master key is generated inside an TEE and never exposed externally. 2. The MPC node signs a public verification key, which is shared with the network. 3. The MPC node generate a attestation proof that prove the key generation and storage are done in a genuine TEE. For FHE Computation: 4. Users encrypt data using FHE and send to FHE server. 5. FHE finished the computation and encrypt the result with the MPC key. 6. The MPC nodes in TEE decrypting intermediate results and return the result to users. ## Case Studies
🔐 Fairblock: Building Unruggable AI with an MPC-TEE Hybrid Architecturefairblock\_tee\_registry.pngFairblock GitHub
🗳️ Mind Network: Leverage TEE and FHE Build Blind Votingfhe\_tee\_voting.pngMind Network Case Study
# Private Proving with TEE Source: https://docs.phala.com/phala-cloud/cases/tee_with_zk_and_zkrollup Private Proving - Running ZK provers inside TEE to protect sensitive data during proof generation. ## What is Private Proving? **Private Proving** is a method of generating cryptographic proofs inside a Trusted Execution Environment (TEE) that ensures input data (the witness) remains completely confidential during proof generation. This means neither the verifier nor the prover can see an application's input data. Private Proving Overview - Source: Succinct Labs ### The Problem: Witness Data Exposure in Traditional ZK Proving Traditional zero-knowledge proving systems verify computation correctness, but the witness data (private inputs) is exposed to the prover operator during proof generation: * **Privacy-focused rollups**: Prover operators can see user transaction amounts, recipients, and trading strategies * **Private DeFi protocols**: Witness data includes sensitive positions, balances, and trading logic * **Confidential identity systems**: User credentials and personal data are exposed to prover infrastructure * **Private payments**: Transaction amounts and sender/receiver information visible to prover **The core issue**: Zero-knowledge proofs hide data from the verifier, but they don't protect data from the prover. The prover sees all witness data during proof generation. ### The Solution: Run ZK Provers Inside TEE Private proving runs the proof generation process inside hardware-isolated environments, ensuring witness data never leaves the secure enclave: ``` Witness Data (Private) → TEE [zkVM Prover] → ZK Proof (Public) ``` **How it works**: * Witness data enters the TEE and remains hardware-encrypted throughout proving * The zkVM (like SP1) executes inside the TEE to generate the proof * Only the final proof leaves the TEE - witness data stays protected * TEE attestation proves the proving environment was secure **Key benefits**: 1. **Complete Witness Privacy**: No one, including the prover operator, can see witness data 2. **Hardware-Level Security**: Data encrypted at the silicon level (Intel TDX, AMD SEV, AWS Nitro) 3. **Developer-Friendly**: Write private applications in Rust using familiar zkVM frameworks 4. **Performance**: GPU TEE (H200) provides hardware acceleration for fast proof generation 5. **Verifiable Security**: Remote attestation cryptographically proves the secure execution environment **Impact**: Private proving transforms privacy from a specialized technical challenge into a standard feature. As Succinct states: "With Succinct, privacy will be a standard rather than an exception." Learn more about Private Proving from Succinct Labs' [Private Proving announcement](https://blog.succinct.xyz/private-proving/). ## Benefits vs Traditional ZK | Feature | Traditional ZK | Private Proving (ZK in TEE) | | ------------------------ | ------------------- | ----------------------------- | | **Proof Correctness** | ✅ Cryptographic | ✅ Cryptographic | | **Witness Privacy** | ❌ Exposed to prover | ✅ Hardware encrypted | | **Verifiable Execution** | ❌ No guarantees | ✅ TEE attestation | | **Developer Experience** | Circuit development | ✅ No code changes (Rust/zkVM) | | **Performance** | Fast | Minimal overhead with GPU TEE | | **GPU Acceleration** | ✅ Yes | ✅ Yes (H100/H200) | | **Trust Assumptions** | Cryptography only | Cryptography + hardware | ## Private Proving in Production ### Phala Cloud: Hardware-Accelerated Private Proving Phala Cloud provides production-ready infrastructure for private proving with GPU TEE:
🚀 Phala Network: Run zk-Rollup STF (state transition function) in TEE Use-Cases-Multi-Proof.png Phala SGXProver
🔒 Primus: Build Trustless zkTLS with TEE tee\_zktls.png Primus zkTLS
💾 Running SP1 zkVM in TEE H200 with Low Overhead zkvm-sp1-in-gpu-tee.jpg SP1 zkVM in TEE
## Multi-Proof Systems: Defense-in-Depth (SP1-2FA) While private proving protects data privacy, some applications require maximum security through defense-in-depth. **Multi-proof systems** like [Succinct's SP1-2FA](https://blog.succinct.xyz/sp1-2fa/) generate both ZK and TEE proofs in parallel: SP1-2FA Multi-Proof Architecture - Source: Succinct Labs **How it differs from private proving**: * **Private Proving**: ZK prover runs *inside* TEE (focus: data privacy) * **SP1-2FA**: ZK and TEE run in *parallel* (focus: maximum security) **When to use SP1-2FA**: * Critical infrastructure (blockchain consensus, bridges) * High-value DeFi protocols (significant TVL at risk) * Defense against undiscovered ZK circuit bugs * Regulatory requirements for multiple independent proofs **Trade-off**: SP1-2FA provides stronger security guarantees but doesn't specifically protect data privacy during ZK proving. For most applications requiring data privacy, private proving is the right choice. Learn more about SP1-2FA from [Succinct's blog post](https://blog.succinct.xyz/sp1-2fa/). **Resources**: * [Getting Started Guide](/phala-cloud/getting-started/overview) * [Performance Benchmarks](/phala-cloud/references/performance-report) * [TEE Attestation](/phala-cloud/attestation/overview) * [Deploy with Docker](/phala-cloud/cvm/overview) ## References and Further Reading * **Succinct Labs**: [Private Proving Blog Post](https://blog.succinct.xyz/private-proving/) | [SP1-2FA Technical Details](https://blog.succinct.xyz/sp1-2fa/) * **Vitalik Buterin**: [Multi-Proof Systems Presentation](https://hackmd.io/@vbuterin/zk_slides_20221010#/) * **Justin Drake**: [2FA ZK-Rollups with SGX](https://ethresear.ch/t/2fa-zk-Rollups-using-sgx/14462) * **Phala Network**: [SP1 zkVM Performance Benchmark](https://phala.com/posts/performance-benchmark-running-sp1-zkvm-in-tee-h200-with-low-overhead) # Benchmark Source: https://docs.phala.com/phala-cloud/confidential-ai/benchmark Performance benchmarks and metrics for LLM inference in GPU TEE environments. ## Key Results **TEE mode on H100/H200 GPUs runs up to 99% efficiency, nearly matching native performance.** This means you get confidential computing with minimal performance penalty. The efficiency improves as your workload grows. Larger models and longer sequences see even better performance relative to native execution. ## What We Tested We benchmarked LLM inference in GPU TEE mode using NVIDIA H100 and H200 GPUs. TEE mode encrypts your data and computation while it runs on the GPU. Performance comparison table showing TPS and QPS metrics for LLama-3.1-8B, Phi3-14B-128k, and LLama-3.1-70B models on H100 and H200 GPUs with TEE-on vs TEE-off overhead percentages ## Performance Insights * **Larger models perform better in TEE mode.** Models like Phi3-14B-128k and Llama3.1-70B show higher efficiency because they spend more time computing and less time on I/O operations. * **Longer sequences boost efficiency.** When you process more tokens (input + output), the computation-to-I/O ratio improves. This makes TEE overhead less noticeable. * **The sweet spot: High-computation workloads.** TEE mode shines when your GPU is doing heavy lifting. The encryption/decryption overhead becomes trivial compared to the actual computation time. ## Resources For detailed metrics and analysis, check our [benchmark paper](https://arxiv.org/pdf/2409.03992). See detailed [GPU TEE performance specifications](https://phala.com/gpu-tee) for production deployments, including [H100 TEE](https://phala.com/gpu-tee/h100) and [H200 TEE](https://phala.com/gpu-tee/h200) configurations. # Deploy and Verify GPU TEE Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-gpu/deploy-and-verify Deploy or fine-tune AI models with hardware-level security using TEE-protected GPUs and verify they run in genuine TEE hardware GPU TEE gives you dedicated H100, H200, or B200 GPUs running inside trusted execution environments for custom AI workloads. You get full control over your environment with Docker container support, and you can verify hardware authenticity using NVIDIA's local verification tools. This option works when you need to train, fine-tune, or run inference on proprietary datasets with custom code. For standard LLM inference, the [Confidential AI API](/phala-cloud/confidential-ai/confidential-model/confidential-ai-api) or [Model Template](/phala-cloud/confidential-ai/confidential-gpu/model-template) are simpler alternatives. Each instance includes NVIDIA Driver `570.133.20` and CUDA `12.8`. You can scale from 1 to 8 GPUs per instance. ## Prerequisites * Phala Cloud account with sufficient credits * Basic understanding of Jupyter notebooks * Familiarity with command-line tools ## Step 1: Deploy GPU TEE instance ### Launch the deployment wizard Sign in to [cloud.phala.com](https://cloud.phala.com), click **GPU TEE** in the navigation bar, then click **Start Building** to open the Launch GPU Instance wizard. Check your credit balance in the upper right corner. GPU instances incur hourly charges, so confirm your balance before launching. ### Choose GPU hardware Select your GPU type based on your compute needs. GPU TEE hardware selection interface showing H100, H200, and B200 GPU options with specifications Available options include: | GPU type | Region | vCPU cores | VRAM | RAM | Storage | Price\* | | -------- | ------ | ---------- | ------ | ------ | ------- | --------------- | | H200 | US | 24 | 141 GB | 256 GB | 200 GB | \$2.56/GPU/hour | | H200 | India | 15 | 141 GB | 384 GB | 200 GB | \$2.30/GPU/hour | | B200 | US | 12 | 180 GB | 192 GB | 200 GB | \$3.80/GPU/hour | \*Pricing may vary. Check the dashboard for current rates. Click your preferred GPU card to highlight it in green. ### Configure GPU count Choose the number of GPUs for your instance. You can scale from **1 to 8 GPUs** per instance. The UI updates resource totals dynamically: | GPU count | Example: B200 | Total vCPU | Total VRAM | Total RAM | Total storage | | --------- | ------------- | ---------- | ---------- | --------- | ------------- | | 1 GPU | Single | 12 cores | 180 GB | 192 GB | 200 GB | | 8 GPUs | Multi | 96 cores | 1 TB | 1 TB | 1 TB | ### Configure deployment Give your deployment a name or use the auto-generated name like `gpu-tee-1p1qp`. For the template, choose **Jupyter Notebook (PyTorch)** to get a GPU-accelerated JupyterLab environment with PyTorch and CUDA pre-installed. This template works well for running verification scripts and custom experiments. You can also choose vLLM for an inference server or Custom Configuration to provide your own Docker Compose file. For this tutorial, we'll use Jupyter Notebook because it gives us terminal access to run verification commands. You can always deploy different containers later. Your initial template choice isn't permanent. ### Select pricing plan Choose a commitment period: | Plan | Rate | Notes | | ------------------ | --------------------------- | ------------------------------------------ | | 6-month commitment | \~\$2.88/GPU/hour | Includes storage, saves \~18% vs on-demand | | 1-month commitment | \~\$3.20/GPU/hour | Includes storage, short-term commitment | | On-Demand | \~\$3.50/GPU/hour + storage | Pay-as-you-go, no commitment | Review the **Pricing Summary** showing estimated costs per hour, day, and month. ### Launch instance Before launching, review the **Instance Summary** to confirm your GPU model and count, VRAM, RAM, and storage allocations, plus your total estimated costs. GPU TEE order summary showing selected hardware configuration, pricing breakdown, and submit order button Click **Launch Instance** when you're ready to proceed. Launching creates hourly charges. Confirm your configuration and budget before proceeding. Provisioning takes approximately 1 day. ## Step 2: Access your GPU TEE instance After provisioning completes, your instance appears under the **GPU TEE** tab with connection details including the JupyterLab URL. Navigate to the **GPU TEE** tab in your dashboard and find your instance in the GPU Instances list. Click **View Details** to see the JupyterLab URL, then open that URL in your browser to access your instance. Monitor provisioning status in the GPU Instances list. Instances progress from **Preparing** → **Starting** → **Running**. ## Step 3: Verify GPU TEE attestation Open a terminal in JupyterLab (**File** → **New** → **Terminal**) to verify your instance runs on genuine TEE hardware. ### Check GPU and TEE status First, confirm your GPU is detected and confidential compute mode is active. Run `nvidia-smi` to check GPU status: ```bash theme={"system"} nvidia-smi ``` Look for your GPU model (H100/H200/B200), driver version 570.133.20, and CUDA version 12.8. Then check confidential compute status: ```bash theme={"system"} nvidia-smi conf-compute -q ``` Expected output: ``` # nvidia-smi conf-compute -q ==============NVSMI CONF-COMPUTE LOG============== CC State : ON Multi-GPU Mode : None CPU CC Capabilities : INTEL TDX GPU CC Capabilities : CC Capable CC GPUs Ready State : Ready ``` The key indicators are `CC State: ON` and `CPU CC Capabilities: INTEL TDX`, confirming your instance runs in TEE mode. ### Run attestation verification Install NVIDIA's attestation verification tools: ```bash theme={"system"} pip install nv-local-gpu-verifier nv_attestation_sdk ``` Run the verifier to get cryptographic proof of hardware authenticity: ```bash theme={"system"} python -m verifier.cc_admin ``` The verifier confirms your GPUs are genuine NVIDIA devices, checks confidential compute mode is enabled, verifies driver and firmware versions, and generates cryptographic evidence of your TEE status. Successful verification means your GPU hardware is authentic, confidential compute mode is active, and the driver version matches the expected TEE-enabled version. If verification fails, do not use the instance for confidential workloads. Contact [Phala Support](/phala-cloud/support) with the error details. ## Step 4: Confirm GPU functionality Verify GPU functionality with PyTorch. Open a new notebook in JupyterLab (**File** → **New** → **Notebook**) and run: ```python theme={"system"} import torch print(f"CUDA available: {torch.cuda.is_available()}") print(f"GPU: {torch.cuda.get_device_name(0)}") ``` This confirms PyTorch can detect and access the GPU. ## Next steps You've deployed and verified a GPU TEE instance! Now you can: Learn how to fetch and verify attestations programmatically Understand how GPU and CPU attestations create a complete trust chain Make your GPU TEE workloads accessible over HTTPS # Dedicated Models Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-gpu/model-template Deploy dedicated GPU resources for AI models with hourly pricing. Same models as the API but with dedicated performance. Dedicated Models let you host large language models on dedicated GPU hardware inside trusted execution environments (TEEs). You get the same pre-deployed models as the [Confidential AI API](/phala-cloud/confidential-ai/confidential-model/confidential-ai-api), but with dedicated resources and hourly pricing instead of per-request billing. Choose this option when you need predictable performance or cost efficiency for high-volume inference workloads. For custom workloads beyond pre-deployed models, see [GPU TEE](/phala-cloud/confidential-ai/confidential-gpu/deploy-and-verify). This option uses the same OpenAI-compatible API as API Access, including all advanced features like tool calling and vision. The only difference is dedicated GPU resources with hourly billing instead of shared infrastructure. ## Prerequisites * Phala Cloud account with sufficient credits * Basic familiarity with LLM deployment ## Step 1: Deploy your model ### Access the deployment page Sign in to your Phala Cloud account and select **Confidential AI Models** from the top navigation bar. Confidential AI Models ### Choose a model Locate the model you want to deploy (e.g., `qwen-2.5-7b-instruct`) and click **Deploy** to open the deployment wizard. Review the model description to understand its capabilities. For example, Qwen 2.5 models are instruction-tuned language models optimized for chat and task completion. Don't see your preferred model? We can add custom models—reach out through [Support](/phala-cloud/support). ### Configure deployment Fill out the deployment form with these details: **Service Name**: Give your deployment a memorable name (e.g., `qwen-2-5-7b-instruct`). Use the auto-generated name or create your own. **Node & Image**: Select your preferred deployment location and image version. Available options depend on current capacity (e.g., `gpu1 us-west-1` with `dstack-nvidia-0.5.0`). **Resource Plan**: Review the hardware allocation for your chosen model. For example, `qwen-2.5-7b-instruct` typically includes: * H200 GPU with 141 GB VRAM * 24 vCPUs * 256 GB RAM * 40 GB disk * \$3.20 per hour pricing **Scheduled Destroy**: Set an auto-destruction date to avoid ongoing charges, or leave empty for manual control. ### Launch the deployment Review your configuration and click **Deploy** to launch. Provisioning takes a few minutes. Deploy LLM Model Monitor the status in the Active Workloads section on the Confidential AI Models page. Your model will progress through these states: * **Preparing**: Allocating resources * **Starting**: Launching the model * **Running**: Ready to use Billing starts when the model enters Running status. Remember to stop or schedule destruction when you no longer need the instance. ## Step 2: Access Open WebUI Once your model reaches Running status, you can access it through Open WebUI, a browser-based chat interface. ### Open the interface When the workload shows **RUNNING** status, a **Connect UI** button appears under the model card. The button initially reads "Initializing..." while the interface loads. Click it when active to open a new tab. Model Details Page ### Create admin account On your first visit, create an admin account with your name, email, and password. Click **Create Admin Account** to continue. **Critical**: Choose a strong password and store it securely. Because your model runs in a confidential environment, even we (the cloud provider) cannot access or reset your password. If you forget it, you'll lose access to all chat history permanently. After account creation, click **Okay, Let's Go!** to access the chat interface. ## Step 3: Interact with your model Open WebUI provides a standard chat interface. Type your questions in the prompt box, manage conversation history in the left sidebar, and use the tools menu (+) for advanced features like file uploads. The interface is intuitive and works like other chat applications you may be familiar with. Open WebUI Chat Interface ## Step 4: Access via API Beyond the WebUI, you can access your dedicated model programmatically through a REST API. Navigate to the **Overview** page for your running instance. In the **Inference API** section, click **Quickstart** to see example commands and integration code. To make API calls, you'll need an API key from Open WebUI. Open the WebUI, navigate to **Settings** → **Account** → **API keys**, and create a new key. Use this key as a Bearer token in your requests. Protect your API key like a password. Anyone with the key can access your model. Here's a quick example for chat completions: ```bash theme={"system"} curl -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-your-api-key" \ -d '{ "model": "qwen/qwen-2.5-7b-instruct", "messages": [ {"role": "user", "content": "Hello! Can you tell me a fun fact about space?"} ] }' \ https:///v1/chat/completions ``` For TEE verification, your instance exposes attestation endpoints: ```bash theme={"system"} # Get attestation report curl -H "Authorization: Bearer sk-your-api-key" \ "https:///v1/attestation/report?model=MODEL_NAME" # Get signature for a chat completion curl -H "Authorization: Bearer sk-your-api-key" \ "https:///v1/signature/{chat_completion_id}?model=MODEL_NAME&signing_algo=ecdsa" ``` See [Verify Attestation](/phala-cloud/confidential-ai/verify/verify-attestation) and [Verify Signature](/phala-cloud/confidential-ai/verify/verify-signature) for how to use these endpoints to confirm your requests were processed inside a TEE. ## Managing your instance Return to the Confidential AI Models page anytime to monitor your model: **Overview button**: Opens a dashboard with resource metrics, logs, and attestation data. **Connect UI button**: Always launches the Open WebUI for your model. Billing continues while the instance is running. Stop or schedule destruction when you no longer need the model to avoid ongoing charges. ## Troubleshooting ### Connect UI button stuck on "Initializing..." Wait 2-3 minutes after the model reaches Running status. The interface needs time to fully start. Refresh the Confidential AI Models page if the button doesn't activate. ### Cannot access Open WebUI Verify your model status is **Running** in the Active Workloads section. Check that you're using the correct URL (should end in `dstack-prod5.phala.network`). ### Forgot admin password We can't reset your admin password because the model is running in a confidential environment that even the cloud provider doesn't have access to. You will lose all the chat history if you forget your password. ## Next steps You've deployed a dedicated model and accessed it through Open WebUI! Now you can: Confirm your model runs in genuine TEE hardware Deploy custom models with full infrastructure control # Troubleshooting Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-gpu/troubleshooting Common issues and warnings when running GPU TEE workloads ## Serial Output Warnings These warnings appear in serial output but are safe to ignore. ### NVIDIA Fabric Manager Service Failed ``` [FAILED] Failed to start NVIDIA fabric manager service. ``` Safe to ignore for single-GPU instances. The fabric manager service is only needed for multi-GPU NVLink configurations. ### NVRM Assertion Error ``` NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256 ``` Safe to ignore. This is a known warning that doesn't affect GPU functionality. # Attestation Report Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/api-reference/attestation Fetch TEE attestation evidence for a Confidential AI model. ## Endpoint ```bash theme={"system"} GET https://api.redpill.ai/v1/attestation/report?model={model_id}&nonce={nonce}&signing_address={address} ``` The attestation report proves a model endpoint is backed by TEE hardware and provides the evidence needed for hardware, software, and signer binding checks. Always include a fresh random `nonce` when fetching attestations for security-sensitive verification. A nonce prevents replay of an older valid attestation. ## Parameters Model ID to attest. Examples: `phala/qwen3.5-27b`, `phala/qwen-2.5-7b-instruct`, `openai/gpt-oss-120b`, `z-ai/glm-5`. Fresh 32-byte random value encoded as 64 hex characters. The nonce is embedded in the TEE report data. Ethereum address or public key used to filter attestations in multi-server deployments. Use this when binding a response signature to a specific TEE signer. ## Examples ```bash cURL theme={"system"} NONCE=$(openssl rand -hex 32) curl "https://api.redpill.ai/v1/attestation/report?model=phala/qwen3.5-27b&nonce=$NONCE" \ -H "Authorization: Bearer " ``` ```python Python theme={"system"} import secrets import requests nonce = secrets.token_hex(32) response = requests.get( "https://api.redpill.ai/v1/attestation/report", params={ "model": "phala/qwen3.5-27b", "nonce": nonce, }, headers={"Authorization": "Bearer "}, ) attestation = response.json() ``` ## Response Formats The response format depends on the provider behind the model. ### Phala / NearAI Two-Layer Format Models may return separate gateway and model attestations: ```json theme={"system"} { "gateway_attestation": { "signing_address": "0x...", "signing_algo": "ecdsa", "intel_quote": "hex-encoded-tdx-quote", "event_log": [], "report_data": "...", "request_nonce": "...", "info": { "vm_config": "..." } }, "model_attestations": [ { "model_name": "phala/qwen3.5-27b", "signing_address": "0x...", "signing_algo": "ecdsa", "intel_quote": "hex-encoded-tdx-quote", "nvidia_payload": "{...json gpu attestation...}", "event_log": [], "info": { "tcb_info": "{...app_compose...}", "vm_config": "..." } } ] } ``` ### Chutes Format Some models return Chutes-style instance attestations: ```json theme={"system"} { "attestation_type": "chutes", "nonce": "...", "all_attestations": [ { "instance_id": "uuid", "nonce": "...", "intel_quote": "base64-encoded-tdx-quote", "gpu_evidence": [ { "certificate": "...", "evidence": "...", "arch": "HOPPER" } ], "e2e_pubkey": "..." } ] } ``` ### Flat Format Older Phala-native responses may expose fields at the top level: ```json theme={"system"} { "signing_address": "0x...", "signing_algo": "ecdsa", "request_nonce": "...", "intel_quote": "hex-encoded-tdx-quote", "nvidia_payload": "{...}", "info": { "tcb_info": "{\"app_compose\":\"...\"}" } } ``` ## Important Fields | Field | Description | | --------------------------- | ----------------------------------------------------- | | `signing_address` | Address or key used by the TEE to sign responses | | `signing_algo` | Signature algorithm, commonly `ecdsa` | | `request_nonce` / `nonce` | Nonce included in the attestation | | `intel_quote` | Intel TDX quote for CPU TEE verification | | `nvidia_payload` | NVIDIA GPU attestation payload | | `event_log` | Boot event log for software stack verification | | `info.vm_config` | VM configuration evidence | | `info.tcb_info.app_compose` | Docker Compose application evidence | | `gateway_attestation` | Gateway TEE attestation | | `model_attestations` | One or more model runtime attestations | | `all_attestations` | Provider-specific list of model instance attestations | ## Verification Flow 1. Generate a fresh nonce. 2. Fetch an attestation report for the exact model. 3. Verify the Intel TDX quote. 4. Verify GPU evidence when `nvidia_payload` or `gpu_evidence` is present. 5. Confirm the report data binds the nonce and expected signing address. 6. Verify application measurements such as compose hash and image provenance when available. For a walkthrough, see [Verify Attestation](/phala-cloud/confidential-ai/verify/verify-attestation). # Chat Completions Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/api-reference/chat-completions Create OpenAI-compatible chat completion responses with Confidential AI models. ## Endpoint ```bash theme={"system"} POST https://api.redpill.ai/v1/chat/completions ``` Creates a response for a chat conversation. Use the same OpenAI-compatible request shape you already use with the OpenAI SDK, then set the base URL to `https://api.redpill.ai/v1`. ## Request Body Model ID to use for completion. Examples: `phala/qwen3.5-27b`, `phala/gemma-3-27b-it`, `z-ai/glm-5`, `openai/gpt-oss-120b`. Conversation messages. Each message includes `role` and `content`. ```json theme={"system"} [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain GPU TEE in one paragraph."} ] ``` Sampling temperature. Typical range is `0` to `2`. Maximum number of output tokens for most open models and GPU TEE models. Maximum output tokens for newer OpenAI reasoning models that do not accept `max_tokens`. Set to `true` to receive server-sent event chunks. Function/tool definitions that supported models can call. Controls whether the model may call tools. Common values are `auto`, `none`, or a specific tool selection object. Requests structured output from supported models, including JSON schema mode. ## Examples ```bash cURL theme={"system"} curl https://api.redpill.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '{ "model": "phala/qwen3.5-27b", "messages": [ {"role": "user", "content": "What privacy guarantees does GPU TEE provide?"} ] }' ``` ```python Python theme={"system"} from openai import OpenAI client = OpenAI( api_key="", base_url="https://api.redpill.ai/v1", ) response = client.chat.completions.create( model="phala/qwen3.5-27b", messages=[ {"role": "user", "content": "What privacy guarantees does GPU TEE provide?"} ], ) print(response.choices[0].message.content) ``` ```typescript TypeScript theme={"system"} import OpenAI from "openai"; const client = new OpenAI({ apiKey: "", baseURL: "https://api.redpill.ai/v1", }); const response = await client.chat.completions.create({ model: "phala/qwen3.5-27b", messages: [ { role: "user", content: "What privacy guarantees does GPU TEE provide?" }, ], }); console.log(response.choices[0].message.content); ``` ## Response ```json theme={"system"} { "id": "chatcmpl-123", "object": "chat.completion", "created": 1677652288, "model": "phala/qwen3.5-27b", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "GPU TEE protects inference by..." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 16, "completion_tokens": 48, "total_tokens": 64 } } ``` The `id` field is the request ID. Use it with [Request Signature](/phala-cloud/confidential-ai/confidential-model/api-reference/signature) when you need cryptographic proof for this specific response. ## Feature Notes * Streaming uses the same `stream: true` option as the OpenAI API. * Vision models accept multimodal `content` arrays with `image_url` entries. * Tool calling uses OpenAI-compatible `tools`, `tool_choice`, assistant `tool_calls`, and tool response messages. * Structured output uses `response_format` on supported models. ## Next Steps Discover available Confidential AI models and capabilities Fetch the signature for a chat completion response # List Embedding Models Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/api-reference/embedding-models List embedding-capable models available through the Confidential AI API. ## Endpoint ```bash theme={"system"} GET https://api.redpill.ai/v1/embeddings/models ``` Returns models designed for vector embeddings. Use this endpoint when selecting embedding models for retrieval, RAG, clustering, or similarity search. ## Examples ```bash cURL theme={"system"} curl https://api.redpill.ai/v1/embeddings/models \ -H "Authorization: Bearer " ``` ```python Python theme={"system"} import requests response = requests.get( "https://api.redpill.ai/v1/embeddings/models", headers={"Authorization": "Bearer "}, ) for model in response.json()["data"]: print(model["id"]) ``` ```typescript TypeScript theme={"system"} const response = await fetch("https://api.redpill.ai/v1/embeddings/models", { headers: { Authorization: "Bearer ", }, }); const models = await response.json(); for (const model of models.data) { console.log(model.id); } ``` ## Response ```json theme={"system"} { "object": "list", "data": [ { "id": "qwen/qwen3-embedding-8b", "name": "Qwen3 Embedding 8B", "created": 1704067200, "input_modalities": ["text"], "output_modalities": ["embeddings"], "context_length": 32768, "max_output_length": 4096, "pricing": { "prompt": "0.00000001", "completion": "0" }, "description": "Embedding model for semantic search and retrieval" } ] } ``` ## Response Fields | Field | Description | | ------------------- | ------------------------------------------ | | `id` | Model identifier for `POST /v1/embeddings` | | `name` | Human-readable model name | | `input_modalities` | Input types accepted by the model | | `output_modalities` | Output types produced by the model | | `context_length` | Maximum input context length | | `max_output_length` | Embedding dimensions or max output size | | `pricing.prompt` | Input price per token | | `description` | Model description and use case | # Embeddings Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/api-reference/embeddings Create vector embeddings with OpenAI-compatible embedding models. ## Endpoint ```bash theme={"system"} POST https://api.redpill.ai/v1/embeddings ``` Generate vector embeddings for retrieval, semantic search, clustering, and similarity workloads. ## Request Body Embedding model ID. Examples: `qwen/qwen3-embedding-8b`, `sentence-transformers/all-minilm-l6-v2`. Input text or list of inputs to embed. Embedding encoding format. Common values are `float` and `base64`. Requested output dimensions, when supported by the selected model. ## Examples ```bash cURL theme={"system"} curl https://api.redpill.ai/v1/embeddings \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "model": "qwen/qwen3-embedding-8b", "input": "Confidential AI keeps inference data private." }' ``` ```python Python theme={"system"} from openai import OpenAI client = OpenAI( api_key="", base_url="https://api.redpill.ai/v1", ) response = client.embeddings.create( model="qwen/qwen3-embedding-8b", input="Confidential AI keeps inference data private.", ) vector = response.data[0].embedding print(len(vector)) ``` ```typescript TypeScript theme={"system"} import OpenAI from "openai"; const client = new OpenAI({ apiKey: "", baseURL: "https://api.redpill.ai/v1", }); const response = await client.embeddings.create({ model: "qwen/qwen3-embedding-8b", input: "Confidential AI keeps inference data private.", }); console.log(response.data[0].embedding.length); ``` ## Response ```json theme={"system"} { "object": "list", "data": [ { "object": "embedding", "index": 0, "embedding": [0.0023, -0.0015, 0.0042] } ], "model": "qwen/qwen3-embedding-8b", "usage": { "prompt_tokens": 8, "total_tokens": 8 } } ``` ## Common Models | Model | Dimensions | Context | Notes | | ---------------------------------------- | ---------- | ------- | ---------------------------------- | | `qwen/qwen3-embedding-8b` | 4096 | 32K | Large confidential embedding model | | `sentence-transformers/all-minilm-l6-v2` | 384 | 512 | Low-cost compact embedding model | Use [List Embedding Models](/phala-cloud/confidential-ai/confidential-model/api-reference/embedding-models) for the live embedding catalog. # List Models Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/api-reference/models List available Confidential AI models, providers, modalities, context windows, and pricing metadata. ## Endpoints ```bash theme={"system"} GET https://api.redpill.ai/v1/models GET https://api.redpill.ai/v1/models/phala ``` Use the live model catalog before hardcoding model IDs. The catalog returns model IDs, context windows, pricing, providers, modalities, and TEE metadata when available. ## Examples ```bash All Models theme={"system"} curl https://api.redpill.ai/v1/models \ -H "Authorization: Bearer " ``` ```bash Phala Models theme={"system"} curl https://api.redpill.ai/v1/models/phala \ -H "Authorization: Bearer " ``` ```python Python theme={"system"} from openai import OpenAI client = OpenAI( api_key="", base_url="https://api.redpill.ai/v1", ) models = client.models.list() for model in models.data: print(model.id) ``` ## Response ```json theme={"system"} { "data": [ { "id": "phala/qwen3.5-27b", "name": "Qwen3.5 27B", "created": 1677652288, "description": "Qwen model running through Phala GPU TEE infrastructure", "context_length": 262144, "pricing": { "prompt": "0.00000030", "completion": "0.00000240" }, "providers": ["phala"], "metadata": { "tee": true, "appid": "..." }, "architecture": { "modality": "text->text", "input_modalities": ["text"], "output_modalities": ["text"] } } ] } ``` ## Model Object Fields | Field | Description | | -------------------------------- | --------------------------------------------------------------------------------- | | `id` | Model identifier for API calls | | `name` | Human-readable model name | | `description` | Model or provider description | | `context_length` | Maximum context window | | `pricing.prompt` | Input token price per token; multiply by 1,000,000 for per-million-token pricing | | `pricing.completion` | Output token price per token; multiply by 1,000,000 for per-million-token pricing | | `providers` | Infrastructure providers such as `phala`, `near-ai`, `tinfoil`, or `chutes` | | `metadata.tee` | Whether the model is marked as a TEE model | | `metadata.appid` | Present when the model supports the attestation flow | | `architecture.input_modalities` | Supported input types, such as `text` or `image` | | `architecture.output_modalities` | Supported output types, such as `text` or `embeddings` | ## Find Verifiable TEE Models Filter for models that expose TEE provider metadata: ```bash theme={"system"} curl https://api.redpill.ai/v1/models \ -H "Authorization: Bearer " | \ jq '.data[] | select(.metadata.tee == true or any(.providers[]?; test("phala|near-ai|tinfoil|chutes"))) | {id, providers, appid: .metadata.appid}' ``` For production verification, test [Attestation Report](/phala-cloud/confidential-ai/confidential-model/api-reference/attestation) with the exact model ID before relying on it in your application. # Request Signature Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/api-reference/signature Fetch a cryptographic signature for a Confidential AI response. ## Endpoint ```bash theme={"system"} GET https://api.redpill.ai/v1/signature/{request_id}?model={model}&signing_algo={algo} ``` Use this endpoint after a chat completion request. The signature proves a specific response was signed by a TEE key. Bind that key to fresh attestation evidence before treating the response as fully verified. ## Parameters The `id` returned by `POST /v1/chat/completions`. The model ID used for the original request. Signature algorithm. Common values include `ecdsa`, `ecdsa-p256`, and `rsa`; use the algorithm supported by the model response. ## Examples ```bash cURL theme={"system"} RESPONSE=$(curl -s https://api.redpill.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '{"model":"phala/qwen3.5-27b","messages":[{"role":"user","content":"hello"}]}') REQUEST_ID=$(echo "$RESPONSE" | jq -r '.id') curl "https://api.redpill.ai/v1/signature/$REQUEST_ID?model=phala/qwen3.5-27b" \ -H "Authorization: Bearer " ``` ```python Python theme={"system"} import requests chat_response = requests.post( "https://api.redpill.ai/v1/chat/completions", headers={ "Authorization": "Bearer ", "Content-Type": "application/json", }, json={ "model": "phala/qwen3.5-27b", "messages": [{"role": "user", "content": "hello"}], }, ) request_id = chat_response.json()["id"] signature_response = requests.get( f"https://api.redpill.ai/v1/signature/{request_id}", params={"model": "phala/qwen3.5-27b"}, headers={"Authorization": "Bearer "}, ) signature_data = signature_response.json() ``` ## Response ```json theme={"system"} { "text": "phala/qwen3.5-27b:116478638341bd2b...:3d0b2a2df73dc93a...", "signature": "0xee817b30e13ec3c320997ec37076a600e194dc64...", "signing_address": "0x56d070df1c6be444b007839ef9cf67cec7c12b8b", "signing_algo": "ecdsa" } ``` ## Response Fields | Field | Description | | ----------------- | ------------------------------------------------------------------------------------------------ | | `text` | Signed text. Format is either `request_hash:response_hash` or `model:request_hash:response_hash` | | `signature` | Signature over `text` | | `signing_address` | TEE signing address or public key | | `signing_algo` | Signature algorithm used | When `text` has three colon-separated parts, the first part is the model name used inside the signing path. It may differ from the alias you sent if the gateway rewrote the model ID internally. ## Bind to Attestation For production verification, use the returned `signing_address` to fetch fresh attestation evidence: ```bash theme={"system"} NONCE=$(openssl rand -hex 32) curl "https://api.redpill.ai/v1/attestation/report?model=phala/qwen3.5-27b&nonce=$NONCE&signing_address=$SIGNING_ADDRESS" \ -H "Authorization: Bearer " ``` The response is verified only when: 1. The request and response hashes in `text` match the bytes you sent and received. 2. The signature is valid for `text`. 3. The attestation report binds the same `signing_address` to genuine TEE evidence and your fresh nonce. # On-demand API Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/confidential-ai-api Get started quickly with pre-deployed models running in GPU TEE with OpenAI-compatible interface. Pay per request. ## Overview On-demand Confidential AI API provides a secure, OpenAI-compatible interface for running AI models in TEE on GPU hardware. Pay per request with no infrastructure management. This enables developers to integrate AI applications with hardware-level privacy protection, ensuring user data remain confidential during inference. Browse [available confidential AI models](https://phala.com/confidential-ai-models) for your application. For dedicated GPU resources with hourly pricing, see [Dedicated Models](/phala-cloud/confidential-ai/confidential-gpu/model-template). Both options use the same API with identical features - the only difference is billing and resource allocation. ## Prerequisites Before you begin, ensure you have enough funds to get the API key. You need at least \$5 in your account. Go to **Dashboard** and click **Deposit** to add funds. Navigate to **Dashboard** → **Confidential AI API** and click **Enable**. Then create your first API key and click the key to copy. GPU TEE API Generate Key Once you get the API Key, you can start making requests to the Confidential AI API. ## Make Your Secure Request Replace `` with your actual API key. The examples below use `phala/qwen3.5-27b`; use [List Models](/phala-cloud/confidential-ai/confidential-model/api-reference/models) to choose a model for your workload. ```python Python theme={"system"} # Install OpenAI SDK: `pip3 install openai` from openai import OpenAI client = OpenAI( api_key="", base_url="https://api.redpill.ai/v1", ) response = client.chat.completions.create( model="phala/qwen3.5-27b", messages=[ {"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "What is your model name?"}, ], ) print(response.choices[0].message.content) ``` ```typescript TypeScript theme={"system"} import OpenAI from 'openai'; const client = new OpenAI({ baseURL: 'https://api.redpill.ai/v1', apiKey: '', }); async function main() { const completion = await client.chat.completions.create({ model: 'phala/qwen3.5-27b', messages: [ { role: 'user', content: 'What is the meaning of life?', }, ], }); console.log(completion.choices[0].message); } main(); ``` ```bash CLI theme={"system"} curl -X 'POST' \ 'https://api.redpill.ai/v1/chat/completions' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer ' \ -d '{ "messages": [ { "content": "You are a helpful assistant.", "role": "system" }, { "content": "What is your model name?", "role": "user" } ], "model": "phala/qwen3.5-27b" }' ``` ### Available Models Confidential AI models are available through several GPU TEE providers. The live catalog is authoritative; query it before hardcoding model IDs: ```bash theme={"system"} curl https://api.redpill.ai/v1/models \ -H "Authorization: Bearer " ``` To list Phala-backed models only: ```bash theme={"system"} curl https://api.redpill.ai/v1/models/phala \ -H "Authorization: Bearer " ``` The following table reflects the current model families added in the RedPill model catalog update. Pricing and availability can change; use the API response for production routing. #### Phala Provider | Model ID | Context | Modality | Pricing (input/output per 1M tokens) | | ---------------------------------------- | ------- | ------------- | ------------------------------------ | | `phala/qwen3.5-27b` | 262K | Text | $0.30 / $2.40 | | `phala/qwen3-vl-30b-a3b-instruct` | 128K | Vision + Text | $0.20 / $0.70 | | `qwen/qwen3-embedding-8b` | 32K | Embeddings | $0.01 / $0 | | `phala/gemma-3-27b-it` | 53K | Vision + Text | $0.11 / $0.40 | | `phala/glm-4.7-flash` | 202K | Text | $0.10 / $0.43 | | `phala/gpt-oss-20b` | 131K | Text | $0.04 / $0.15 | | `phala/qwen-2.5-7b-instruct` | 32K | Text | $0.04 / $0.10 | | `phala/qwen2.5-vl-72b-instruct` | 128K | Vision + Text | $0.40 / $1.20 | | `phala/uncensored-24b` | 32K | Text | $0.20 / $0.90 | | `sentence-transformers/all-minilm-l6-v2` | 512 | Embeddings | $0.005 / $0 | `phala/qwen2.5-vl-72b-instruct` is a legacy alias that may route to `phala/qwen3-vl-30b-a3b-instruct`. Prefer the canonical ID returned by `/v1/models`. #### NearAI Provider | Model ID | Context | Modality | Pricing (input/output per 1M tokens) | | ---------------------------------- | ------- | -------- | ------------------------------------ | | `z-ai/glm-5` | 203K | Text | $1.20 / $3.50 | | `deepseek/deepseek-chat-v3.1` | 164K | Text | $1.05 / $3.10 | | `openai/gpt-oss-120b` | 131K | Text | $0.10 / $0.49 | | `qwen/qwen3-30b-a3b-instruct-2507` | 262K | Text | $0.15 / $0.55 | | `z-ai/glm-4.7` | 131K | Text | $0.85 / $3.30 | #### Chutes Provider | Model ID | Context | Modality | Pricing (input/output per 1M tokens) | | ------------------------ | ------- | ------------ | ------------------------------------ | | `z-ai/glm-5.1` | 203K | Text | $1.21 / $4.20 | | `moonshotai/kimi-k2.6` | 262K | Text + Image | $1.09 / $4.60 | | `qwen/qwen3.5-397b-a17b` | 262K | Text | $0.55 / $3.50 | | `qwen/qwen3-coder-next` | 262K | Text | $0.18 / $1.20 | | `minimax/minimax-m2.5` | 197K | Text | $0.20 / $1.38 | | `xiaomi/mimo-v2-flash` | 262K | Text | $0.10 / $0.30 | | `deepseek/deepseek-v3.2` | 164K | Text | $0.32 / $0.48 | | `moonshotai/kimi-k2.5` | 262K | Text + Image | $0.60 / $3.00 | #### Tinfoil Provider | Model ID | Context | Modality | Pricing (input/output per 1M tokens) | | ------------------------------------- | ------- | -------- | ------------------------------------ | | `qwen/qwen3-coder-480b-a35b-instruct` | 262K | Text | $2.00 / $2.00 | | `moonshotai/kimi-k2-thinking` | 262K | Text | $2.00 / $2.00 | | `deepseek/deepseek-r1-0528` | 163K | Text | $2.00 / $2.00 | | `meta-llama/llama-3.3-70b-instruct` | 131K | Text | $2.00 / $2.00 | TEE provider presence and attestation support are not identical for every provider and model. For production verification, test [Attestation Report](/phala-cloud/confidential-ai/confidential-model/api-reference/attestation) with the exact model ID you plan to use. ## Verify Your AI is Running Securely After you make a request, use [Request Signature](/phala-cloud/confidential-ai/confidential-model/api-reference/signature) to fetch the signature for that response. Then fetch a fresh [Attestation Report](/phala-cloud/confidential-ai/confidential-model/api-reference/attestation) with the returned `signing_address` to bind the response to TEE evidence. ## Next Steps Use the API reference and feature guides for the next step: * [Chat Completions](/phala-cloud/confidential-ai/confidential-model/api-reference/chat-completions) documents the core request and response shape. * [List Models](/phala-cloud/confidential-ai/confidential-model/api-reference/models) shows how to discover models programmatically. * [Embeddings](/phala-cloud/confidential-ai/confidential-model/api-reference/embeddings) covers embedding model calls. * [Tool Calling](/phala-cloud/confidential-ai/confidential-model/tool-calling) helps you call tools from your AI models. * [Images and Vision](/phala-cloud/confidential-ai/confidential-model/images-and-vision) helps you use image-capable models. * [Structured Output](/phala-cloud/confidential-ai/confidential-model/structured-output) helps you get JSON responses. * [Streaming](/phala-cloud/confidential-ai/confidential-model/streaming) helps you consume streaming responses. * [Playground](/phala-cloud/confidential-ai/confidential-model/playground) helps you test models in a private environment. # Images and Vision Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/images-and-vision Learn how to use images and vision models in Confidential AI. This feature works for both API Access and Dedicated Models. ## Images and Vision in Confidential AI Confidential AI supports various images and vision models, enabling you to leverage the power of visual data in a secure and confidential manner. ## Analyze images You can use vision models to analyze images, including image recognition, object detection, and image classification. Replace `` with your actual API key in the examples below. ```python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.redpill.ai/v1", api_key="", ) response = client.chat.completions.create( model="phala/qwen3-vl-30b-a3b-instruct", messages=[{ "role": "user", "content": [ { "type": "text", "text": "What is in this image?" }, { "type": "image_url", "image_url": { "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/6/67/Giant_Pandas_having_a_snack.jpg/2560px-Giant_Pandas_having_a_snack.jpg" } } ] }], ) print(response.choices[0].message.content) ``` ``` Here's what I see in the image: * **Giant Pandas:** There are four young giant pandas in the image. They are black and white and appear to be playful. * **Bamboo:** The scene is full of lush, green bamboo. The pandas are among the bamboo and seemingly eating it. * **Greenery:** Besides the bamboo, there is other green vegetation surrounding the pandas. * **Dirt mound:** The pandas are partially sitting on a raised earth mound. * **Outdoor Setting:** It appears to be a natural or zoo-like outdoor enclosure. The overall impression is of a cute and peaceful scene with baby pandas enjoying their habitat. ``` ### Supported Models for Image Analysis * `phala/gemma-3-27b-it` * `phala/qwen3-vl-30b-a3b-instruct` * `phala/qwen2.5-vl-72b-instruct` (legacy alias) # Playground Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/playground Learn how to use Open-WebUI to play with Confidential AI models This feature works for both API Access and Dedicated Models. ## How to setup playground environment If you want to interact with Confidential AI models with web UI instead of command line tools, we recommend using Open-WebUI. It is a web-based interface for Confidential AI models. It allows you to play with models, test prompts, and visualize the results in real-time. ## Deploy Open-WebUI You could deploy Open-WebUI on your local machine or Phala Cloud. It is recommended to deploy Open-WebUI on Phala Cloud with this [Open-WebUI one-click deploy template](https://cloud.phala.com/templates/open-webui). Or if you prefer try it locally, you could deploy it on your local machine with this [docker compose file](https://github.com/Phala-Network/phala-cloud/blob/main/templates/prebuilt/openwebui/docker-compose.yml). If you deploy Open-WebUI on Phala Cloud, once it is deployed, you could access it by opening the `Endpoint #1` in Network tab of CVM instance details. Open-WebUI Endpoint ## Configure Open-WebUI Open-WebUI need to be configured with API key and base URL to connect to Confidential AI API. 1. Click right top user icon, and click `Admin Panel` to open Settings page. 2. Click `Connection` navigation item to open Connection settings page. 3. In Connection settings page, click setting icon to fill in the `URL` with `https://api.redpill.ai/v1`, input Bearer token with your API key, and then click `Save` button. 4. Once the connection is saved, you could start using Open-WebUI to play with Confidential AI models. Open-WebUI Connection Settings In this private playground environment, you could play with Confidential AI models with highly secure and private data without any data leakage. # Streaming Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/streaming Learn how to use streaming in Confidential AI API This feature works for both API Access and Dedicated Models. ## Streaming in Confidential AI API Confidential AI API supports streaming, enabling you to receive responses in a streaming fashion. This is particularly useful for applications that require real-time data processing or integration with other systems. ### Example of Streaming Replace `` with your actual API key in the examples below. ```python Python theme={"system"} from openai import OpenAI client = OpenAI( api_key="", base_url="https://api.redpill.ai/v1", ) stream = client.chat.completions.create( model="phala/qwen3.5-27b", messages=[ { "role": "user", "content": "say `Hello` 2 times fast, no other output", }, ], stream=True, ) for chunk in stream: content = chunk.choices[0].delta.content if content: print(content, end="") ``` ```typescript TypeScript theme={"system"} import OpenAI from "openai"; const client = new OpenAI({ apiKey: "", baseURL: "https://api.redpill.ai/v1", }); const stream = await client.chat.completions.create({ model: "phala/qwen3.5-27b", messages: [ { role: "user", content: "say `Hello` 2 times fast, no other output" }, ], stream: true, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content || ""); } ``` ``` HelloHello ``` ### Supported Models All models support streaming. # Structured Output Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/structured-output Learn how to use structured output in Confidential AI This feature works for both API Access and Dedicated Models. ## Structured Output in Confidential AI Confidential AI supports structured output, enabling you to receive responses in specific formats such as JSON structures. This is particularly useful for applications that require data to be processed or integrated into other systems. ### Example of Structured Output Replace `` with your actual API key in the examples below. ```python theme={"system"} import requests response = requests.post( "https://api.redpill.ai/v1/chat/completions", headers={ "Authorization": "Bearer ", "Content-Type": "application/json", }, json={ "model": "phala/gpt-oss-20b", "messages": [ {"role": "user", "content": "What is the weather like in Los Angeles?"}, ], "response_format": { "type": "json_schema", "json_schema": { "name": "weather", "strict": True, "schema": { "type": "object", "properties": { "location": { "type": "string", "description": "City or location name", }, "temperature": { "type": "number", "description": "Temperature in Celsius", }, "conditions": { "type": "string", "description": "Weather conditions description", }, }, "required": ["location", "temperature", "conditions"], "additionalProperties": False, }, }, }, }, ) data = response.json() info = data["choices"][0]["message"]["content"] print(info) ``` ```json theme={"system"} { "location":"Los Angeles", "temperature":75, "conditions":"Sunny with a few clouds" } ``` ### Supported Models for Structured Output Confidential AI supports structured output for the following models: * `phala/gemma-3-27b-it` * `phala/gpt-oss-20b` * `phala/gpt-oss-120b` * `phala/qwen3.5-27b` * `phala/qwen3-vl-30b-a3b-instruct` # Tool Calling Source: https://docs.phala.com/phala-cloud/confidential-ai/confidential-model/tool-calling Quick guide for how to tool calling (or function calling) of Confidential AI API. This feature works for both API Access and Dedicated Models. ## Tool Calling in Confidential AI Tool calling, also known as function calling, enables AI models to interact with external systems by invoking predefined functions or APIs. This allows the model to access up-to-date information, perform specific actions, or integrate with other services, extending its capabilities beyond static knowledge. Deploy [custom fine-tuned models](https://phala.com/solutions/fine-tuned-models) with proprietary tool definitions for specialized use cases. Here we provide a simple example of tool calling. ## Define the Tool Use following request to create a function named `get_weather`. ```bash theme={"system"} curl -s -X POST 'https://api.redpill.ai/v1/chat/completions' \ -H 'Authorization: Bearer sk-xx' \ -H 'Content-Type: application/json' \ -d '{ "messages": [ { "content": "What is the weather like in New York?", "role": "user" } ], "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a specified city", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "metric": {"type": "string", "enum": ["celsius", "fahrenheit"]} }, "required": ["city"] } } } ], "tool_choice": "auto", "stream": false, "model": "phala/gpt-oss-20b" }' ``` ```json theme={"system"} { "id": "chatcmpl-28f745c2b7ee44f2ba36a8b4b409c74a", "object": "chat.completion", "created": 1754381277, "model": "phala/gpt-oss-20b", "choices": [ { "index": 0, "message": { "role": "assistant", "content": null, "refusal": null, "annotations": null, "audio": null, "function_call": null, "tool_calls": [ { "id": "chatcmpl-tool-471f2c795ab64a27ab671a9e302c94db", "type": "function", "function": { "name": "get_weather", "arguments": "{\"city\": \"New York\", \"metric\": \"celsius\"}" } } ], "reasoning_content": null }, "logprobs": null, "finish_reason": "tool_calls", "stop_reason": null } ], "service_tier": null, "system_fingerprint": null, "usage": { "prompt_tokens": 319, "total_tokens": 353, "completion_tokens": 34, "prompt_tokens_details": null }, "prompt_logprobs": null, "kv_transfer_params": null } ``` Extract the tool id from the response, and in this example it would be `chatcmpl-tool-471f2c795ab64a27ab671a9e302c94db`. ## Invoke the Tool Use the tool id extracted from previouse request to call the tool. ```bash theme={"system"} curl -s -X POST 'https://api.redpill.ai/v1/chat/completions' \ -H 'Authorization: Bearer sk-xx' \ -H 'Content-Type: application/json' \ -d '{ "messages": [ {"role": "user", "content": "What is the weather like in New York?"}, { "role": "assistant", "content": null, "tool_calls": [ { "id": "chatcmpl-tool-471f2c795ab64a27ab671a9e302c94db", "type": "function", "function": { "name": "get_weather", "arguments": "{\"city\": \"New York\", \"metric\": \"celsius\"}" } } ] }, { "role": "tool", "content": "{\"city\": \"New York\", \"temperature\": 15, \"description\": \"clear sky\", \"metric\": \"celsius\"}", "tool_call_id": "chatcmpl-tool-471f2c795ab64a27ab671a9e302c94db" } ], "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a specified city", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "metric": {"type": "string", "enum": ["celsius", "fahrenheit"]} }, "required": ["city"] } } } ], "tool_choice": "auto", "stream": false, "model": "phala/gpt-oss-20b" }' ``` ```json theme={"system"} { "id": "chatcmpl-a46eff3d335c42c39bbe4ea69fc97462", "object": "chat.completion", "created": 1754381325, "model": "phala/gpt-oss-20b", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The current weather in New York is clear sky with a temperature of 15°C.", "refusal": null, "annotations": null, "audio": null, "function_call": null, "tool_calls": [], "reasoning_content": null }, "logprobs": null, "finish_reason": "stop", "stop_reason": null } ], "service_tier": null, "system_fingerprint": null, "usage": { "prompt_tokens": 394, "total_tokens": 413, "completion_tokens": 19, "prompt_tokens_details": null }, "prompt_logprobs": null, "kv_transfer_params": null } ``` ## Supported Models * `phala/gpt-oss-20b` * `phala/qwen3.5-27b` * `qwen/qwen3-coder-next` * `qwen/qwen3-coder-480b-a35b-instruct` # FAQs Source: https://docs.phala.com/phala-cloud/confidential-ai/faqs Frequently asked questions about Confidential AI. The Private ML SDK leverages the TEE features provided by Intel and NVIDIA GPUs to support running any GPU workload as a Docker container in a GPU TEE. In the Private ML SDK, dstack is a framework that can run Docker containers, while vllm-proxy is a specific container within it. The vllm-proxy container operates as a server that forwards requests to a vllm container (hosting a large language model) and generates quotes to attach to responses. This separation allows vllm to remain unmodified and compatible with its latest versions. No, CUDA and NVIDIA drivers are not required on the host for the Private ML SDK with GPU support. The host should not have NVIDIA drivers installed to avoid conflicts with TDX and VFIO passthrough. Drivers (e.g., 550.54.15) and CUDA (e.g., 12.4) are loaded inside the guest CVM via Docker containers (e.g., kvin/cuda-notebook). The vllm and vllm-proxy services also run entirely within the TEE guest, not on the host. Yes, CUDA works in the GPU TEE environment. By default, you don't need to install it on the host environment when using private-ml-sdk to create a CVM. If you want to use CUDA in your CVM, you can install it manually inside the CVM. You can refer to our benchmark analysis with Succinct, where the SP1 VM also involves CUDA. Yes, you can build your application into a docker image and prepare a docker-compose file. The SDK will create a TDX-based virtual machine to launch your application. However, unlike Google Cloud's Confidential Space, which uses virtual machines, dstack and private-ml-sdk require a bare-metal environment. dstack is launched on a bare-metal environment. However, for CVMs created by dstack, a hypervisor (qemu) is used to launch the VM with TDX enabled. # Deploy AI Models in GPU TEE Source: https://docs.phala.com/phala-cloud/confidential-ai/gpu-tee-deployment-guide End-to-end guide for deploying custom LLMs on GPU TEE with vLLM and accessing them via OpenAI-compatible API Deploy a custom LLM on Phala Cloud GPU TEE and serve it through an OpenAI-compatible API. This guide walks you through the full workflow from choosing hardware to making your first inference request with verified TEE attestation. For pre-deployed models that don't require infrastructure setup, use [On-demand API](/phala-cloud/confidential-ai/confidential-model/confidential-ai-api) or [Dedicated Models](/phala-cloud/confidential-ai/confidential-gpu/model-template) instead. ## Prerequisites * Phala Cloud account with sufficient credits for GPU instances * A model you want to deploy (Hugging Face model ID or custom weights) * Basic familiarity with Docker Compose and LLM serving ## Choose your GPU hardware Pick a GPU based on your model's VRAM requirements. Larger models need more VRAM, and you can scale up to 8 GPUs per instance for models that support tensor parallelism. | GPU type | VRAM per GPU | Best for | | -------- | ------------ | ---------------------------------- | | H200 | 141 GB | Most 70B+ models, long context | | B200 | 180 GB | Largest models, maximum throughput | A 7B model fits comfortably on a single GPU. A 70B model typically needs 2-4 GPUs depending on quantization. Check your model's documentation for exact VRAM requirements. GPU instances incur hourly charges starting from provisioning. Check current pricing on the [GPU TEE](https://cloud.phala.com) dashboard before launching. ## Deploy your instance Sign in to [cloud.phala.com](https://cloud.phala.com), click **GPU TEE** in the navigation bar, then click **Start Building**. Select your GPU type and count. In the deployment configuration, select **vLLM** as your template. This gives you a production-ready inference server with OpenAI-compatible API endpoints out of the box. If you prefer full control, choose **Custom Configuration** and provide your own Docker Compose file. See the [custom vLLM compose example](#custom-vllm-docker-compose) below. Set the model you want to serve. For Hugging Face models, use the model ID directly (e.g., `meta-llama/Llama-3.1-70B-Instruct`). Add any required environment variables like `HUGGING_FACE_HUB_TOKEN` as encrypted secrets if your model needs authentication. Review the pricing summary and click **Launch Instance**. GPU provisioning takes approximately 1 day. Monitor the status in your dashboard as it progresses through **Preparing**, **Starting**, and **Running**. ## Custom vLLM Docker Compose If you chose Custom Configuration, here's a Docker Compose file for serving a model with vLLM. This example serves Qwen 2.5 7B on a single GPU. ```yaml theme={"system"} services: vllm: image: vllm/vllm-openai:v0.8.5@sha256:abc123... # Pin to digest for attestation ports: - "8000:8000" environment: - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN} command: > --model qwen/Qwen2.5-7B-Instruct --tensor-parallel-size 1 --max-model-len 32768 --trust-remote-code deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] ``` For multi-GPU serving, increase `--tensor-parallel-size` to match your GPU count. For example, with 4 GPUs: ```yaml theme={"system"} command: > --model meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4 --max-model-len 65536 deploy: resources: reservations: devices: - driver: nvidia count: 4 capabilities: [gpu] ``` Pin your Docker images by SHA256 digest (not tags) for attestation verification. Tags like `vllm/vllm-openai:latest` are mutable and break the chain of trust. See [Verify Your Application](/phala-cloud/attestation/verify-your-application) for details. ## Access the OpenAI-compatible API Once your instance is running, vLLM exposes an OpenAI-compatible API. Find your instance's URL in the dashboard under your GPU TEE instance details. ```bash curl theme={"system"} curl https:///v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen/Qwen2.5-7B-Instruct", "messages": [ {"role": "user", "content": "What is confidential computing?"} ], "max_tokens": 256 }' ``` ```python Python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https:///v1", api_key="not-needed" # No auth by default; add if configured ) response = client.chat.completions.create( model="qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "user", "content": "What is confidential computing?"} ], max_tokens=256 ) print(response.choices[0].message.content) ``` ```typescript TypeScript theme={"system"} import OpenAI from 'openai'; const client = new OpenAI({ baseURL: 'https:///v1', apiKey: 'not-needed', }); const response = await client.chat.completions.create({ model: 'qwen/Qwen2.5-7B-Instruct', messages: [ { role: 'user', content: 'What is confidential computing?' } ], maxTokens: 256, }); console.log(response.choices[0].message.content); ``` You can also check the available models and server health: ```bash theme={"system"} # List loaded models curl https:///v1/models # Health check curl https:///health ``` ## Verify TEE attestation Your GPU TEE instance runs on genuine Intel TDX + NVIDIA confidential compute hardware. Verify this to confirm your model is actually running inside a trusted execution environment. Open JupyterLab (if using the Jupyter template) or SSH into your instance and run: ```bash theme={"system"} nvidia-smi conf-compute -q ``` Confirm `CC State: ON` and `CPU CC Capabilities: INTEL TDX` in the output. Install NVIDIA's attestation tools and run the verifier: ```bash theme={"system"} pip install nv-local-gpu-verifier nv_attestation_sdk python -m verifier.cc_admin ``` Successful output confirms your GPUs are genuine NVIDIA hardware with confidential compute enabled. Fetch and verify the CPU attestation through Phala's verification API: ```bash theme={"system"} # Get the attestation quote from your instance curl https:///attestation -o attestation.json # Verify through Phala's API curl -X POST https://cloud-api.phala.com/api/v1/attestations/verify \ -H "Content-Type: application/json" \ -d @attestation.json ``` You can also paste the quote into the [TEE Attestation Explorer](https://proof.t16z.com/) for a visual breakdown. For programmatic attestation verification in your application, see [Verify Attestation](/phala-cloud/confidential-ai/verify/verify-attestation). ## Add a custom domain Expose your model API on your own domain by adding dstack-ingress to your Docker Compose. This handles DNS and TLS certificates automatically. ```yaml theme={"system"} services: dstack-ingress: image: dstacktee/dstack-ingress:20250924@sha256:40429d78060ef3066b5f93676bf3ba7c2e9ac47d4648440febfdda558aed4b32 ports: - "443:443" environment: - DOMAIN=ai.mycompany.com - TARGET_ENDPOINT=http://vllm:8000 - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN} - GATEWAY_DOMAIN=_.${DSTACK_GATEWAY_DOMAIN} - CERTBOT_EMAIL=${CERTBOT_EMAIL} - SET_CAA=true volumes: - /var/run/dstack.sock:/var/run/dstack.sock - cert-data:/etc/letsencrypt vllm: # ... your vLLM config from above volumes: cert-data: ``` See [Set Up a Custom Domain](/phala-cloud/networking/setup-custom-domain) for the full guide including DNS provider configuration and troubleshooting. ## Next steps Programmatically verify your GPU TEE hardware and software stack Confirm inference responses are authentic using cryptographic signatures See how TEE mode performs compared to native GPU execution Expose services, set up gRPC, and configure TCP endpoints # Overview Source: https://docs.phala.com/phala-cloud/confidential-ai/overview This overview introduces Confidential AI, running AI models with hardware-level privacy in GPU TEEs. Explain how to use Confidential AI to inference LLMs, deploy custom models, and run AI workloads with verifiable attestation and near-native performance. Confidential AI overview - run AI models with hardware-level privacy in GPU TEEs ## Why Confidential AI? Traditional cloud AI deployments expose your models and data to the cloud provider. Confidential AI runs inference, training, and fine-tuning inside GPU TEEs — the cloud provider cannot access your models or data. You get cryptographic proof that execution happened in a trusted environment. Phala Cloud offers two ways to run confidential AI workloads: pre-deployed **Models** (API or Dedicated) for quick integration, or **GPU TEE** for custom infrastructure. See [available models](https://phala.com/confidential-ai-models) for supported models, use cases, and deployment options. **Pre-deployed models, pay per request** Best for quick integration. 5 minute setup with OpenAI-compatible API and no infrastructure management. **Same models, dedicated performance** Best for high-volume workloads. Same API as API Access, but with hourly billing and dedicated GPU resources. **Custom infrastructure, full control** Best for custom models. Rent dedicated GPU TEE servers for training, fine-tuning, or any custom workload. ## Quick Tour of Confidential AI ### Models: API and Dedicated **API Access** provides pre-deployed LLMs with OpenAI-compatible APIs for quick integration. Pay per request with no infrastructure to manage. Start with [API Access](/phala-cloud/confidential-ai/confidential-model/confidential-ai-api). For advanced API features, explore [Tool Calling](/phala-cloud/confidential-ai/confidential-model/tool-calling) to enable LLMs to interact with external tools and APIs securely within TEE. **Dedicated Models** give you the same pre-deployed models but with dedicated GPU resources and hourly pricing. Choose this for predictable performance or high-volume workloads. See [Dedicated Models](/phala-cloud/confidential-ai/confidential-gpu/model-template). ### GPU TEE: Custom Infrastructure For complete infrastructure control beyond pre-deployed models, use [GPU TEE](/phala-cloud/confidential-ai/confidential-gpu/deploy-and-verify) to rent dedicated GPU servers. Run any workload including custom models for inference, training, or fine-tuning. Configure GPU, CPU, RAM, and storage to match your exact needs. ### Verify Attestation and Signature To ensure your workloads run securely in TEE, you can [Verify Attestation](./verify-attestation) to check the TEE hardware, operating system, source code, and distributed root-of-trust attestations. Then you can [Verify Signature](./verify-signature) to confirm the integrity of your Confidential AI API requests and responses. ### Benchmark Our [benchmark](/phala-cloud/confidential-ai/benchmark) shows GPU TEE mode achieves 99% of native performance on H100/H200 GPUs. ### FAQs Check [FAQs](/phala-cloud/confidential-ai/faqs) for frequently asked questions about Confidential AI. ## What makes Phala Cloud Confidential AI Different? * **Drop-in compatibility**: OpenAI-compatible API with popular models (DeepSeek, Llama, GPT-OSS, Qwen) ready for immediate use * **Verifiable security**: Hardware-enforced privacy with cryptographic attestation proving execution in genuine TEE environments * **Flexible deployment**: Choose from API access (pay per request), dedicated models (hourly dedicated GPU), or GPU TEE (full infrastructure control) ## Open Source Foundation Our underlying technology is open source. Check out the [dstack](https://github.com/Dstack-TEE/dstack) repository to see how LLMs run securely in GPU TEEs. # Overview Source: https://docs.phala.com/phala-cloud/confidential-ai/verify/overview Learn how to verify your AI workloads run in genuine TEE hardware with attestations and get integrity proof that your requests and responses are untampered. ## Why verification matters Verification ensures your AI workloads run securely in genuine TEE hardware. You get cryptographic proof that: * **Hardware is authentic** - GPUs and CPUs are from trusted vendors and running in secure TEE mode * **Software is untampered** - The entire software stack from OS to application code is verified * **Responses are trustworthy** - Every AI response comes with proof it ran in a secure TEE The verification process follows these steps: Verify the physical hardware stack, including CPU and GPU, is genuine and running in a secure TEE. Confirm the signing key is cryptographically bound to the hardware. Verify that the software stack, including operating system, Docker compose configuration, container build provenance, network end-to-end encryption, and distributed root-of-trust, is secure and trustworthy. Verify the integrity of request and response content through cryptographic signatures. Here's how the verification workflow looks: ```mermaid theme={"system"} %%{init: {'theme': 'base', 'themeVariables': { 'fontSize': '25px', 'fontFamily': 'arial', 'lineWidth': '3px' }}}%% flowchart TD subgraph Attestation [Verify Attestation] direction TB A0[Generate Nonce] ==> A1[Get Attestation] A1 ==> A2[Verify TEE Hardware Stack] A2 ==> A3[Verify Report Data Binding] A3 ==> A4[Verify TEE Software Stack] end subgraph Signature [Verify Integrity] S1[Get Signature] ==> S2[Verify Request/Response Hashes] S2 ==> S3[Recover Signing Address] S3 ==> S4[Verify Attestation for Address] end Request[Secure Request] ==> Attestation Request ==> Signature ``` ### How the trust chain works These verification steps create a complete security chain: * **Fresh Nonce** prevents replay attacks - ensures attestations are generated fresh for your request * **TEE Hardware Stack** establishes the physical foundation - genuine NVIDIA GPUs and Intel CPUs with TEE enabled * **Report Data Binding** proves the signing key belongs to the TEE - signing address and nonce are cryptographically embedded in hardware-signed attestation * **TEE Software Stack** extends trust to the software layer - verified OS, exact Docker compose configuration, and container build provenance * **Integrity Proof** ensures data authenticity - cryptographic signatures prove responses came from the verified TEE Together, they guarantee your AI workloads run securely across the entire system with end-to-end cryptographic proof. Click the cards below to learn more about details. Learn how to verify hardware authenticity, report data binding, operating system integrity, Docker compose configuration, container build provenance, network end-to-end encryption, and distributed root-of-trust Learn how to verify cryptographic signatures to ensure request and response integrity # Attestation Source: https://docs.phala.com/phala-cloud/confidential-ai/verify/verify-attestation Learn how to verify cryptographic proof your AI workloads run in genuine TEE hardware with verified software ## Why attestation matters When you verify attestation, you prove your AI runs on real TEE hardware with authentic software. This gives you cryptographic proof that your hardware comes from NVIDIA or Intel, not some counterfeit supplier. You also confirm the software hasn't been tampered with and your workloads stay protected. ## Verify TEE hardware stack Let's start by verifying your hardware is genuine. You need to check both GPUs and CPUs because TEE protection requires both to work together. If either one fails verification, your entire security model breaks down. First, you'll generate a fresh nonce to ensure attestation freshness. Then you'll verify NVIDIA GPUs are real and running in TEE mode. Next you'll check Intel CPUs have TEE protections enabled and verify the report data binds the signing key and nonce. Finally, you'll confirm the cryptographic features actually work. Here's a [Python example](https://github.com/Phala-Network/private-ml-sdk/blob/main/vllm-proxy/verifiers/attestation_verifier.py) that walks through the entire verification process. ### Generate fresh nonce Before fetching the attestation, generate a random nonce. This nonce gets embedded in the TEE's cryptographic proof, ensuring the attestation was generated fresh for your request and not replayed from an old attestation. ```python theme={"system"} import secrets # Generate a random 32-byte nonce (64 hex characters) request_nonce = secrets.token_hex(32) ``` Without the nonce, you'd have no way to prove the attestation is fresh. An attacker could replay an old valid attestation from compromised hardware. ### Get the attestation report Now fetch the attestation report with your nonce. This report contains all the cryptographic proofs you need. ```python theme={"system"} import requests # Fetch attestation report with nonce response = requests.get( f"https://api.redpill.ai/v1/attestation/report?model={model}&nonce={request_nonce}", headers={"Authorization": f"Bearer {api_key}"} ) report = response.json() # Response shape depends on the provider backing the model. ``` The report gives you Intel TDX evidence, optional NVIDIA GPU evidence, signing key information, and software measurement data. The exact shape depends on the provider: * **Phala / NearAI two-layer format**: `gateway_attestation` plus `model_attestations`. * **Chutes format**: `attestation_type: "chutes"` plus `all_attestations`. * **Flat format**: older Phala-native responses expose fields such as `intel_quote`, `nvidia_payload`, and `signing_address` at the top level. Use [Attestation Report](/phala-cloud/confidential-ai/confidential-model/api-reference/attestation) for the endpoint schema. ### Select the attestation to verify For Phala and NearAI two-layer responses, verify both the gateway attestation and the model attestation when both are present. The gateway protects routing and request handling; the model attestation protects the inference runtime. ```python theme={"system"} def get_attestations(report): attestations = [] if "gateway_attestation" in report: attestations.append(("gateway", report["gateway_attestation"])) for item in report.get("model_attestations", []): attestations.append(("model", item)) for item in report.get("all_attestations", []): attestations.append(("model", item)) if "intel_quote" in report: attestations.append(("model", report)) return attestations attestations = get_attestations(report) assert attestations, "No attestation evidence found" ``` ### Verify NVIDIA GPU attestation Now verify NVIDIA GPUs when GPU evidence is present. Phala-style responses expose `nvidia_payload`; Chutes-style responses expose `gpu_evidence`. ```python theme={"system"} import json import base64 for name, attestation in attestations: if "nvidia_payload" not in attestation: continue # Parse and verify GPU payload nonce gpu_payload = json.loads(attestation["nvidia_payload"]) assert gpu_payload["nonce"].lower() == request_nonce.lower() # Send to NVIDIA's Remote Attestation Service response = requests.post( "https://nras.attestation.nvidia.com/v3/attest/gpu", json=gpu_payload ) result = response.json() # Decode the JWT verdict jwt_token = result[0][1] payload_b64 = jwt_token.split(".")[1] padded = payload_b64 + "=" * ((4 - len(payload_b64) % 4) % 4) verdict_data = json.loads(base64.urlsafe_b64decode(padded)) assert verdict_data["x-nvidia-overall-att-result"] == True ``` The GPU payload must use the same nonce you generated. NVIDIA returns a JWT with `x-nvidia-overall-att-result: True` for verified authentic hardware. ### Verify Intel TDX CPU attestation For Intel CPUs, you'll verify the TDX quote using Phala's verification service. This service decodes and validates Intel's cryptographic proof. ```python theme={"system"} import base64 import re def quote_to_hex(quote): value = quote.removeprefix("0x") if re.fullmatch(r"[0-9a-fA-F]+", value): return value return base64.b64decode(value).hex() for name, attestation in attestations: if "intel_quote" not in attestation: continue response = requests.post( "https://cloud-api.phala.com/api/v1/attestations/verify", json={"hex": quote_to_hex(attestation["intel_quote"])} ) intel_result = response.json() assert intel_result["quote"]["verified"] == True ``` This confirms the CPU is genuine Intel hardware running in TDX mode. The `intel_result` contains the decoded quote data we'll use next, including `reportdata` and `mrconfig` fields. For manual verification, you can paste the `intel_quote` value into the [TEE Attestation Explorer](https://proof.t16z.com/) to see decoded details about TDX version and security features. You can also verify the quote locally using our open source Intel DCAP verifier [dcap-qvl](https://github.com/Phala-Network/dcap-qvl). ### Verify report data binding Now verify that the TDX report data cryptographically binds the signing key and your nonce to the hardware. The first 64 bytes of TDX `reportdata` contain: * Bytes 0-31: Signing address (ECDSA or Ed25519 public key) * Bytes 32-63: Your request nonce ```python theme={"system"} # Extract report data from verified quote report_data_hex = intel_result["quote"]["body"]["reportdata"] report_data = bytes.fromhex(report_data_hex.removeprefix("0x")) # Parse signing address based on algorithm signing_address = attestation["signing_address"] signing_algo = attestation.get("signing_algo", "ecdsa") if signing_algo == "ecdsa": # ECDSA: 20-byte Ethereum address signing_address_bytes = bytes.fromhex(signing_address.removeprefix("0x")) else: # Ed25519: 32-byte public key signing_address_bytes = bytes.fromhex(signing_address) # Verify report data contains signing address and nonce embedded_address = report_data[:32] embedded_nonce = report_data[32:64] assert embedded_address == signing_address_bytes.ljust(32, b"\x00") assert embedded_nonce.hex() == request_nonce ``` This verification proves that: 1. The signing key was generated inside the TEE (it's embedded in hardware-attested report data) 2. The attestation is fresh (it contains your unique nonce) 3. The signing address you'll use for signature verification actually belongs to this TEE instance ## Verify TEE software stack Your hardware checks out. Now you need to verify the software running on that hardware is exactly what you expect. The software verification detects supply chain attacks where someone modifies the OS, injects malicious code, or breaks the chain of trust between hardware and application. You'll verify each layer - OS, application code, and cryptographic keys - to ensure the entire stack is authentic. ### 1. Verify operating system integrity First, check the OS hasn't been tampered with. The TEE measures every byte of the operating system when it boots, creating a cryptographic fingerprint. You'll compare this fingerprint against known good values. Follow [this verification process](https://github.com/Phala-Network/dstack-verifier/blob/95689c41/src/verification/osVerification.ts#L13-L27) to measure the OS image and compare it with TCB (Trusted Computing Base) values. The TCB values come from the [dstack-os reproducible build result](https://github.com/Dstack-TEE/meta-dstack?tab=readme-ov-file#reproducible-build-the-guest-image), and represent a clean, unmodified system. If even one byte changes, the fingerprint won't match. ### 2. Verify Docker compose manifest Next, verify your application code hasn't been modified. The TEE measures the entire Docker Compose configuration and embeds the hash in the TDX quote's `mr_config` field. ```python theme={"system"} from hashlib import sha256 # Extract compose manifest from attestation tcb_info = attestation["info"]["tcb_info"] if isinstance(tcb_info, str): tcb_info = json.loads(tcb_info) app_compose = tcb_info["app_compose"] compose_hash = sha256(app_compose.encode()).hexdigest() # Verify mr_config matches compose hash mr_config = intel_result["quote"]["body"]["mrconfig"] expected_mr_config = "0x01" + compose_hash assert mr_config.lower().startswith(expected_mr_config.lower()) # View the Docker Compose content docker_compose = json.loads(app_compose)["docker_compose_file"] print(docker_compose) ``` This verification proves the exact Docker Compose configuration running inside the TEE. The `mr_config` measurement is part of the TDX quote that Intel's hardware signed, so you know this configuration hasn't been modified after TEE boot. ### 3. Verify build provenance Finally, verify the container images in your Docker Compose were built from expected source repositories. The verification extracts all container image digests and checks their Sigstore provenance. ```python theme={"system"} import re # Extract all @sha256:xxx image digests from compose digests = set(re.findall(r'@sha256:([0-9a-f]{64})', docker_compose)) # Check Sigstore provenance for each image for digest in digests: sigstore_url = f"https://search.sigstore.dev/?hash=sha256:{digest}" response = requests.head(sigstore_url, timeout=10) if response.status_code < 400: print(f"✓ {sigstore_url}") else: print(f"✗ {sigstore_url} (HTTP {response.status_code})") ``` When Sigstore links return HTTP 200, you can visit them to: * Verify the container was built from the expected GitHub repository * Review the GitHub Actions workflow that built the image * Audit the build provenance and supply chain metadata This proves the containers running in your TEE were built from known source code through verified build processes. ### 4. Verify distributed root-of-trust The KMS ties everything together - hardware, OS, and application. It's the distributed system that manages all cryptographic operations and decides which applications can boot. The KMS generates two root keys when it first boots: one for signing TLS certificates and another for deriving application-specific keys. These keys never leave the TEE. The [KMS Verifier code](https://github.com/Phala-Network/dstack-verifier/blob/95689c415cad165033c700979cd26225c56c8d1c/src/verifiers/kmsVerifier.ts#L27) shows how to verify the KMS itself runs in a verified TEE and that its keys haven't been tampered with. ### 5. Verify network end-to-end encryption Finally, verify that all network traffic stays encrypted and under TEE control. The TEE generates its own TLS keys internally - they never exist outside the secure enclave. This means even if someone compromises the host system, they can't intercept your traffic. For more details, your can check the docs of [Domain Attestation](/phala-cloud/networking/domain-attestation). ## Complete verification example For a complete Python script that performs all the verifications above, see the [attestation verifier](https://github.com/Phala-Network/private-ml-sdk/blob/main/vllm-proxy/verifiers/attestation_verifier.py). This script handles: * Generating fresh nonce for attestation * Fetching the attestation report from Confidential AI API * Verifying NVIDIA GPUs through their attestation service * Verifying Intel TDX quote * Validating report data binds signing address and nonce * Verifying Docker compose manifest matches mr\_config * Checking Sigstore provenance for all container images ## Next step Hardware and software stack verified! Now [verify integrity proof](./verify-signature) to ensure your AI outputs are authentic. # Integrity Proof Source: https://docs.phala.com/phala-cloud/confidential-ai/verify/verify-signature Learn how to verify the integrity of your Confidential AI API requests. ## Why integrity proof matters Every Confidential AI response comes with a cryptographic signature. This signature proves the AI output came from verified TEE hardware and nobody tampered with it along the way. The signature cryptographically binds the request and response to the TEE's signing key. Any modification to the AI's response or injected fake data will break the signature verification. Without this check, you're trusting the transport layer. With it, you have end-to-end cryptographic proof from the TEE itself. ## Get the signature After you make an AI request, you'll get a `chat_id` in the response. Use it to fetch the cryptographic signature: ```python theme={"system"} import requests # After getting AI response with chat_id chat_id = ai_response["id"] model = "phala/qwen3.5-27b" # or your model # Fetch the signature sig_response = requests.get( f"https://api.redpill.ai/v1/signature/{chat_id}?model={model}", headers={"Authorization": f"Bearer {api_key}"} ) signature_data = sig_response.json() # signature_data contains: # - text: "request_hash:response_hash" or "model:request_hash:response_hash" # - signature: The ECDSA or Ed25519 signature # - signing_address: The address that signed this response # - signing_algo: The signature algorithm ``` The response gives you everything needed for verification. The `text` field contains hashes of your request and the AI's response. Some responses include the model name first, so the format is either `request_hash:response_hash` or `model:request_hash:response_hash`. The `signature` is the cryptographic proof from the TEE. The `signing_address` identifies which TEE instance signed this response. ## Verify request and response hashes Confirm the hashes in the `text` field match your actual request and response. Hashes are byte-sensitive, so production verifiers should hash the exact serialized request body and response body sent over the wire. ```python theme={"system"} from hashlib import sha256 def sha256_text(text: str): """Calculate SHA256 hash of text""" return sha256(text.encode()).hexdigest() # Calculate hashes of your original request and response request_body_json = '{"model": "...", "messages": [...]}' # Your request JSON response_body = '{"id": "...", "choices": [...], ...}' # Full response JSON request_hash = sha256_text(request_body_json) response_hash = sha256_text(response_body) # Parse the signed hashes. # Format can be either request_hash:response_hash or model:request_hash:response_hash. parts = signature_data["text"].split(":") if len(parts) == 3: signed_model, request_hash_server, response_hash_server = parts else: request_hash_server, response_hash_server = parts # Verify they match assert request_hash == request_hash_server assert response_hash == response_hash_server print("Request and response hashes verified") ``` This ensures the data wasn't modified or swapped after signing. The signature covers these exact hashes. ## Verify signature Now verify that the signature actually came from a TEE. You'll recover the signing address from the signature, then fetch and verify the attestation for that specific address. ### Recover signing address from signature For ECDSA signatures (most common), recover the Ethereum address: ```python theme={"system"} from eth_account.messages import encode_defunct from eth_account import Account # Recover the address that created this signature message = encode_defunct(text=signature_data["text"]) recovered_address = Account.recover_message( message, signature=signature_data["signature"] ) # Verify it matches the claimed signing address signing_address = signature_data["signing_address"] assert recovered_address.lower() == signing_address.lower() print(f"Signature valid from: {signing_address}") ``` For Ed25519 signatures, the verification process differs - check the signing algorithm in your attestation response. ### Fetch attestation for signing address Now fetch a fresh attestation for this specific signing address. This proves the signing key belongs to verified TEE hardware: ```python theme={"system"} import secrets # Generate fresh nonce for attestation nonce = secrets.token_hex(32) # Fetch attestation for this signing address attestation_response = requests.get( f"https://api.redpill.ai/v1/attestation/report?model={model}&nonce={nonce}&signing_address={signing_address}", headers={"Authorization": f"Bearer {api_key}"} ) attestation_report = attestation_response.json() def find_attestation_for_signer(report, signing_address): candidates = [] if "gateway_attestation" in report: candidates.append(report["gateway_attestation"]) candidates.extend(report.get("model_attestations", [])) candidates.extend(report.get("all_attestations", [])) if "signing_address" in report: candidates.append(report) for item in candidates: if item.get("signing_address", "").lower() == signing_address.lower(): return item raise ValueError("No attestation found for signing address") attestation = find_attestation_for_signer(attestation_report, signing_address) print(f"Found attestation for: {attestation['signing_address']}") ``` In multi-server or two-layer deployments, the response may include several attestations. Filter by `signing_address` to find the one matching your response signature. ### Verify the attestation Finally, verify this attestation using all the checks from [Verify Attestation](./verify-attestation): * Verify Intel TDX quote * Verify report data binds the signing address and nonce * Verify NVIDIA GPU attestation * Verify Docker compose manifest and mr\_config * Verify Sigstore build provenance When all checks pass, you've proven end-to-end that your AI response came from verified TEE hardware running verified software. ## Verify using Etherscan (optional) Want to double-check visually? Use [Etherscan's signature tool](https://etherscan.io/verifiedSignatures). Enter the `signing_address` from your attestation report, paste the `text` field from the signature response, and add the `signature` value. Click verify and Etherscan will confirm everything matches. This gives you an independent third-party verification that the signature is valid. ## Complete example For a full raw Python implementation that verifies both attestation and signatures, see the [signature verifier example](https://github.com/Phala-Network/private-ml-sdk/blob/main/vllm-proxy/verifiers/signature_verifier.py). This script demonstrates the complete flow: 1. Send chat completion request (streaming or non-streaming) 2. Fetch signature for the response 3. Verify request and response hashes match 4. Recover signing address from ECDSA signature 5. Fetch fresh attestation for that signing address 6. Verify full attestation (TDX quote, GPU, report data, compose manifest, Sigstore) You've now verified both the TEE environment (attestation) and individual AI responses (signatures). Your Confidential AI setup is fully verified and trustworthy. # CVM with Docker Compose Source: https://docs.phala.com/phala-cloud/cvm/create-with-docker-compose Deploy a CVM from a docker-compose.yml file using the Phala Cloud dashboard. ## Prerequisites * [Phala Cloud account](/phala-cloud/getting-started/sign-up-for-cloud-account) * Basic understanding of Docker Compose syntax * A valid `docker-compose.yml` for your application ## Deploy in Advanced Mode Phala Cloud supports standard Docker Compose syntax in a secure TEE environment. 1. Go to the deployment section in the dashboard. 2. Switch to the **Advanced** tab. 3. Edit and validate your Docker Compose content in the built-in editor. ## What You Can Configure With Docker Compose deployment, you can: * Specify Docker image names and versions * Configure port mappings for public services * Define environment variables * Configure volume mounts for persistence * Deploy multiple services in one CVM ## Multi-Service Deployments All services in one compose file run inside the same CVM, enabling secure service-to-service communication within the trusted environment. ## Private Container Registry For proprietary or restricted images, Phala Cloud supports private registries including **Docker Hub**, **GHCR**, and **AWS ECR**. ➡️ [Learn more about private registry deployment](/phala-cloud/cvm/create-with-private-docker-image) ## Example Interface Docker Compose deployment interface The built-in editor provides syntax highlighting to help you review configuration before deployment. ## Next Steps * [Set secure environment variables](/phala-cloud/cvm/set-secure-environment-variables) * [Use a private container registry](/phala-cloud/cvm/create-with-private-docker-image) * [Expose your service over HTTPS](/phala-cloud/networking/expose-http-service) # Private Container Registry Source: https://docs.phala.com/phala-cloud/cvm/create-with-private-docker-image Configure private container registry access for CVM deployments on Phala Cloud. ## Prerequisites * [Phala Cloud account](/phala-cloud/getting-started/sign-up-for-cloud-account) * Credentials for one of the supported registries: * Docker Hub * GitHub Container Registry (GHCR) * AWS ECR ## Overview Phala Cloud supports private image pulls during CVM startup. Registry credentials are passed as encrypted environment variables and only used at runtime inside the trusted boot flow. ## Method 1: Configure from the Dashboard 1. Open CVM creation and go to **Advanced Features**. 2. Open **Private Container Registry**. 3. Select one provider: * **No private registry** * **Docker Hub** * **GHCR** * **AWS ECR** 4. Fill in credentials for the selected provider. 5. Submit deployment. Private container registry provider selection All registry credentials are encrypted before submission and consumed only at runtime. ## Method 2: Configure with Encrypted Secrets Set the following environment variables for CLI or API workflows. ### Docker Hub Use your Docker Hub username and password (or [access token](https://docs.docker.com/security/for-developers/access-tokens/)). Image paths follow the format `username/image:tag` or `docker.io/username/image:tag`. See [Docker Hub documentation](https://docs.docker.com/docker-hub/) for account and repository setup. | Variable | Required | Description | | ------------------------ | -------- | ----------------------------------- | | `DSTACK_DOCKER_USERNAME` | Yes | Docker Hub username | | `DSTACK_DOCKER_PASSWORD` | Yes | Docker Hub password or access token | Docker Hub is the default registry — you do **not** need to set `DSTACK_DOCKER_REGISTRY`. ### GHCR GHCR credential fields Use your GitHub username and a PAT with `read:packages` scope. Image paths follow the format `ghcr.io/OWNER/IMAGE:TAG`. With [GitHub CLI](https://cli.github.com/), check your token scopes, add `read:packages` if missing, and print the token: ```bash theme={"system"} gh auth status # check current scopes gh auth refresh -s read:packages # add read:packages if missing gh auth token # print token for DSTACK_DOCKER_PASSWORD ``` You can also create a PAT manually at [GitHub Settings → Personal access tokens](https://github.com/settings/tokens). See [GHCR documentation](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) for package visibility and permission details. | Variable | Required | Description | | ------------------------ | -------- | ----------------------------- | | `DSTACK_DOCKER_USERNAME` | Yes | GitHub username | | `DSTACK_DOCKER_PASSWORD` | Yes | GitHub PAT | | `DSTACK_DOCKER_REGISTRY` | Yes | Must be `ghcr.io` | ### AWS ECR AWS ECR credential fields Use an IAM access key with `ecr:GetAuthorizationToken` and `ecr:BatchGetImage` permissions. Image paths follow the format `ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/REPO:TAG`. With [AWS CLI](https://aws.amazon.com/cli/): ```bash theme={"system"} aws sts get-caller-identity # verify current identity aws ecr describe-repositories --region # list repos to confirm access aws iam create-access-key --user-name # create a new access key ``` Verify registry login: ```bash theme={"system"} aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com ``` See [Amazon ECR documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) for repository creation and IAM policy setup. | Variable | Required | Description | | ------------------------------ | -------- | --------------------- | | `DSTACK_AWS_ACCESS_KEY_ID` | Yes | AWS access key ID | | `DSTACK_AWS_SECRET_ACCESS_KEY` | Yes | AWS secret access key | | `DSTACK_AWS_REGION` | Yes | ECR region | | `DSTACK_AWS_ECR_REGISTRY` | Yes | Full ECR registry URL | ## Deploy with Phala Cloud CLI You can deploy with [`phala` CLI](/phala-cloud/phala-cloud-cli/overview) and pass secrets via `-e .env`. ### GHCR example `.env`: ```bash theme={"system"} DSTACK_DOCKER_USERNAME=your_github_username DSTACK_DOCKER_PASSWORD=your_github_pat DSTACK_DOCKER_REGISTRY=ghcr.io ``` `docker-compose.yml`: ```yaml theme={"system"} version: '3.8' services: app: image: ghcr.io/your-org/your-image:v1.0.0 ports: - "3000:3000" environment: - DSTACK_DOCKER_USERNAME=${DSTACK_DOCKER_USERNAME} - DSTACK_DOCKER_PASSWORD=${DSTACK_DOCKER_PASSWORD} - DSTACK_DOCKER_REGISTRY=${DSTACK_DOCKER_REGISTRY} ``` Deploy: ```bash theme={"system"} phala deploy -n my-private-app -c docker-compose.yml -e .env ``` ### AWS ECR example `.env`: ```bash theme={"system"} DSTACK_AWS_ACCESS_KEY_ID=AKIA... DSTACK_AWS_SECRET_ACCESS_KEY=... DSTACK_AWS_REGION=us-east-1 DSTACK_AWS_ECR_REGISTRY=123456789012.dkr.ecr.us-east-1.amazonaws.com ``` ## Deploy with Phala Cloud API When using the API, set the same environment variables listed above. See [Phala Cloud API docs](/phala-cloud/phala-cloud-api/overview) for endpoint details. ## Pull Rate Limits Each registry enforces pull rate limits. During frequent debugging and redeployment cycles, these limits can block image pulls and cause CVM startup failures. ### Docker Hub | Account type | Limit | | ------------------------------------- | -------------------------- | | Unauthenticated | 100 pulls / 6 hours per IP | | Docker Personal (free, authenticated) | 200 pulls / 6 hours | | Docker Pro / Team / Business | Unlimited | CVM instances pull images on every startup. If you are iterating quickly on a public Docker Hub image **without** setting `DSTACK_DOCKER_USERNAME` / `DSTACK_DOCKER_PASSWORD`, pulls are unauthenticated and share the 100-pull limit across the host IP. Always configure Docker Hub credentials during active development to get the higher authenticated limit. See [Docker Hub pull usage and limits](https://docs.docker.com/docker-hub/usage/pulls/) for details. ### GHCR GitHub does not publish hard pull rate limits for GHCR. Public image pulls are effectively unlimited. Private image pulls are subject to your GitHub plan's data transfer quota. See [GitHub billing for Packages](https://docs.github.com/en/billing/managing-billing-for-github-packages/about-billing-for-github-packages) for details. ### AWS ECR Private ECR does not impose per-image pull rate limits. API calls (e.g. `GetAuthorizationToken`) are throttled at \~20 TPS per region per account. See [Amazon ECR service quotas](https://docs.aws.amazon.com/AmazonECR/latest/userguide/service-quotas.html) for details. ## Troubleshooting If private image deployment fails: 1. Verify credentials are valid and not expired. 2. Ensure image path and tag exist. 3. Confirm pull permissions for the account used. 4. Check CVM serial logs for pre-launch errors. ### GHCR-specific checks * `DSTACK_DOCKER_REGISTRY` is set to `ghcr.io` * PAT contains at least `read:packages` * The package visibility and permission settings allow this account to pull For additional assistance, join support groups: 🌍 [Global](https://t.me/+nbhjx1ADG9EyYmI9), 🇨🇳 [Chinese](https://t.me/+4PcAE9qTZ1kzM2M9). # Deployment Cheat Sheet Source: https://docs.phala.com/phala-cloud/cvm/deployment-cheat-sheet Common gotchas and quick fixes when deploying CVMs, including cross-architecture image builds and Docker log sizing. ## Platform Compatibility ### Build Docker Images on x86 Linux When developing on Mac ARM (M1/M2/M3), build your Docker images for `linux/amd64` before deploying. Phala Cloud runs x86\_64 — ARM images will fail to start. **Resources:** * [Docker Multi-platform Builds](https://docs.docker.com/build/building/multi-platform/) * [Building x86 Docker Images on M1 Macs](https://blog.jaimyn.dev/how-to-build-multi-architecture-docker-images-on-an-m1-mac/) * [Running x86\_64 Docker Images on Apple Silicon](https://nesin.io/blog/x86-x86-amd64-docker-mac) ## Performance Optimization ### Limit Docker Log Size Configure logging options for all services to prevent excessive disk usage: ```yaml theme={"system"} x-common: &common-config restart: always logging: driver: "json-file" options: max-size: "100m" max-file: "5" services: example: <<: *common-config image: example:0.1.0 container_name: example ports: - "8000:8000" ``` # OS Image Source: https://docs.phala.com/phala-cloud/cvm/image/overview Deploy confidential applications with dstack - a TEE-based infrastructure platform. > [Phala Cloud](/) is the managed version of dstack deployment that enables developers to deploy programs to CVM (Confidential VM), and to follow the security best practices by default. The dstack SDK is designed to simplify the steps for developers to deploy programs to CVM (Confidential VM), and to follow the security best practices by default. The main features include: * Deploy any docker containers to a CVM on supported TEEs * Remote Attestation API and a chain-of-trust visualization on Web UI * Automatic RA-HTTPS wrapping with content addressing domain on `0xABCD.dstack.host` * Decouple the app execution and state persistent from specific hardware with decentralized Root-of-Trust Now we use an example to show how dstack works. We assume user application constitutes multiple docker container services, and is configured with tools like docker compose. Confidential Virtual Machine architecture overview showing multiple Docker containers running inside TEE As the architecture shown below, multiple docker containers can run inside one single CVM. The underlying infrastructure we provide is to make sure the application is secure and verifiable. The container uses dstack API to communicate with the underlying `dstack-guest-agent`. `dstack-guest-agent` is the component that is responsible to setup the CVM environment, do remote attestation, and manage the whole lifecycle of docker containers, running inside CVM. `dstack-guest-agent` communicates with a decentralized Key Management Service (KMS), that derives deterministic encryption keys for the application. The keys will be utilized to encrypt storage specific to the application and to protect the its data integrity. With KMS operating independently from any specific TEE instance, your applications avoid vendor lock-in and can be securely migrated between different hardware environments without any data loss. dstack CVM architecture diagram showing dstack API, dstack-guest-agent components, and decentralized KMS integration ## Verify If An Application is Running Inside a TEE When the application launched successfully, the RA Report can be exported with specific interfaces provide `dstack`. The RA Report is bound with the application runtime information, such as the docker image hash, the initial arguments passed to the container, and the environment variables. In addition to signature signed by the key hardcoded in the TEE hardware, the RA report will also be signed with a specific key that bond to the application. Anyone can verify the report with tools supports TEE RA report verification. For applications deployed on Phala Intel TDX workers, their RA report will default be exported and verified, a [TEE Attestation Explorer](https://proof.t16z.com/) is provided for people to check. ## Conclusion Now that there is an introduction to the dstack, we can begin the fun part of building on the SDK. There are a few ways to get started depending on your starting point. Let's take a look at the options in [Getting Started](/phala-cloud/getting-started/overview). Or if you are interested in building on Phala Cloud without runing dstack at your own hardware, you can check out the [Phala Cloud](/) page. # Scale with Multiple Replicas Source: https://docs.phala.com/phala-cloud/cvm/multi-replica-scaling Understand when replicas make sense, what you get from them, and where they fall short. For concrete replication steps, see Replicating CVMs. Replicas let you run several copies of the same application side by side on Phala Cloud. This page covers the concepts and the decision criteria. For concrete dashboard and CLI steps, see [Replicating CVMs](/phala-cloud/cvm/replicating-cvms). ## What a Replica Is A replica is an independent CVM instance provisioned from an existing source CVM. Replicas of the same app share an `app_id` and the same compose hash, but each replica has its own: * `vm_uuid`, endpoint URL, and internal IP * TEE attestation report * Process state, memory, and local disk * Billing meter Replicas are created one at a time from an existing source, not as an atomic group. If you ask for three replicas, Phala Cloud creates three independent CVMs whose only formal relationship is the shared `app_id`. ## What Replicas Give You **Horizontal capacity.** More replicas serve more concurrent traffic, assuming your workload is stateless or has a shared data layer behind it. **Failure isolation.** If one replica's node goes down for maintenance, the others keep serving. A single CVM has no such fallback. **Regional placement.** You can pin different replicas to nodes in different regions by passing a `node_id` when you replicate, which reduces latency for geographically distributed users. **Independent attestation.** Each replica produces its own TDX quote. A relying party can verify any individual replica without trusting the others, which matters when the attestation is part of your security story. ## What Replicas Do Not Give You **Load balancing.** Phala Cloud does not distribute traffic across replicas. Each replica gets its own public endpoint. You need an external load balancer, DNS round-robin, or service mesh to split traffic. **Shared state.** Replicas do not share filesystems, memory, or any in-process state. Two replicas writing to their local disks write to two separate disks. If your app keeps state locally, replicating it will cause divergence. **Automatic propagation of compose updates.** Updating the compose of one replica does not update the others. Each replica is its own CVM and must be updated individually. See [Upgrade Application](/phala-cloud/update/upgrade-application) for the update flow. **Linear performance scaling.** Two replicas do not automatically serve twice the throughput. The upstream bottleneck (database, queue, external API) often dominates. Measure before assuming. ## When to Use Replicas Use replicas when: * The workload is **stateless**, or reads and writes go to a shared backing store (managed Postgres, object storage, an external queue). * You need **high availability** and one CVM outage would be unacceptable. * You want to **distribute traffic geographically** and you have a way to route users to the nearest replica. * The attestation model assumes **independent instances** — each replica must be verifiable on its own. Avoid replicas when: * The workload keeps **state on local disk** and you have no replication layer. Two divergent copies of the same database do not make a cluster. * The workload assumes it is the **only writer** to an external resource (a singleton job scheduler, a leader-elected worker). * You are trying to solve a **vertical scaling** problem. If one replica is slow because it is CPU- or memory-bound, bigger instance types often help more than more copies. See [Resize Resources](/phala-cloud/update/resize-resource). ## Cost Model Each replica is billed as an independent CVM at the full rate of its instance type. Three `tdx.medium` replicas cost three times one `tdx.medium`. There is no bulk discount for adding replicas. Factor this into your scaling budget before raising the replica count. Stopping a replica pauses compute billing but keeps its disk allocation, which is billed separately. Deleting a replica releases both. ## How to Create Replicas The concrete steps live in [Replicating CVMs](/phala-cloud/cvm/replicating-cvms), which covers: * The dashboard **Scale** dialog on the app detail page * The `phala cvms replicate` CLI command * Onchain KMS workflows for contract-governed apps * The replicate API for custom integrations ## Related Documentation * [Replicating CVMs](/phala-cloud/cvm/replicating-cvms) — concrete dashboard, CLI, and API steps. * [Upgrade Application](/phala-cloud/update/upgrade-application) — how to update compose or environment, including across replicas. * [Resize Resources](/phala-cloud/update/resize-resource) — vertical scaling when more replicas is not the answer. * [Deploy Your First CVM](/phala-cloud/getting-started/deploy-first-cvm) — single-CVM getting started. # CVM Overview Source: https://docs.phala.com/phala-cloud/cvm/overview Landing page for CVM creation and management on Phala Cloud. Links to deployment methods, scaling, and lifecycle operations. ## Choose Your Deployment Method Phala Cloud offers multiple ways to create and deploy a Confidential Virtual Machine (CVM). Learn about [Confidential Virtual Machine technology](https://phala.com/confidential-vm) and its enterprise applications. ### Option 1: Phala Cloud dashboard The dashboard walks you through CVM creation with a visual interface. Use it if you are new to Phala Cloud, prefer graphical tooling, or want to deploy without touching the command line. [**Create with Docker Compose →**](/phala-cloud/cvm/create-with-docker-compose) ### Option 2: `phala` CLI The CLI is the right tool for automated deployments, CI/CD pipelines, and scriptable infrastructure. Use it if you are comfortable in a terminal and want reproducible deployments. [**Follow the CLI Guide →**](/phala-cloud/phala-cloud-cli/start-from-cloud-cli) ## CVM Management Features After deployment, manage your CVMs from the dashboard or the API. ### Monitoring * **CVM status** — track the health and status of your CVMs * **Resource usage** — monitor CPU, memory, and storage consumption * **Logs** — view application logs and system messages ### Upgrades * **Environment variables** — update your application's encrypted secrets * **Docker images** — upgrade your application's Docker image * **Resource allocation** — adjust CPU, memory, and storage ### Scaling * **Horizontal scaling** — run multiple replicas of the same CVM for availability and regional distribution. See [Scale with Multiple Replicas](/phala-cloud/cvm/multi-replica-scaling) for concepts and [Replicating CVMs](/phala-cloud/cvm/replicating-cvms) for concrete steps. * **Vertical scaling** — [resize a single CVM's CPU, memory, or disk](/phala-cloud/update/resize-resource) to handle more load per instance. ### Security * **Data protection** — end-to-end encryption of sensitive data at rest * **Access control** — restrict who can reach your CVMs and their logs ## CVM Management API Phala Cloud provides an [API](/phala-cloud/phala-cloud-api/overview) for managing CVMs programmatically. ## Next Steps After creating your CVM, you probably want to: * [Set secure environment variables](/phala-cloud/cvm/set-secure-environment-variables) * [Expose an HTTP service](/phala-cloud/networking/expose-http-service) * [Set up monitoring](/phala-cloud/monitoring/public-logs) * [Replicate the CVM for scale or availability](/phala-cloud/cvm/replicating-cvms) # Replicating CVMs Source: https://docs.phala.com/phala-cloud/cvm/replicating-cvms Scale an existing CVM by creating replicas on the same or different nodes. Covers the dashboard flow, the phala CLI, Cloud KMS, and Onchain KMS approval workflows. Replication creates a new CVM instance with the same configuration as an existing source CVM. Use it to scale out, move a workload to a different node, or split traffic across replicas in different regions. This guide focuses on the two workflows most users should use: the Phala Cloud dashboard and the `phala` CLI. An API reference is included at the end for programmatic integrations. ## Mental Model A replica is a **new, independent CVM instance** provisioned from a **source CVM**. It gets its own `vm_uuid`, endpoint, and attestation. It does not share storage, memory, or running state with the source. **Copied from source (you do not re-specify these):** * The compose file and pre-launch script (same `compose_hash`) * The application ID (all replicas of an app share one `app_id`) * The KMS type and chain * The encrypted environment variables **Things you can change at replication time:** * The target node (pin the replica to a specific node, or let the platform pick one) * The encrypted environment variables (optional override — pass a new encrypted blob to replace the inherited one) **Not carried over:** * Running processes, in-memory state * Contents of persistent volumes and any data written at runtime * Active network connections ### The three replication workflows Which workflow applies depends on the source CVM's KMS setup. You do not pick it — it is determined by how the source was deployed. | Workflow | Applies when | What you do | | ---------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | **Simple** | Source uses Cloud KMS or no KMS | One step: request the replica and wait for it to come up. | | **Onchain, auto-registered** | Source uses Onchain KMS and the DstackApp owner is an EOA with an accessible private key | The CLI detects missing on-chain registrations, signs `addComposeHash` / `addDevice` with the provided key, then creates the replica. | | **Onchain, manual approval** | Source uses Onchain KMS and the DstackApp owner is a Safe, timelock, or DAO | Prepare the replica, collect the on-chain payload, obtain multisig approval, then commit. | Onchain KMS adds gates because the DstackApp contract — not Phala Cloud — decides which compose hashes and devices are allowed to run under the app's identity. If the new replica lands on a node whose device is not already allowlisted, someone has to sign `addDevice` before the KMS will release keys. See [Deploying with Onchain KMS](/phala-cloud/key-management/deploying-with-onchain-kms) for the full model. ## Prerequisites Before replicating: * The **source CVM** exists in your current workspace. It can be running or stopped. * The **target node** (if you pin one) is deployable for your workspace, has the required OS image, and has enough vCPU, memory, and disk. * Your **workspace has enough credit** to cover the new instance. Replicas are billed like any other CVM. For Onchain KMS workflows only: * **Gas** in your signing wallet on the chain the DstackApp lives on (typically Base — 0.001 ETH is plenty for the first few transactions). * **An RPC URL** for that chain. The CLI uses a default public endpoint, or you can pass `--rpc-url` / set `ETH_RPC_URL`. * **Either a private key** (for auto-registration) **or a multisig wallet** (for manual approval). ## Replicate from the Dashboard The dashboard wraps replication in a single "Scale" action on the app detail page. It is the fastest way to add replicas interactively. ### Steps 1. Open the app detail page for the application you want to scale. 2. Click **Scale** to open the Scale dialog. The dialog shows the current list of replicas and lets you add or remove instances. 3. **Pick a source configuration.** If your app has multiple live configurations (different compose hashes), choose the one you want to replicate. If there is only one, the dashboard selects it automatically. 4. **Choose a target node.** Leave it on **Auto** to let the platform schedule the replica, or pick a specific node from the list. The list is filtered to nodes your workspace can deploy to. 5. **Set the number of replicas to add.** Use the `+` / `−` controls, then click the add button. 6. The dashboard creates replicas one by one and shows their status as they provision. ### When the dashboard prompts for on-chain action If the source uses Onchain KMS and the target node's device is not yet allowlisted, the Scale dialog warns you before creating the replica. You then have two options: * **Approve from the dashboard with a connected wallet.** The dashboard walks you through signing `addDevice` (and `addComposeHash` if needed) using the wallet connected to the app. This is the recommended path when the DstackApp owner is an EOA you control. * **Approve externally.** If the owner is a Safe or another contract, use the CLI's manual workflow described below to obtain a commit token, sign on-chain through your multisig, then commit the replica. ## Replicate with the CLI The CLI is the right tool for scripting, CI/CD, and any workflow that does not start from the dashboard. It also exposes the lower-level prepare / commit flow used for multisig-gated apps. Install or update the CLI before starting: ```bash theme={"system"} npm install -g phala phala login ``` ### Basic replication Replicate a CVM to a specific node: ```bash theme={"system"} phala cvms replicate --node-id prod6 ``` `--node-id` accepts either a numeric node ID or a node name as shown in `phala nodes ls` — for example `prod6`, `prod7`, or `use2`. Ambiguous names are rejected rather than guessed, so if a name matches more than one node the CLI asks you to use the numeric ID instead. Omit `--node-id` to let the backend schedule the replica on any node with capacity and a compatible image: ```bash theme={"system"} phala cvms replicate ``` `` can be a CVM UUID, an `app_id`, or a unique CVM name. Prefer UUIDs in scripts — names can collide and `app_id` is ambiguous when an app has more than one live configuration (see [Multi-instance apps](#multi-instance-apps) below). On success the CLI prints the new replica's UUID, node, and dashboard URL: ``` Source CVM ID: app_abc123 Team: my-workspace CVM UUID: 550e8400-e29b-41d4-a716-446655440000 App ID: 0x1234... Name: my-app-replica-1 Status: provisioning Node: prod6 (ID: 5) vCPUs: 2 Memory: 4096 MB Disk Size: 40 GB App URL: https://cloud.phala.com/my-workspace/apps/0x1234.../instances/550e8400... ``` ### Replicating with new environment variables By default the replica inherits the source CVM's encrypted environment. See [Environment Variables](/phala-cloud/cvm/set-secure-environment-variables) for the full encryption model and the rules around `allowed_envs`. To change an env var on the replica — for example, to give a staging replica a different database URL — pass `--env-file`: ```bash theme={"system"} phala cvms replicate --node-id prod6 --env-file .env.prod ``` The CLI: 1. Parses the env file locally. 2. Fetches the CVM's per-application encryption public key from the backend. 3. Encrypts each env var with that key. 4. Sends only the ciphertext to the replicate endpoint. Plaintext env values never leave your machine. Only the running CVM, after the KMS releases its keys, can decrypt them. Env var names you pass with `--env-file` must already appear in the app's `allowed_envs` list — the list is fixed by the compose hash, so replication cannot introduce new names. ### Multi-instance apps When an app has multiple live CVMs with different compose hashes — for example, a canary deployment alongside the stable version — the CLI needs to know which configuration you want to replicate. Running the basic command with just the `app_id` fails: ``` $ phala cvms replicate app_abc123 Error: ERR-03-009 — This app has multiple live CVM instances. Please specify compose_hash to choose which revision to replicate. ``` Two ways to fix it: * **Point at a specific source instance** by UUID. The UUID uniquely identifies one compose hash. ```bash theme={"system"} phala cvms replicate 550e8400-e29b-41d4-a716-446655440000 --node-id prod6 ``` * **Pass `--compose-hash` explicitly.** Look up the compose hash of the configuration you want (from the dashboard or `phala cvms list`) and pass it on the command line: ```bash theme={"system"} phala cvms replicate app_abc123 --compose-hash 0xabcd... --node-id prod6 ``` ### Onchain KMS: auto-registration When the source CVM uses Onchain KMS and you own the DstackApp through an EOA, you can register and replicate in one command by passing a private key. The CLI detects whether the target node's device and the compose hash are already allowlisted and only writes the transactions that are missing. ```bash theme={"system"} export PRIVATE_KEY=0x... export ETH_RPC_URL=https://base-mainnet.example.com phala cvms replicate \ --node-id prod6 \ --private-key $PRIVATE_KEY \ --rpc-url $ETH_RPC_URL ``` The sequence is: 1. The backend prepares the replica and computes the `compose_hash` and the target node's `device_id`. 2. The CLI reads the DstackApp contract to see which registrations are missing. 3. If the device is not allowlisted, the CLI calls `addDevice` and waits for confirmation. 4. If the compose hash is not allowlisted, the CLI calls `addComposeHash` and waits for confirmation. 5. The CLI commits the prepared replica, the backend creates the instance, and the replica boots. If everything is already registered, the CLI skips straight to step 5 and the replica comes up immediately. ### Onchain KMS: manual approval (multisig) When the DstackApp owner is a Safe, a timelock, or any other contract that cannot sign from the CLI, use the prepare + commit flow. **Step 1. Prepare.** Ask the backend to reserve the replica and emit the on-chain payload: ```bash theme={"system"} phala cvms replicate --node-id prod6 --prepare-only ``` The command does not create the replica. Instead it prints a commit token and the values that need to be authorized on-chain: ``` CVM replica prepared successfully (pending on-chain approval). Compose Hash: 0xabcd1234... App ID: 0x1234... Device ID: 0xdevice... Commit Token: prep_token_xyz... On-chain Status: Compose Hash: NOT registered Device ID: NOT registered To complete the replica after on-chain approval: phala cvms replicate \ --commit \ --token prep_token_xyz... \ --compose-hash 0xabcd1234... \ --transaction-hash ``` The commit token is valid for 14 days. Save it somewhere your approvers can reach. **Step 2. Sign on-chain.** From your Safe (or whichever contract owns the DstackApp), call the missing writes on the DstackApp address printed above. Typically: * `addComposeHash(0xabcd1234...)` — if the `Compose Hash` line above shows `NOT registered`. * `addDevice(0xdevice...)` — if the `Device ID` line shows `NOT registered`. Submit the transaction(s) through the Safe UI and wait for execution. Record the final transaction hash. **Step 3. Commit.** Pass the commit token, compose hash, and transaction hash back to the CLI: ```bash theme={"system"} phala cvms replicate \ --commit \ --token prep_token_xyz... \ --compose-hash 0xabcd1234... \ --transaction-hash 0x5678... ``` If both writes are already on-chain and you just want to finalize, pass `--transaction-hash already-registered`. The backend re-reads the contract, verifies the state, and creates the replica. See [Multisig and Governance](/phala-cloud/key-management/multisig-governance) for guidance on structuring the Safe ownership itself. ## API Reference For programmatic integrations — custom portals, CI systems, or orchestration that cannot shell out to the CLI — Phala Cloud exposes the replicate endpoints directly. Prefer the dashboard or CLI for interactive use; the raw API gives you no convenience on top of what the CLI already does. ### POST `/cvms/{cvm_id}/replicas` Create a replica from a source CVM. | Field | Location | Required | Description | | ---------------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `cvm_id` | path | yes | Source CVM identifier. UUID, `app_id`, or unique name. | | `node_id` | body | no | Numeric target node ID. Omit to let the backend pick. The API only accepts numeric IDs; name resolution (e.g. `prod6`) is a CLI feature. | | `compose_hash` | body | no | Explicit compose hash. Required when the source app has more than one live configuration. | | `encrypted_env` | body | no | Hex-encoded encrypted env blob. Omit to inherit the source's env. | | `X-Prepare-Only` | header | no | Set to `true` to prepare the replica without creating it, for the manual approval flow. | On success the response is a `VM` object representing the new replica. On Onchain KMS prerequisites failure the response is HTTP 465 with a structured body containing `commit_token`, `compose_hash`, `device_id`, and `onchain_status`; use those fields as inputs to the commit endpoint after completing on-chain approval. ### POST `/apps/{app_id}/cvms/{vm_uuid}/replicas` Alternative form that takes the app and source instance UUIDs in the URL. Same request body and response shape as the CVM-scoped endpoint. Use this when your integration already tracks `(app_id, vm_uuid)` pairs. ### POST `/cvms/{vm_uuid}/commit-replica` Finalize a replica prepared with `X-Prepare-Only: true`. | Field | Required | Description | | ------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------ | | `token` | yes | Commit token from the prepare response. | | `compose_hash` | yes | The same compose hash returned during prepare. | | `transaction_hash` | yes | On-chain transaction hash for the registration write, or `"already-registered"` when the prerequisites were met before commit. | The response is the same `VM` object you would get from a direct replicate call. ## Errors Replicate calls share the same structured error envelope as the rest of the Phala Cloud API. Common codes you may hit during replication: | Code | Meaning | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | `ERR-01-005` | Onchain KMS requires compose hash or device registration (HTTP 465). Use auto-registration or the prepare + commit flow. | | `ERR-02-003` / `ERR-02-004` / `ERR-02-005` | Target node is out of vCPU, memory, or slots. Drop `--node-id` or pick a different node. | | `ERR-03-006` | The source CVM's OS image is not available on the target node. | | `ERR-03-008` | The source instance is not visible in the current workspace. | | `ERR-03-009` | The source app has multiple live instances; pass `--compose-hash` or use a specific source UUID. | | `ERR-04-001` | Workspace credit balance is too low. | See [Error Codes](/phala-cloud/references/error-codes) for the full catalog, HTTP status mappings, and the exception class behind each code. ## Related Documentation * [Scale with Multiple Replicas](/phala-cloud/cvm/multi-replica-scaling) — When replicas help and when they do not. * [Environment Variables](/phala-cloud/cvm/set-secure-environment-variables) — Encrypted secrets model, `allowed_envs`, and update semantics. * [Deploying with Onchain KMS](/phala-cloud/key-management/deploying-with-onchain-kms) — First-deploy workflow for contract-governed CVMs. * [Multisig and Governance](/phala-cloud/key-management/multisig-governance) — Transferring DstackApp ownership to a Safe and running updates through it. * [Device Management](/phala-cloud/key-management/device-management) — How device IDs are derived and allowlisted. * [Error Codes](/phala-cloud/references/error-codes) — Full error reference. # Environment Variables Source: https://docs.phala.com/phala-cloud/cvm/set-secure-environment-variables Manage encrypted secrets and built-in environment variables for your CVM. ## Encrypted Secrets When your application requires environment variables, **never set them directly in the Docker Compose file**. Instead, use the **Encrypted Secrets** section to ensure your sensitive data remains secure. All encrypted secrets are **encrypted locally on the client side** (using X25519 + AES-256-GCM) before being sent to the server. The Phala Cloud server never sees the plaintext values — only the CVM's TEE can decrypt them at boot time. Encrypted secrets are a list of key-value pairs that are passed into the docker compose file in the same way as the variables defined in `.env` files. You should first define the encrypted secrets in the dashboard (or CLI), and then reference them in the docker compose file using the `${KEY}` syntax. A typical use case is passing secrets to your containers via environment variables, using the `environment:` Docker Compose directive. Updating encrypted secrets is a **full replacement** operation. You cannot update a single variable — every update requires submitting the complete set of all variables. This applies to both the dashboard and the CLI. 1. **Declare Environment Variables in Docker Compose** Define your environment variables in the Docker Compose file using variable substitution: ```yaml theme={"system"} services: your-service: environment: - OPENAI_API_KEY=${OPENAI_API_KEY} - TWITTER_API_KEY=${TWITTER_API_KEY} ``` > **Important:** Do not use double quotation marks around variables:\ > ❌ `OPENAI_API_KEY="${OPENAI_API_KEY_IN_ENV}"` 2. **Set Values in Encrypted Secrets** Configure the actual values in the **Encrypted Secrets** section of the dashboard. Setting environment variables in Encrypted Secrets Secret names don't need to match your Docker Compose environment variable names. You can name secrets anything in the UI, then reference them in Docker Compose using `${KEY}` syntax. Besides the environment variables, you can also reference the encrypted secrets in any other place like the `command:` docker compose directive. However, you should be careful to not leak the secret values in the logs or other places. Learn more about Docker .env files [here](https://docs.docker.com/compose/how-tos/environment-variables/set-environment-variables/). We recommend using **Text** type for environment variables if you have many variables to set. ## Set Secrets via CLI The CLI encrypts variables locally before sending them, just like the dashboard does. ```bash theme={"system"} # Pass individual variables phala deploy -e OPENAI_API_KEY=sk-xxx -e DATABASE_URL=postgres://... # Or pass a .env file phala deploy -e .env ``` To update secrets on an existing CVM, redeploy with the new values. Remember that this is a full replacement, so include all variables every time. ## Set Secrets via SDKs Each SDK provides methods to update environment variables programmatically. The encryption happens client-side before the API call. ```typescript JavaScript theme={"system"} import { createClient } from "@phala/cloud"; const client = createClient({ apiKey: process.env.PHALA_CLOUD_API_KEY }); await client.updateCvmEnvs({ id: "app_abc123", envs: { OPENAI_API_KEY: "sk-xxx", DATABASE_URL: "postgres://user:pass@host:5432/db", }, }); ``` ```python Python theme={"system"} from phala_cloud import create_client client = create_client() client.update_cvm_envs({ "id": "app_abc123", "envs": { "OPENAI_API_KEY": "sk-xxx", "DATABASE_URL": "postgres://user:pass@host:5432/db", }, }) ``` ```go Go theme={"system"} package main import ( "context" "log" phala "github.com/Phala-Network/phala-cloud/sdks/go" ) func main() { client, _ := phala.NewClient() err := client.UpdateCVMEnvs(context.Background(), "app_abc123", &phala.UpdateCVMEnvsRequest{ Envs: map[string]string{ "OPENAI_API_KEY": "sk-xxx", "DATABASE_URL": "postgres://user:pass@host:5432/db", }, }) if err != nil { log.Fatal(err) } } ``` ## Set Secrets via Terraform The Terraform provider supports auto-encryption mode where you provide plaintext values and the provider handles encryption. ```hcl theme={"system"} resource "phala_app" "web" { name = "my-app" size = "tdx.medium" region = "US-WEST-1" disk_size = 40 env = { OPENAI_API_KEY = var.openai_api_key DATABASE_URL = var.database_url } docker_compose = <<-YAML services: app: image: myregistry/my-app:latest ports: - "80:80" environment: - OPENAI_API_KEY=${OPENAI_API_KEY} - DATABASE_URL=${DATABASE_URL} YAML } ``` Even in auto-encryption mode, plaintext values are stored in your Terraform state file. Use a remote backend with encryption at rest, or switch to manual encrypted mode for maximum security. ## Best Practices **Never put secrets in the compose file.** Always use encrypted secrets and reference them with `${KEY}` syntax. The compose file is stored on the server, but encrypted secrets are only decryptable inside the TEE. **Use a `.env` file locally for development.** Keep a `.env.example` in your repo with placeholder values and add `.env` to `.gitignore`. This makes it easy for teammates to set up their own credentials. **Rotate secrets by redeploying.** When you update encrypted secrets, the CVM restarts to pick up the new values. Plan for brief downtime, or run [multiple replicas](/phala-cloud/cvm/replicating-cvms) and rotate them one at a time to maintain availability. **Keep secret names consistent.** Use the same key names in your compose file and encrypted secrets to avoid confusion. While Phala Cloud allows different names, matching them makes your configuration easier to audit. ## Built-in Environment Variables Phala Cloud attaches a [default pre-launch script](https://github.com/Dstack-TEE/dstack-examples/tree/main/phala-cloud-prelaunch-script) to every CVM deployment. This script runs before your Docker Compose services start, handling private registry authentication, root access setup, and injecting environment variables you can reference in your `docker-compose.yml`. The built-in variables documented below are provided by the **default pre-launch script**. If you replace it with a custom script via [`--pre-launch-script`](/phala-cloud/phala-cloud-cli/deploy#deploy-with-pre-launch-script), these variables will no longer be available unless your script implements them. The pre-launch script is versioned (currently **v0.0.14**). New deployments always use the latest version, but existing CVMs keep the version they were deployed with — there is no automatic update. To use a newer version, you need to redeploy your CVM. ### Private Docker Registry Authentication Set the following encrypted secrets to pull images from a private Docker registry (Docker Hub, GHCR, etc.): | Variable | Required | Description | | ------------------------ | -------- | ---------------------------------------------------------------------- | | `DSTACK_DOCKER_USERNAME` | Yes | Registry username | | `DSTACK_DOCKER_PASSWORD` | Yes | Registry password or access token | | `DSTACK_DOCKER_REGISTRY` | No | Registry URL. Defaults to `docker.io`. Set to `ghcr.io` for GHCR, etc. | The pre-launch script logs in to the registry before pulling your images. If login fails, the CVM boot is aborted. ### AWS ECR Authentication Set the following encrypted secrets to pull images from Amazon ECR: | Variable | Required | Description | | ------------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------- | | `DSTACK_AWS_ACCESS_KEY_ID` | Yes | AWS access key ID | | `DSTACK_AWS_SECRET_ACCESS_KEY` | Yes | AWS secret access key | | `DSTACK_AWS_REGION` | Yes | AWS region of your ECR repository | | `DSTACK_AWS_ECR_REGISTRY` | Yes | Your ECR registry URL | | `DSTACK_AWS_SESSION_TOKEN` | No | For temporary AWS credentials (e.g. assumed roles). If expired, the CVM continues to boot but may fail to pull images. | The AWS CLI is installed inside the CVM when ECR credentials are detected — you do not need to include it in your Docker image. ### Root Access | Variable | Required | Description | | ------------------------ | -------- | --------------------------------------------------------------------------------------------------------------- | | `DSTACK_ROOT_PASSWORD` | No | Linux root password for dstack OS. If not set, a random 32-character password is generated. | | `DSTACK_ROOT_PUBLIC_KEY` | No | SSH public key added to `/home/root/.ssh/authorized_keys` | | `DSTACK_AUTHORIZED_KEYS` | No | SSH authorized keys for `/home/root/.ssh/authorized_keys`. Overwrites `DSTACK_ROOT_PUBLIC_KEY` if both are set. | For security, `DSTACK_ROOT_PASSWORD`, `DSTACK_ROOT_PUBLIC_KEY`, and `DSTACK_AUTHORIZED_KEYS` are **unset** from the environment immediately after they are applied. They will **not** be visible to your Docker Compose services. ### App Metadata The following variables are automatically injected by the pre-launch script. You do not need to set them — they are available in your `docker-compose.yml`. | Variable | Description | | ----------------------- | ------------------------------------------------------------------------------------------ | | `DSTACK_APP_ID` | Unique identifier for the CVM app | | `DSTACK_GATEWAY_DOMAIN` | Gateway domain for accessing the app | | `DSTACK_APP_DOMAIN` | Default app domain. Equals `${DSTACK_APP_ID}.${DSTACK_GATEWAY_DOMAIN}`, routes to port 80. | You can reference these in your Docker Compose file: ```yaml theme={"system"} services: my-app: environment: - APP_ID=${DSTACK_APP_ID} - PUBLIC_URL=https://${DSTACK_APP_DOMAIN} ``` `DSTACK_APP_DOMAIN` points to port 80 by default. To access a different port, use the pattern `${DSTACK_APP_ID}-.${DSTACK_GATEWAY_DOMAIN}`. For example, to access port 3000: `${DSTACK_APP_ID}-3000.${DSTACK_GATEWAY_DOMAIN}`. # FAQ Source: https://docs.phala.com/phala-cloud/faqs Deploy confidential applications with Phala Cloud's managed TEE infrastructure. ## Frequently Asked Questions ### What is Phala Cloud? Phala Cloud is a platform that provides TEE virtual machines. These VMs offer hardware-level isolation through Intel TDX technology, allowing for secure and private computation. ### How are the VMs provisioned? Each application gets one CVM (Confidential Virtual Machine), which is one TDX. TDX is a VM technology that provides hardware-level isolation powered by Intel CPU. ### What are the limitations of the TEE environment in Phala? Can my app access file and network IO? * Apps in CVM cannot directly access the host file/network IO. * Internet access for the app is allowed but must go through the proxy module. * Local filesystem access within the CVM is supported, similar to how a docker container accesses its host machine. * Direct access to the host's filesystem is not allowed when deployed on Phala Cloud ### How can I serve a REST API in Phala's TEE environment? You need a web server for serving REST APIs. Requests will be forwarded through a proxy module (dstack-gateway) to CVM. Exposed ports in the docker-compose file will generate a public endpoint for accessing your service. ### Can I run an app with a database inside the container? Yes, local disk access is supported within the container environment, allowing you to run an app with a database. ### Can the encrypted environment variable be accessed by any other part, including the host OS? No, it's encrypted on the client side and sent to the CVM using X25519 encryption scheme. The variables can only be decrypted inside the CVM. ### Can Docker logs be accessed by any other part, including the host OS? Currently, the logs are not end-to-end encrypted. However, you can decide whether to make them public under **Advanced Features → Public Logs** during deployment. If you choose public, anyone with the log URL can view them, and the Log URL can be inferred from your instance ID in the public endpoints. If you choose private, no one can access the logs, and you'll need to configure a log viewer to view them elsewhere. ### Is application data persistent on the disk? Yes, the data you write to the filesystem inside Docker will persist on the disk and be encrypted. Restarting or upgrading will not affect data recovery. To save data on disk, you need to [configure volumes](https://docs.docker.com/reference/compose-file/volumes/) in the Docker Compose file and write data to the correct path. ### Can I deploy / manage / upgrade CVMs programmatically? Yes. Resources for programmatic management: * [Cloud API docs](/phala-cloud/phala-cloud-api/overview) * [API examples (JavaScript)](https://github.com/Phala-Network/phala-cloud) * [CI/CD pipeline setup](/phala-cloud/phala-cloud-cli/ci-cd-automation/setup-a-ci-cd-pipeline) ### Does the “update” feature sync the latest Docker image? No. By default, Docker does not proactively update the image unless the image reference is changed (e.g., by modifying the tag or the hash), even if an update is triggered. This is because the update feature is essentially equivalent to running: `docker compose down && docker compose up`. To ensure the image is actually updated, it's recommended to always update the tag. Additionally, for best security practices, it's advised to include the image hash to trace the TEE proof back to the exact Docker image file. ### Does a CVM support multiple containers? Yes, you can deploy multiple containers per CVM through the docker compose file. ### Does the cloud support a firewall? You can apply firewall rules inside your containers. For example, add Nginx to apply IP address filters or WAF rules. There is no cloud-level firewall currently. ### Does it support Terraform deployment? Yes! The Terraform provider is now available. See the [Terraform Provider documentation](/phala-cloud/references/terraform-provider/overview) for setup and usage. ### Can CVM accept arbitrary TCP / TLS connections? CVMs support both HTTP-over-TLS (Zero Trust HTTPS) and TCP-over-TLS. See the [TCP port forwarding example](https://github.com/Dstack-TEE/dstack-examples/tree/main/tcp-port-forwarding) for exposing SSH or other TCP ports. Under the hood, the TEE wraps the tcp ports you have exposed in docker with TLS. The gateway then routes the TLS traffic based on SNI, the domain name attached to every TLS connection. TLS is needed because it's the only way to multiplex the tcp traffic. When using the tcp-port-forwarding example, it automatically helps you set up the domain and TLS certificate securely in TEE (thus “Zero Trust”), and reverse-proxy the incoming traffic to your tcp port. You can also manually accept TLS traffic in your CVM and handle the certificate on the application level. ### Is there a simulator or free tier available for testing? Yes. The Phala Cloud CLI includes a [simulator](/phala-cloud/phala-cloud-cli/simulator) for local testing. New accounts also receive free credits to deploy a small CVM. ### Is Phala Cloud based on serverless functions or traditional VMs? What are the startup and shutdown times? Phala Cloud uses a VM-based model, not serverless functions. While exact boot time can vary, users can create a CVM to measure startup latency. For most scenarios, it's best to keep a small instance running continuously and scale up during traffic spikes. Spinning up a new VM for every request may not be efficient. There is currently no TEE-based serverless function, but the team is open to collaborating on building one. ### How can I access my docker volume data after deploy? The docker volume is persisted in the TEE encrypted volume. By default it's not accessible by humans or the cloud. However, if your business logic requires to expose the volume or part of the volume, you can do it with your backend code manually. For example if you want to allow users to upload images and download it later, you can implement a simple backend web server with file uploading and static file serving. Here are the example tutorials from Flask ([upload](https://flask.palletsprojects.com/en/stable/patterns/fileuploads/), [static file serving](https://flask.palletsprojects.com/en/stable/tutorial/static/)). And it's also easy to find the tutorials for other web servers (nginx, express, etc). ### Can I send encrypted values (like API keys) through the Phala Cloud API for deployment? Yes, Phala Cloud supports sending encrypted values (e.g., Twitter API keys/secrets) through the API. You must encrypt the variables locally before sending them. The encrypted values can then be included in your deployment request. You can also send them through the feature of Encrypted Environment Variable. If you use CLI, you can set it like [here](https://github.com/Phala-Network/phala-cloud-cli?tab=readme-ov-file#environment-variables-management), or if you on the Cloud UI, use the "Encrypted Environment Variables" feature during VM creation. ### How long does it take to deploy a Docker image in Phala Cloud? Deployment time depends on the Docker image size and decompression requirements: * A small Python-based image (e.g., kennethreitz/httpbin) typically takes about 1-2 minutes. * A larger image like ElizaOS/Eliza (1.4GB) takes 20-25 minutes, with most time spent decompressing after a fast pull. ### How fast can a simple program (e.g., multiplying two numbers) execute in a TEE? Execution speed for a simple operation like multiplying two numbers within a TEE depends on the setup. If the TEE environment (e.g., a Docker container) is already running, execution is near-instantaneous (milliseconds), similar to non-TEE environments. However, if deployment or cold start is required, the total time includes the container pull and decompression (1-2 minutes, as above). For a function-as-a-service (FaaS) experience with seconds-scale execution, the TEE server must be pre-deployed and running, not started on-demand. ### Can I update specific environment variables for a deployed Docker image without changing the entire set? Currently, Phala Cloud does not support updating individual environment variables—you must update all variables together. However, this feature is on the development roadmap. As a workaround: Use two separate environment files: one for static variables (e.g., required for the app to run) and another for dynamic, user-editable variables (e.g., trading parameters). Alternatively, integrate a dedicated HTTPS API endpoint within your deployed instance to allow users to update specific variables without redeploying.\ Discuss custom solutions with the Phala team if this is critical for your use case. ### How can I deploy multiple Docker instances on a single VM to optimize costs? You can deploy multiple Docker instances on a single Phala Cloud VM using a "Docker-in-Docker" approach, similar to what large clients have implemented (e.g., running 150+ instances). To do this: Step 1: Create a larger VM (e.g., 10 vCPUs, 20GB RAM).\ Step 2: Deploy your Dockerized images inside this VM, adjusting ports or configurations as needed to avoid conflicts.\ Step 3: Use custom environment variables per instance for differentiation, for example, set the image name or version through environment variable.\ This reduces overhead (e.g., from multiple OS instances) and can lower costs. Phala can assist with implementation—reach out for support. ### What endpoint do I use to view logs for a specific instance? To access logs for a specific instance, if you use Cloud UI, you can check the log on the container page. If you use the Cloud API, you can use the endpoint GET /api/v1/cvms/app\_\/composition. This returns details about all containers within the Compute VM (CVM), including the log endpoint for each. Replace \ with your instance’s unique identifier. Find the full API specification at /phala-cloud/phala-cloud-api/overview. Ensure you authenticate with your API key in the x-api-key header. ### Can I set my own TLS certificate for my CVM network access? Yes. After configuring your certificates, expose port `8080:443` in your Docker Compose file. To access the service over your TLS certificate, use the public endpoint from the Network section and append `s` to the port number — for example, `-8080s.`. ### How does Phala Cloud ensure that the data in a CVM isn’t accessed by other programs running in the same machine? Phala Cloud CVM uses a hardware root key unique to the TEE platform, from which individual encryption keys are derived for each application. This key hierarchy ensures that each TEE app has its own distinct key, preventing other programs in the same TEE from accessing your encrypted data. Additionally, you should attest the programs running on Phala Cloud (e.g., via remote attestation) to verify they’re not malicious, enhancing security. ### How can others verify that my application is running inside a TEE? Once the application is running, you can prove this by providing the RA Report, which is exported through an endpoint by your Docker application. See [How to Generate Attestation Reports](https://phala.com/posts/how-to-generate-attestation-repport-and-prove-your-application-runs-in-tee) for details. ### What’s the best approach to verify Phala Cloud Attestation Report in a smart contract? You can verify the Attestation Report with [Automata's onchain DCAP verifier](https://github.com/automata-network/automata-dcap-attestation) written in Solidity and deployed on multiple blockchains. To verify the report, user can get the report hex data from Phala Cloud dashboard “Attestation” tab, and call the smart contract method verifyAndAttestOnChain, check the example [here](https://github.com/Leechael/ra-quote-explorer/blob/37a86a96f0f5059359b33fdb61b1f507fd1ca291/src/components/onchain_attestation.tsx#L152). ## Pricing and Resources ### What is the current pricing structure for Phala Cloud? Check the [pricing page](/phala-cloud/pricing) for more details. **Quick summary:** * **Compute**: Billed per-minute based on instance type (only while running) * **Storage**: $0.000139/GB/hour (~$0.10/GB/month) - charged even when stopped * **GPU TEE**: Billed hourly with 24-hour minimum for on-demand ### What kind of user levels do we have on Phala Cloud? * **Free**: User who registered on Phala Cloud, with limited resources * **Pro**: Paid users with self-service capabilities * **Enterprise**: Customized pricing, available through Business Development only ### What are the account tier limits? | Tier | Max CVMs | Max vCPUs | Max Memory | Max Total Disk | | -------------- | --------- | --------- | ---------- | -------------- | | **Tier 1** | 8 | 16 | 32GB | 160GB | | **Tier 2** | 16 | 32 | 64GB | 1TB | | **Enterprise** | Unlimited | Unlimited | Unlimited | Unlimited | New accounts start at Tier 1. Contact [cloud@phala.network](mailto:cloud@phala.network) to upgrade. **Important:** Check your tier limits BEFORE deploying multiple CVMs. You'll receive a "No available nodes found" error if you exceed your limits. #### What are the resource limits for the Free tier users? Users under **Free** tier can only create one single CVM. Verify your account and set up payment to access Tier 1 limits. ### What is the billing granularity for Phala Cloud? Is it by hour or minute? **Regular CVMs:** Billed by seconds. The pricing page displays hourly rates for ease of understanding, but actual billing is based on precise second-level usage. **GPU TEE:** Billed by hours with a 24-hour minimum. See [GPU TEE Billing](/phala-cloud/pricing#gpu-tee-billing-rules) for details. ### How does GPU TEE billing work? For On-Demand GPU TEE (e.g., H200): * **Minimum charge**: \$84 upfront for the first 24 hours * **When billing starts**: Only when the instance is actually launched * Topping up your account or configuring Docker settings does NOT incur charges * Usage beyond 24 hours is billed hourly * If you stop and restart later, the 24-hour minimum applies again For commitment plans (1-month or 6-month), you get lower hourly rates. ### What happens if my balance reaches zero? Phala follows a no-surprise policy: * Existing clients will have a 2-month grace period after reaching zero balance * Your VMs will not be shut down without your consent during this period * The Business Development team will contact you to confirm pricing and terms ### Are there any anti-spam measures? Yes, Phala implements several measures: * Payment verification is required even for free VMs to prevent automated registrations * LLM APIs are not provided for free without a specific reason * Free VMs may be offered when they don't impact paying users * Certain Docker base images may be blocked if they've been used for abuse If your deployment is blocked with "violates terms & conditions", try using official base images (e.g., `python:3.11`, `node:20`) or contact support if you believe it's a false positive. ### Can I deploy CVMs through GitHub Actions / CI/CD? Yes! Phala Cloud supports automated deployments through GitHub Actions. See the [CI/CD Pipeline Guide](/phala-cloud/phala-cloud-cli/ci-cd-automation/setup-a-ci-cd-pipeline) for: * Setting up repository secrets * Configuring the deployment workflow * Using the `phala-deploy-action` You can also use the [Phala Cloud API](/phala-cloud/phala-cloud-api/overview) for custom automation. ### Is Phat Contract / DevPhase / Swanky still supported? **Phat Contract has been sunset.** DevPhase and Swanky are no longer supported. The tech stack has moved to **dstack**-based infrastructure. For new projects, use: * [Phala Cloud CVMs](/phala-cloud/what-is/what-is-phala-cloud) for TEE deployments * [dstack](/dstack/overview) for the underlying TEE infrastructure # Get and Verify Attestation Source: https://docs.phala.com/phala-cloud/getting-started/attestation Get a TEE attestation report from your CVM and verify it step by step. ## Introduction Remote Attestation (RA) lets a remote verifier confirm that an application runs inside a genuine TEE with the expected configuration and code. This guide shows how to access and verify TEE attestation reports in Phala Cloud. ### How to Get the Attestation Report Navigate to **Attestations** in your CVM's sidebar to view the full Remote Attestation report. Phala Cloud attestation page showing TEE proof and remote attestation report ## How to Verify Attestation Scroll down to the **Certificate Chain** section and click the **Check Attestation** button next to **App Cert**. Certificate chain section with Check Attestation button highlighted for App Cert verification This opens the [TEE Attestation Explorer](https://proof.t16z.com/) where you can verify the attestation quote. You can share this quote with anyone as proof that your application is running inside a genuine TEE. TEE Attestation Explorer showing verified attestation status with detailed verification report You can also request customized Remote Attestation reports programmatically via API: [Generate Remote Attestation](/phala-cloud/attestation/get-attestation). For more details, see [how dstack attestation works](https://github.com/Dstack-TEE/dstack/blob/master/attestation.md). # Core Concepts Refresher Source: https://docs.phala.com/phala-cloud/getting-started/core-concepts-refresher Key building blocks of Phala Cloud — CVMs, key management, and GPU TEE. Before deploying, familiarize yourself with the three core building blocks of Phala Cloud. Confidential VM (CVM) Key Management Service (KMS) GPU TEE # Deploy Your First CVM Source: https://docs.phala.com/phala-cloud/getting-started/deploy-first-cvm Go from zero to a running Confidential VM on Phala Cloud using the Dashboard, CLI, or SDK. You can deploy a CVM on Phala Cloud in under five minutes. This guide walks you through the full flow — sign up, write a compose file, deploy, and verify — using whichever tool you prefer. We'll deploy an nginx web server as the example app. The same steps apply to any Docker image. ## Prerequisites * A [Phala Cloud account](/phala-cloud/getting-started/sign-up-for-cloud-account) * For CLI/SDK: an [API key](/phala-cloud/references/api-key) from **Settings > API Keys** in the dashboard ## Choose an Instance Type Phala Cloud offers several instance sizes. For a simple web server, `tdx.small` is plenty. | Instance Type | vCPU | RAM | Good For | | ------------- | ---- | ----- | ---------------------------------- | | `tdx.small` | 2 | 4 GB | Lightweight services, testing | | `tdx.medium` | 4 | 8 GB | Production APIs, small databases | | `tdx.large` | 8 | 16 GB | Multi-container apps, ML inference | See all options with [`phala instance-types`](/phala-cloud/phala-cloud-cli/instance-types). ## Write Your Docker Compose File Create a `docker-compose.yml` that defines your services. Here's a minimal nginx example: ```yaml theme={"system"} services: web: image: nginx:stable ports: - "80:80" ``` Every service in the compose file runs inside the same CVM, so containers can communicate over `localhost`. ## Deploy Pick the method that fits your workflow. All three produce the same result: a running CVM with a public HTTPS endpoint. ### Option A — Dashboard UI ### Open the Deploy page Navigate to your workspace and click **Deploy** in the top-right corner of the CVMs page. ### Configure the CVM Enter a name (e.g. `my-nginx`), paste your `docker-compose.yml` into the editor, select **Phala** as the KMS provider, pick a region, and choose **tdx.small** as the instance type. ### Click Deploy You'll be redirected to the CVMs page. The status changes from **starting** to **running** in about one to two minutes. ### Option B — CLI ### Install the CLI ```bash theme={"system"} npm install -g phala ``` ### Deploy Run `phala deploy` from the directory containing your `docker-compose.yml`. The CLI auto-selects a region and instance type if you don't specify them. ```bash theme={"system"} export PHALA_CLOUD_API_KEY="phak_your_key" phala deploy -n my-nginx -t tdx.small --wait ``` The `--wait` flag blocks until the CVM is fully running. ### Option C — SDK ```typescript JavaScript theme={"system"} import { createClient } from "@phala/cloud"; const client = createClient({ apiKey: process.env.PHALA_CLOUD_API_KEY }); const provision = await client.provisionCvm({ name: "my-nginx", vcpu: 2, memory: 4096, diskSize: 20, composeFile: { dockerComposeFile: ` services: web: image: nginx:stable ports: - "80:80" `, }, }); const cvm = await client.commitCvmProvision({ appId: provision.appId, composeHash: provision.composeHash, transactionHash: provision.transactionHash, }); console.log("CVM deployed:", cvm); ``` ```python Python theme={"system"} from phala_cloud import create_client client = create_client() # reads PHALA_CLOUD_API_KEY from env result = client.provision_cvm({ "name": "my-nginx", "compose_file": { "docker_compose_file": "services:\n web:\n image: nginx:stable\n ports:\n - '80:80'", }, "vcpu": 2, "memory": 4096, "disk_size": 20, }) cvm = client.commit_cvm_provision({ "app_id": result.app_id, "compose_hash": result.compose_hash, "transaction_hash": result.transaction_hash, }) print("CVM deployed:", cvm) ``` ```go Go theme={"system"} package main import ( "context" "fmt" "log" phala "github.com/Phala-Network/phala-cloud/sdks/go" ) func main() { client, err := phala.NewClient() if err != nil { log.Fatal(err) } result, err := client.ProvisionCVM(context.Background(), &phala.ProvisionCVMRequest{ Name: "my-nginx", ComposeFile: &phala.ComposeFile{ DockerComposeFile: "services:\n web:\n image: nginx:stable\n ports:\n - \"80:80\"", }, VCPU: phala.Int(2), Memory: phala.Int(4096), DiskSize: phala.Int(20), }) if err != nil { log.Fatal(err) } cvm, err := client.CommitCVMProvision(context.Background(), &phala.CommitCVMProvisionRequest{ AppID: result.AppID, ComposeHash: result.ComposeHash, TransactionHash: result.TransactionHash, }) if err != nil { log.Fatal(err) } fmt.Printf("CVM deployed: %+v\n", cvm) } ``` ## Verify Your Deployment Once the status shows **running**, your CVM has a public endpoint. The URL follows this pattern: ``` https://-.dstack-pha-.phala.network ``` You can find the exact URL in the dashboard under **CVM > Home > INGRESS**, or query it from the CLI: ```bash theme={"system"} phala cvms get my-nginx ``` Open the endpoint in your browser. You should see the nginx welcome page. Endpoints may take 30 to 60 seconds to become reachable after the CVM enters the running state. ## Verify TEE Attestation Every CVM generates a Remote Attestation report proving it runs inside a genuine TEE. Navigate to **Attestations** in the CVM sidebar and click **Check Attestation** to verify it in the [TEE Attestation Explorer](https://proof.t16z.com/). For details, see [Get Attestation and Verify](/phala-cloud/getting-started/attestation). ## Next Steps * [Environment Variables](/phala-cloud/cvm/set-secure-environment-variables) — pass secrets securely to your CVM * [Docker Compose Configuration](/phala-cloud/cvm/create-with-docker-compose) — multi-service setups and private registries * [Set Up CI/CD](/phala-cloud/phala-cloud-cli/ci-cd-automation/setup-a-ci-cd-pipeline) — automate deployments on every push * [Scale with Replicas](/phala-cloud/cvm/multi-replica-scaling) — run multiple copies of your app # Deploy ERC-8004 Agent in 5 Minutes Source: https://docs.phala.com/phala-cloud/getting-started/explore-templates/deploy-erc-8004-agent Deploy a verifiable ERC-8004 Trustless Agent using Phala Cloud's VibeVM with TEE attestation ## TL;DR * **ERC-8004** gives us onchain primitives to register and discover *trustless agents*. * To make those registrations meaningful, you want verifiable runtime evidence, not just a signature. * **Confidential VMs (CVMs)** running **dstack** to deliver **deterministic key derivation** + **remote attestation** so your agent can prove *where* it executed. * **Phala Cloud** wraps this in a bow: VibeVM templates, an onchain KMS/registry workflow, and (optionally) confidential inference (GPU TEE) support. * This post walks the whole flow: deploy a CVM → produce attestation → register identity on Sepolia → verify via **TEERegistry**.