Confidential AI Inference

The problem

Modern AI inference forces a three-way trust failure:

1. User to provider. Users hand raw prompts, images, documents, medical records, or code to a remote GPU they do not control. The provider (and anyone with subpoena or coercion power over it) sees everything.
2. Provider to user. Providers must ship model weights to any host that runs the workload. Weights get stolen, leaked, or exfiltrated by customers, insiders, or compromised hypervisors.
3. Both to infrastructure. Cloud operators, hypervisors, orchestration layers, and supply chains sit between user and model and can silently swap models, log prompts, or tamper with outputs.

Confidential inference closes all three gaps by running the model inside a Trusted Execution Environment whose identity (code hash, firmware version, GPU identity) is cryptographically attested to the counterparty before any prompt or weight is sent.

The ideal trust property

A fully confidential inference service satisfies all of the following for every request:

No production system hits 7/7 yet. The state of the art (Apple PCC, NVIDIA H100 CC plus attestation, Tinfoil, Phala, Oasis, Marlin) hits 4-6 depending on threat model.

Architectural patterns

1. Single-TEE CPU inference (small models)

The whole model runs inside an Intel TDX or AMD SEV-SNP confidential VM. No GPU.

• Fits: models below 20B parameters that can tolerate CPU latency, or fine-tuned small transformers (embeddings, classifiers, tool-call routers).
• Providers: Azure DCesv5 (TDX) or DCasv5 (SEV-SNP), GCP C3 (TDX) or N2D (SEV-SNP), AWS M6a/C6a (SEV-SNP).
• Example stacks: Gramine + llama.cpp, Enarx, Occlum, MSFT Confidential Inferencing Containers.
• Limitation: no matrix-multiply accelerator. 50-200x slower than a GPU for large models.

2. CPU-TEE plus GPU-TEE (the modern pattern)

A confidential VM (TDX/SEV-SNP) is paired with a confidential GPU (NVIDIA H100 / H200 / Blackwell in CC mode). Protected CPU–GPU transfers use encrypted bounce buffers over the untrusted link. The CPU and GPU supply attestation evidence that the deployment must validate and bind to its secure channel.

• Fits: frontier LLMs, diffusion, multimodal — anything GPU-bound.
• Providers: Azure NCC H100 v5, GCP A3 Confidential, Oracle OCI H100 CC, Coreweave/Lambda early access.
• Key doc: NVIDIA-GPU-TEE.
• Caveat: 5-15% throughput cost; first-token latency goes up by 30-120ms from the attestation handshake.

3. Split / federated inference

Embedding layer runs in a CPU TEE near the user (or on-device); deep layers run on a remote GPU TEE; activations are encrypted between hops.

• Pro: limits blast radius if one enclave is broken.
• Con: activation-leakage attacks (reconstruction from hidden states) are an active research area. Do not assume "activations ≈ noise."

4. Cryptographic inference (FHE / MPC / ZKML)

Not TEE-based at all: homomorphic encryption, secure multi-party computation, or zero-knowledge ML circuits.

• FHE (Zama Concrete-ML, OpenFHE): input privacy and weight privacy, but ~10³-10⁶× overhead. Only viable for tiny models today.
• MPC (CrypTen, MP-SPDZ): 2-3 non-colluding parties, interactive. Good for specific enterprise joint-compute deals.
• ZKML (EZKL, Giza, RISC Zero + onnx): proves the output was computed correctly from a committed model. Not input privacy by itself. Pair with TEE for the full story.

5. On-device inference

Run the model on the user's own hardware: Apple Neural Engine, Qualcomm Hexagon, local llama.cpp on M-series or 5090. No confidentiality problem because the user is the trusted party.

• Gold standard for privacy when the model fits. Always ask "can this run locally?" before reaching for a TEE.
• See Overview - Confidential Computing §Use Cases.

Production and near-production systems

Apple Private Cloud Compute (PCC), 2024 onward

The most ambitious consumer deployment. An iPhone can offload an Apple-Intelligence request to PCC servers where:

• Custom Apple Silicon server nodes run a hardened variant of iOS/Darwin with Secure Enclave, Secure Boot, and sealed memory.
• Every build image is reproducibly hashed and published to a transparency log; clients refuse to send prompts to any node whose image hash is not in the log.
• No persistent storage, no admin interface, no remote shell. The design aims to prevent retention of user request data while allowing restricted operational metrics.
• End-to-end encrypted session keyed to attestation.

Why it matters: the first mass-market system where "the server operator genuinely cannot see your prompt" is a deployed guarantee rather than marketing. Threat model writeup: Apple Private Cloud Compute: security design.

What it is not: open. Only Apple builds on Apple silicon can join. No third-party audit of running instances; you trust the transparency log and Apple's signing keys.

NVIDIA H100 / H200 / Blackwell Confidential Computing

NVIDIA's Hopper (and Blackwell) GPUs ship a hardware root of trust, protected CPU–GPU data transfers (SPDM-established keys and AES-GCM), and attestable firmware. Paired with a TDX or SEV-SNP host, they form the only production GPU-TEE today. See NVIDIA-GPU-TEE for the full mechanism.

Tinfoil (tinfoil.sh)

• CPU-TEE (SEV-SNP / TDX) plus NVIDIA CC GPU hosting of open-weights LLMs (Llama 3.x, DeepSeek, Mistral).
• Clients verify attestation in-browser before any prompt is transmitted.
• Reproducible container builds published to Sigstore; the attestation quote commits to the exact image digest.
• Threat model: trusted = NVIDIA + AMD/Intel silicon + Tinfoil's build pipeline (transparency-logged). Untrusted = Tinfoil operators, cloud, ISPs.

Phala Cloud / Phala Network

• Decentralized TEE marketplace (Intel SGX, then TDX, with NVIDIA CC support).
• Workloads run on chain-registered worker nodes; attestation quotes are verified by the blockchain before a job is considered valid.
• The dstack SDK packages docker-compose workloads into TDX CVMs with deterministic attestation.

Oasis (Sapphire / ROFL)

• Confidential EVM (Sapphire) runs smart-contract state in SGX.
• ROFL (Runtime Off-chain Logic) runs larger compute, including LLM inference, in TDX with on-chain attestation receipts.
• Good fit when you want "confidential inference as a verifiable oracle."

Marlin (Oyster)

• CVM marketplace (AWS Nitro Enclaves + NVIDIA CC).
• Focus: serverless confidential functions; pay-per-second; attestation returned with every response.

Edgeless Systems (Contrast, Constellation, MarbleRun)

• Enterprise Kubernetes plus CoCo stack; attestation-gated secret injection into confidential pods.
• Constellation: always-encrypted K8s cluster on SEV-SNP / TDX.
• Used for private inference in regulated industries (health, finance).

Secret Network / Fortanix / Anjuna

• Secret Network: SGX-backed confidential smart contracts, some ML-oracle experiments.
• Fortanix Confidential Computing Manager: enterprise TEE orchestration, SGX/TDX.
• Anjuna: "lift and shift" confidential VM wrapper. No code changes; auto-enables SEV-SNP/TDX plus attestation gating on AWS/Azure/GCP.

Attack surface specific to inference

Even a perfect TEE does not give you a perfect confidential-inference system. Model-specific and workflow-specific leaks:

Prompt and activation leakage

• Timing side channels: decode latency varies with token choice, KV-cache state, and speculative decoding hit/miss. An observer of response timing alone can sometimes reconstruct which tokens were generated. Mitigations: constant-time decoding (rare), padded batches, fixed max_tokens, jittered emission.
• Memory access patterns: even inside an enclave, DRAM traffic is visible to the memory controller. Sparse-MoE routing has been shown to leak the active expert, which leaks the token class. Dense models are safer; MoE in TEE is an open problem.
• Speculative decoding across tenants: shared draft models leak cross-request information. Do not share drafts across confidentiality domains.

Output channels

• Model memorization: a confidential model may still regurgitate training-data secrets. TEEs protect this request's data, not the corpus the model memorized.
• Logit leakage via streaming: per-token streaming plus timing equals more side-channel surface than batched completion. Prefer batched for high-sensitivity use.
• Tool-use side effects: the moment the model makes an external HTTP call the enclave guarantee ends. Keep tool execution inside a second enclave, or accept the leak.

Weight exfiltration

• Attester-less weight delivery is a bug. Weights must be encrypted to the enclave measurement, not to a long-lived account key the operator holds.
• Page-swap and cold-boot: older SGX could be coerced into paging EPC; SEV-SNP and TDX mitigate but do not eliminate this. Disable hibernation on confidential hosts.
• Model stealing via queries: unrelated to TEEs. If the API is public, adversaries can distill weights through query volume. Rate-limit and watermark.

Attestation-pipeline attacks

• Stale TCB: an attestation can be valid and vulnerable. Always require a minimum TCB level and reject SEV-SNP microcode below the latest CVE-fixed version.
• Collateral DoS: Intel and AMD attestation services can be unavailable. Cache TCB info and verifier collateral locally; support offline verification.
• Rogue signer: a compromised CPU signing key ends the model. Multi-vendor deployments (TDX and SEV-SNP nodes with quorum) reduce systemic risk; see Attestation-Architecture.

Supply chain

• Build reproducibility: if you cannot rebuild the enclave image bit-for-bit, your attestation quote means "trust the build server," not "trust the code." Use SLSA L3+, sigstore, and transparency-logged image hashes.
• Dependency creep: every added Python package expands MRENCLAVE and shrinks auditability. Minimal runtimes (musl + static llama.cpp) beat full PyTorch stacks for high-assurance contexts.

Threat model checklist (use before deploying)

1. Who is in your TCB? List every party: silicon vendor, firmware signer, BIOS, hypervisor vendor, image builder, transparency-log operator, key-release service. Anyone on that list can break you.
2. Where does the plaintext prompt first appear? Client TLS to TEE public key? Verify the TLS endpoint is inside the enclave, not a front-end proxy.
3. How is the model key released? KMS to enclave via attested channel? Or baked into the image (rotation nightmare)?
4. What does "no logging" mean? Structural (no disk, no syslog compiled in) or policy (config flag)? Only structural counts.
5. Freshness? Does the attestation quote include a client-supplied nonce? Otherwise replay is trivial.
6. Revocation? If a CVE drops today, how do you turn off every vulnerable TCB right now? Pre-wire the kill switch.
7. What does the user verify? A one-line "this is secure" UI is theater. Give them the image hash plus a way to check it against the transparency log, even if 99% ignore it.

Decision matrix

Primary sources

• Apple: Private Cloud Compute Security Guide
• NVIDIA: Confidential Computing on H100 GPUs
• Confidential Computing Consortium: Technical Analysis
• Intel SGX Explained
• Li et al.: A Survey of Secure Computation Using Trusted Execution Environments
• Microsoft: Confidential AI
• Apple Private Cloud Compute: security design
• ONNX Runtime Open Enclave port for confidential inference

Also in Decentralized Compute