Summary: Cohere launched Confidential Computing on Cohere Model Vault into public beta on October 22, 2026, running on NVIDIA confidential GPUs on Microsoft Azure. The interesting part is not just encrypted inference — the client verifies signed attestation evidence against a customer-approved policy before any protected request leaves your environment. For builders handling sensitive prompts or documents, this changes who you have to trust and how that trust is enforced.
Why a confidential VM isn’t enough
Confidential computing protects data in use. Two properties do the work: isolation, where hardware-enforced controls shield workload memory from the host, and attestation, which produces signed evidence about the environment. Together they break the old assumption that privileged access to a host implies access to everything running on it.
For inference, though, the boundary has to extend past the CPU. Much of the actual computation runs on accelerators, so protecting only the CPU would leave sensitive model state exposed exactly where the work happens. NVIDIA’s confidential GPU support encrypts CPU-GPU traffic and protects GPU memory, letting the trusted boundary span the whole inference path. Azure supplies the confidential infrastructure; Model Vault adds the managed serving, encrypted requests, attestation, and verification layers on top.
Encrypting the payload, not just the connection
TLS protects the connection, but intermediaries operating a managed service still sit inside the plaintext path. Model Vault instead encrypts the application payload itself using Oblivious HTTP (OHTTP): requests are encrypted to a public key tied to the attested environment, and only an OHTTP gateway inside that environment holds the private key. Responses are protected the same way on the return path.
The separation this creates is the practical win — routing infrastructure can do its job without seeing request contents. Your application keeps calling the same API and request formats; the protection lives in the client and infrastructure layers. This echoes a theme I covered earlier in confidential GPUs for production inference: the serving stack changes, but the shape of your requests mostly doesn’t.
Attestation as enforcement, not paperwork
Isolation stops outsiders from reading memory, but it doesn’t answer whether the destination is running the configuration you expect. Remote attestation handles that: the environment produces signed measurements covering hardware, firmware, the OS environment, and application software. Evidence answers what is running; your policy answers is this acceptable to me.
What makes this more than an audit artifact is where verification happens. The client checks the destination against policy before sending protected data. If a software measurement changed or the GPU isn’t in the expected confidential mode, the request never leaves. Evidence is refreshed frequently, and each refresh is re-verified before protected traffic continues.
What it does and doesn’t cover
Cohere is candid about the remaining trusted base: the confidential hardware, firmware, the measured software, the reference measurements, and the attestation infrastructure itself. The claim is narrower and more honest — trust gets smaller, explicit, and measurable, not eliminated.
Inside the threat model, prompts, documents, responses, and model execution are protected during processing, and intermediaries — including cloud provider and Cohere operators — can’t read workload memory. Outside it: side-channel attacks, availability (an operator who can’t read a workload can still stop it), application-level issues like prompt injection, and observable metadata. Timing, message size, routing information, and the API credential and model identifier remain visible by design.
Cohere also draws the lines clearly: this is not a data-residency mechanism, not a replacement for DPAs and security reviews, and not homomorphic encryption — the model still operates on plaintext inside the protected boundary.
The takeaway for your stack
The pattern worth borrowing is verify-before-release: make attestation a gate your client enforces, not a report you file away. Model Vault is built on open-source components like Kata Containers and Confidential Containers, and Cohere says it intends to open-source the confidential inference stack — which matters, because verification is only as credible as your ability to inspect what you’re verifying. If regulated workloads are on your roadmap, this is a concrete architecture to evaluate against your own threat model rather than a trust-me pitch.