Keys Out of the Cluster: Vault's Kubernetes KMS Beta and What It Actually Buys You

HashiCorp has put a KMS v2 plugin for Kubernetes into public beta, moving the key encryption key that protects etcd out of the cluster and into Vault Enterprise. It closes a real gap for self-managed clusters — but it fixes a narrower threat than most people will assume, and it couples your API server availability to Vault in ways worth rehearsing before production.

The demo that ends most arguments about etcd

If you ever need to win a budget conversation about cluster hardening, do this in front of the people holding the budget: take a lab cluster, shell onto a control plane node, and run etcdctl get against a secret's registry path. Out comes the object, and out comes the payload — base64, not encryption. Kubernetes secrets are not secret at rest by default. Every backup tarball, every snapshot sitting on an NFS share, every decommissioned control plane disk that went out on a pallet is a plaintext credential store.

The standard fix has been an EncryptionConfiguration with an aescbc or aesgcm provider — which works, and which stores the key in a file on the same node as the thing it protects. That has always been the awkward bit. You've encrypted etcd with a key that anyone who can read the control plane filesystem can also read. It satisfies the audit line item. It doesn't satisfy anyone who thinks about it for more than a minute.

What the KMS v2 interface actually changed

KMS v2 went GA in Kubernetes 1.29 and it is a genuinely better design than v1, which has been deprecated since 1.28. Under v2 the API server generates a data encryption key per object, encrypts the object with it, and asks the external plugin to encrypt only the DEK. The wrapped DEK travels with the ciphertext. The API server caches DEKs, so you are not making a round trip to the key manager on every read, and the plugin reports a key ID that the API server polls — when the ID changes, new writes get wrapped under the new key without a restart.

That's the mechanism HashiCorp's new plugin plugs into, with the key encryption key living in Vault Enterprise rather than on the node. The value proposition is one sentence long: the people who administer the cluster no longer hold the key that protects the cluster's data at rest. That's separation of duties, expressed in cryptography rather than in a RACI chart, and it is worth having.

Be honest about the threat model

Here is where I'd push back on the enthusiasm this will generate. Encrypting etcd with an external KEK defends against offline access to etcd data: stolen snapshots, backups landing in the wrong bucket, disks leaving the building, a compromised backup agent, a supply chain path into your storage tier. Those are real. A cluster backup job writing to a share with far broader read access than anyone realised is exactly the kind of finding this control genuinely closes.

What it does not do is stop the attack path that most incidents actually use. An attacker who has cluster-admin, or a service account with secrets read across namespaces, or a compromised operator with a wide ClusterRole, asks the API server for the secret and the API server obligingly decrypts it for them. That's the front door. Envelope encryption is a lock on the basement window. Fit it, by all means — but do not let it be the reason you defer RBAC tightening, admission control or audit log review, because those address the traffic that is far more likely to matter.

The coupling you are signing up for

KMS plugins talk to the API server over a Unix domain socket on the control plane host. That means the plugin runs adjacent to the API server — typically a static pod — and it means your API server's ability to serve encrypted resources now depends on that process, and transitively on Vault being reachable and unsealed. If the plugin can't reach Vault, writes to encrypted resources fail. Reads may survive on cached DEKs for a while. "For a while" is not a design principle.

The restore path deserves particular attention. An etcd snapshot taken under external KEK protection is only restorable if the Vault key that wrapped those DEKs still exists and is reachable. Your disaster recovery runbook has just acquired a dependency on a system that may itself be running on the cluster you are trying to restore, or in a datacentre you have just lost. Circular dependencies in DR are how a four-hour outage becomes a two-day one. The classic version is a backup catalogue stored on the SAN it was backing up, and the logic here is identical.

Before this goes anywhere near a production control plane, I'd want evidence for each of these:

API server behaviour when Vault is sealed, unreachable, and slow — measured separately, because the failure modes differ
Latency added to secret-heavy operations, particularly pod startup storms after a node drain
The full rotation procedure, including rewriting existing objects so old key versions can be retired
A restore of an etcd snapshot into a clean cluster with only the documented Vault dependencies available
Where the Vault namespace and policy for the KEK sits, and which humans can delete it

Where it fits against the alternatives

If you run managed clusters, your cloud provider already offers this — AWS, Azure and Google all support external KMS for the cluster's encryption provider. The gap this fills is self-managed Kubernetes, on-prem and hybrid, where the choice until now was a local key file or building your own plugin. For anyone running clusters on their own tin with a Vault estate already in place, that is a real gap closed. HashiCorp's own description is consistent with that: the plugin needs you to modify the kube-apiserver manifest and the encryption configuration, which rules out most fully managed control planes.

The other option is architectural rather than cryptographic: don't put the secret in etcd at all. The Secrets Store CSI driver and Vault Agent injection deliver material to the pod directly from Vault, never creating a Kubernetes Secret object. That removes the problem instead of encrypting it. It's more invasive to adopt, and it doesn't help with the secrets you can't avoid — service account tokens, bootstrap credentials — which is exactly why etcd encryption stays on the list regardless.

My posture for now

Public beta means lab only, and it means asking about the support path before anyone builds a compliance narrative on top of it. There's also a procurement dimension worth raising early: this is an Enterprise feature, HashiCorp's licensing moved to BUSL in 2023, and the company now sits inside IBM. None of that is a reason to walk away, but if your architecture review board is going to bake a Vault dependency into the availability of every control plane you run, that conversation belongs at the start rather than at renewal.

Test it, measure the failure modes, write the restore runbook — and keep spending the majority of your effort on who can call the API, because that's where the credentials actually leave.

Source: HashiCorp Ships Public Beta of Vault Kubernetes Key Management