Secret Encryption at Rest

Why Encryption at Rest

RBAC governs who may call the API; it says nothing about bytes on disk. Anyone with filesystem access to etcd — a storage admin, a stolen backup volume, a nosy operator — can read every Secret as base64, TLS keys included.

Kubernetes adds an encryption provider on the API server: objects are encrypted with a data encryption key before etcd sees them, and decrypted on read. Two providers ship in-tree:

• aescbc (or aesgcm) — symmetric encryption with keys supplied in a local file.
• kms — an external KMS or plugin holds the root key, so etcd storage never contains it.

The setup is a YAML file passed to kube-apiserver as --encryption-provider-config.

Configuring and Migrating

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded 32-byte key>
  - identity: {}

Provider order matters: the first entry encrypts new writes, and the list is tried in order on read. identity: {} must remain so pre-existing plaintext objects still open.

Existing Secrets are not rewritten automatically. Force it by reading and re-saving every one:

kubectl get secrets -A -o json | kubectl replace -f -

Until that sweep runs, old objects remain cleartext in etcd even though new writes are encrypted.

Pitfalls and Key Rotation

Pitfall 1 — declaring victory before the sweep. Setting the provider affects new writes only. Verify by reading a raw etcd key: encrypted values begin with k8s:enc:aescbc:v1:key1:; anything else is still plaintext.

Pitfall 2 — losing the key. A lost data encryption key means unreadable Secrets. Back up key material with the same discipline as etcd snapshots, and remember identity last provides an emergency read path for plaintext.

Interview-ready: state that RBAC is not encryption, contrast aescbc with KMS (the root key never enters the API server process), explain the re-encrypt sweep, and walk rotation: add key2 as the first key, restart API servers, re-run the sweep, then remove key1.