Secrets and Key Management Design

The incident report writes itself. An AWS access key was committed to a private repo seven months ago. Deleted from the working tree the same afternoon — but still in git history, still in the CI cache that mirrors the repo, still in the Docker image that was built from it and pushed to a registry three teams now pull from. External scanners found the key at 02:14; the first API call from an unrecognized ASN came at 02:55. Forty-one minutes of unmonitored enumeration was, by the standards of this industry, lucky. Secrets management design is the discipline of making that story impossible to tell, and it starts by mapping where secrets actually live:

The container-image line deserves emphasis because it is the most-repeated mistake: the image is immutable and propagates to every environment at once — anything baked in is leaked everywhere at once. A build-arg secret is in the image config; a COPY .env is in a layer; both survive even when the CI log is scrubbed.

The replacement architecture is a managed vault — HashiCorp Vault, AWS Secrets Manager, or a cloud secret manager — with four properties that static files never have. Access is authenticated by workload identity (the service's role or SPIFFE identity), not by possession of the vault token itself, so there is no second secret to leak. Every read is policy-authorized and audit-logged, so 'who fetched the payments DB password last week' is a query, not an investigation. Secrets rotate underneath consumers. And the strongest version is dynamic: the database backend mints a per-connect role (svc-order-20261014-0231, TTL one hour, revocable as a lease) instead of handing out one shared credential. A leaked dynamic secret expires in an hour; a leaked static password is valid until someone notices.

Consumption patterns finish the design: fetch at startup or via a sidecar/agent on an interval so rotation needs no redeploy, cache in memory with a short TTL rather than writing to disk, and fail closed — a vault outage should degrade reads that need secrets, not silently fall back to a hard-coded copy. That fallback, offered as a resilience feature, is how most vault migrations quietly reinvent the problem they were meant to solve. Failing closed also forces the honest design question — if the vault cannot be reached, should this path serve traffic at all — and the answer is usually a degradation that keeps cached-credential reads up while any request that would mint new secrets waits.

Encryption keys are secrets with a twist: every encrypted byte depends on them, so rotation is a data-plane operation, not just a credential swap. The design that makes rotation tractable is envelope encryption. You ask the KMS to encrypt one small data key under a customer-managed key (the key-encryption key); the service encrypts the actual payload locally with that data key; and the ciphertext is stored together with the key ARN and version. Only the wrapping key lives in the HSM-backed service — millions of objects, one rotatable KEK.

resource "aws_kms_key" "orders" {
  description             = "order-service PII and token wrapping"
  enable_key_rotation     = true        # new backing material annually
  deletion_window_in_days = 30
}

resource "aws_kms_key_policy" "orders" {
  key_id = aws_kms_key.orders.key_id
  # key policy is authoritative: it can deny what IAM allows
  statement {
    sid    = "allow-orders-role-encrypt"
    effect = "Allow"
    principals { identifiers = [aws_iam_role.orders.arn] }
    actions   = ["kms:Encrypt", "kms:GenerateDataKey"]
    resources = ["*"]
  }
}

Caching data keys is the standard performance lever — one KMS call protects a burst of writes, and at roughly a dollar per 100,000 calls the cost argument is real — but state the trade-off out loud: a cached data key extends the window in which process memory holds working key material. Bound it with a TTL (minutes, not hours) and make the compromise story 'dump this pod's memory' rather than 'dump the KMS.'

Then be precise about what auto-rotation does. KMS annual rotation issues new backing material under the same key ID; old versions remain available to decrypt existing ciphertext. It refreshes the wrapping key. Data keys you generated earlier stay wrapped by their original version until the data itself is re-encrypted — lazily on write, or by a bulk re-encryption job. Conflating the two is the classic gap: 'rotation is on' heard as 'old ciphertext is re-keyed' when it is not.

The full menu, with downtime characteristics:

Two rotation classes get forgotten until they fail. TLS certificates now renew on 90-day cycles through ACME automation, and the classic incident is the automation that stopped renewing silently six weeks earlier — probe certificate expiry as a monitored signal, never a calendar reminder. Signing keys and long-lived API tokens have no auto-rotation at all; they need a named owner, an explicit cadence, and a consumer contract that tolerates overlap, because the rotation you cannot rehearse is the rotation you cannot safely run.

Zero-downtime credential rotation has one canonical shape: add before you remove. Create a second credential, roll consumers to fetch and use both (or flip via config), verify, then retire the first. The dual-database-user version creates app_ro_v2, grants it identically, switches the connection string, waits a full TTL cycle, then drops the old user. The rotation that breaks production is always the same one: delete-then-recreate on a Friday, or rotate a key that half the fleet still holds cached without a version-tolerant consumer. Design consumers to select key material by version (a kid in the header, the key ARN beside the ciphertext) before the first rotation, not after it fails at 3 a.m.

Why this whole apparatus is worth it, in one sentence an interviewer will accept: rotation is what turns a leaked credential from a permanent liability into an exposure window measured in days.

The last design question is the one that decides how bad a single compromise gets: if one credential leaks today, what can it reach? The principle is separation of duties between key material and the data it protects — if one leaked principal can both fetch ciphertext and decrypt it, you have one control wearing two costumes.

Separation takes several concrete forms. Encrypt and decrypt are different permissions: ingestion pipelines commonly hold kms:Encrypt only, while a narrow set of readers holds kms:Decrypt. Key administration is a different job from data administration: the team that can rotate or disable a key should not be the team that can query the table, and neither should be the CI role. Placement separates too: keys in a security account, data in workload accounts, cross-account key policies naming exactly which roles may use the key — so a foothold in one account does not inherit the keys of another. And key policies are authoritative in a way people forget: a key policy can deny a principal even when its IAM policy allows, which makes it the inner fence that survives IAM surprises.

Granularity is a genuine trade-off, not a purity test:

Pick boundaries at service and data-classity lines — PII in its own key, tokens in another — and reserve per-tenant keys for the compliance or contractual cases that actually demand them.

The incident playbook then becomes concrete. Detect: every KMS call logs principal, action, and key to the audit log; alarm on Decrypt bursts, first-time principals, and off-hours use. Contain: revoke the offending principal's credentials first — disabling the key itself stops every legitimate consumer too, so it is the last lever, not the first. Rotate: new key version, re-encrypt affected data lazily or in bulk, verify consumers select the new version. Audit: reconstruct scope from the access log — which ciphertexts, which principals, how long. There is a final, hard option: delete a key version and the ciphertext it wrapped becomes permanently unreadable — crypto-shredding. It is real deletion without touching a single storage row, and it collides head-on with legal holds and backup-retention schedules; decide the policy before the incident, not during it.

The interview close that connects this topic to the rest of the phase: you designed separation so a leaked CI role yields one service's encrypt-only capability; per-request policy keeps the role from ever reaching other services; and answering 'who decrypted what, when' requires shipping and alerting on those audit logs — which is precisely the observability machinery Logs, Metrics, and Traces takes next.

Review the credential incident with the owning teams, update the rotation playbook, and test revocation before closing follow-up actions.