Build Once, Promote Many
Build Once, Promote Many
Promotion means the same artifact moves dev → staging → prod — only the environment definition changes. Rebuilding per environment ("works in staging") introduces drift: different base images, dependency versions, and compile results.
In practice: CI builds once and records registry/app@sha256:abc123. Each environment's config references that digest. Promotion is then a one-line change to prod's config, reviewed like any other change, with a full evidence trail (who approved, which tests passed for this exact digest). Retagging mutable aliases (prod → sha) is discouraged — read by digest, always.
Making Promotion a Git Diff
Making Promotion a Git Diff
With Kustomize overlays, promotion is an image field update:
deploy/
├── base/deployment.yaml
├── overlays/staging/kustomization.yaml
└── overlays/prod/kustomization.yaml # promoted line lives here
cd deploy/overlays/prod
kustomize edit set image api=$REG/api@sha256:$DIGEST
git commit -am "promote: api $DIGEST to prod"
The CI gate for that PR: apply to staging first, run e2e, verify signatures (cosign verify $REG/api@sha256:$DIGEST), then auto-open the promotion PR. Argo CD/Flux syncs prod after merge; Flagger can stage the traffic shift 0→100% with metric-based rollback.
Pitfalls and Interview Angles
Pitfalls and Interview Angles
Retagging breaks audit. Overwriting v1.4 with new bits invalidates every environment that referenced the tag — treat the registry as write-once and let digests identify builds.
Rebuild-per-env drift. Staging testing a different artifact than prod runs is the classic "passed QA, broke Friday" failure; enforce one digest across environments.
Verification after start. Pulling before verifying puts an untrusted image on the node; enforce digest+signature admission (Kyverno/OPA Gatekeeper) so unverified images never schedule.
Skipping provenance. An SBOM answers "what is in this image" during an incident; generate at build, attach to the digest, reference it in the promotion PR.
Interview angle: draw the pipeline (build once → evidence → staged promotion as PRs → signed gate → progressive rollout) and state what stops a tampered image.
Quiz
1. What does "build once, deploy many" prevent?
2. In a Kustomize setup, what does promotion to prod actually change?
3. Why verify an image signature at admission rather than only in CI?
Flashcards
Question
Promotion
Click to reveal answer
Answer
Moving the same immutable digest through environments by changing config references, never rebuilding.
Question
kustomize edit set image
Click to reveal answer
Answer
CLI that rewrites a kustomization.yaml images: entry — the standard promotion diff in prod overlay.
Question
cosign verify
Click to reveal answer
Answer
Checks a signature attached to an image digest, proving provenance before deploy.
Question
Admission gate
Click to reveal answer
Answer
Kyverno/OPA policy requiring digest-pinned, signed images before pods are admitted.
Revision Notes
Key Takeaways
- 1. One build, one digest, promoted by editing environment overlays in Git
- 2. Evidence (tests, SBOM, signature) attaches to the digest and gates each promotion PR
- 3. Admission policies backstop the pipeline against anything that slips past CI
Interview Tips
- • Explain build-once/promote-many and show the exact Git diff a promotion produces
- • Describe where verification happens twice (CI and admission) and why both layers exist
Cheat Sheet
Build once → digest → test/scan/sign → promote by editing overlay image line (kustomize edit set image) → PR gate → CD sync → canary. Enforce digest+signature at admission; never retag.