Promoting Changes Across Environments

Directory Layouts per Environment gave every tier its own root, backend, and CI path filter — the structure of isolation. What remains is the ordering: a change to production infrastructure should never be seen for the first time in production. That is the entire argument for ordering: dev is where the plan is read line by line, where provider quirks surface, and where a mistake costs a teardown instead of an incident.

The sequence a mature team runs for every merged change:

1. Plan and apply dev from the merged commit.
2. Read the result — not just the exit code.
3. Plan and apply staging, compare its diff against dev's.
4. Plan production, attach the plan to a gated approval, apply it.

Hotfixes take the same road. Skipping dev because a change is urgent is how a "small fix" acquires an unexpected replacement — the ordering is cheap precisely because each tier costs minutes, not days, and it is the review of tier two and tier three that catches what tier one missed.

Dev gives you three things no static analysis can. First, API reality: IAM policy propagation, eventual consistency on tags, account service quotas — all of them surface as a failed apply in dev rather than at 2 a.m. in prod. Second, the diff, computed against real state: what you review comes from an environment shaped like production, so replacement lines and resource counts mean something. Third, drift visibility: if dev's plan is full of out-of-band changes, your configuration no longer describes reality — and that same lie would have been applied to production.

What you actually read in the dev diff, in order:

1. must be replaced lines first — a replacement is downtime, and it is the line reviewers must argue about.
2. New exposures: 0.0.0.0/0 ingress, public buckets, wildcard IAM principals.
3. Count and index changes — a falling count means deletions; count.index shifting means a renumbering cascade.
4. Sensitive values written into the plan — anything that will sit in an artifact store afterwards.
5. Anything the pull request never mentioned — that is drift, not code, and it changes what you approve.

Produce and consume a saved plan so the reviewed artifact is the executed artifact:

terraform -chdir=envs/dev plan -input=false -var-file=dev.tfvars -out=tfplan
terraform -chdir=envs/dev show -json tfplan > tfplan.json   # machine-readable record
terraform -chdir=envs/dev apply -input=false tfplan         # executes exactly that plan

terraform apply tfplan does not recompute anything: no prompt, no drift re-check, no surprise additions between review and apply. Pair it with -detailed-exitcode in the check step — exit 0 means no changes, 2 means changes present, 1 means error — so the pipeline can report "nothing to do" distinctly from "failed":

terraform -chdir=envs/dev plan -input=false -detailed-exitcode -out=tfplan
case $? in
  0) echo "no changes" ;;
  2) echo "changes to review" ;;
  1) exit 1 ;;
esac

Be honest about what dev rehearsal does not prove. Dev is the tier people break things in: console tweaks, one-off scripts, and kubectl edit leave it drifted, so its apply validates mechanics, not fidelity. Staging — kept clean, changed only through the pipeline — is the faithful mirror, and it is where "does this diff match what dev did?" is worth asking before production sees anything.

The ordering rule survives every tooling change: failures land where they are cheap, and the same versioned configuration walks each environment in sequence — which is what the saved plan artifact makes auditable, in the next chapter.

A saved plan file is not a document about your infrastructure; it is a frozen decision bound to three things: the exact state it was computed against (its serial and lineage), the configuration that produced it, and the provider binaries that evaluated it. Understanding that binding clears up the most common promotion mistake — the belief that you can compute a plan once in dev and reuse it everywhere.

You cannot, and Terraform enforces it. Applying a plan built against one state against another state fails outright — Error: Saved plan is stale when the state has moved since the plan was written, and a hard refusal when the configuration differs. This is a feature: the plan that gates production must be computed against production's own state, inside the pipeline, immediately before the gated apply.

That gives the plan file two distinct jobs across the promotion:

The distinction matters in review: diffing dev's JSON against production's JSON is how you notice that production carries extra resources or drift before approving, rather than discovering it in the apply log afterwards.

The pipeline shape follows directly: a plan job that generates and uploads the artifacts, and a separate apply job — gated by a required reviewer — that downloads that artifact and runs apply:

# plan job (runs on every merge, uploads artifacts)
terraform -chdir=envs/prod init -input=false
terraform -chdir=envs/prod plan -input=false -out=tfplan
terraform -chdir=envs/prod show -json tfplan > tfplan.json
# …upload tfplan and tfplan.json as this run's artifacts…

# apply job (runs after approval, same artifacts, no re-plan)
terraform -chdir=envs/prod apply -input=false -auto-approve tfplan

show -json is what makes the artifact reviewable by machines and by auditors. Feed it to Open Policy Agent or Conftest to fail on rules a human will miss in a 400-line diff:

deny[msg] {
  resource := input.resource_changes[_]
  resource.change.actions == ["delete"]
  resource.type == "aws_db_instance"
  msg := sprintf("refusing to delete database %s", [resource.address])
}

The JSON also drives the pull-request comment bot, so reviewers read the same text the apply will execute, and it is retained with the run as the answer to "what exactly did we approve in March?"

Reading a saved plan without applying it is a skill worth having in review:

terraform -chdir=envs/prod show tfplan                  # human-readable diff
terraform -chdir=envs/prod show -json tfplan | jq -r \
  '.resource_changes[] | select(.change.actions != ["no-op"])
   | "\(.change.actions|join(",")) \(.address)"'

The second command is what reviewers actually want: one line per changed resource — delete aws_db_instance.web, create aws_s3_bucket.logs — small enough for a pull-request comment, and directly diffable against the JSON produced in dev for the same commit.

Store both files as artifacts of the run, not of the job: they are the record of what was approved, and alongside the state's version history they make a promotion reconstructable months later — this plan, this state serial, this approver, this outcome.

One more property earns its keep: if anything writes to production state between the plan job and the apply job, the apply rejects the stale plan and stops. State moving under a reviewed plan is a failure, not a rounding error — somebody changed production outside the pipeline, and the gate just told you so.

The workflow underpinning all of this: a pull request proves the change plans cleanly, the merge starts a promotion, and every environment receives the same commit in order — dev, staging, production — with a human gate in front of the last step.

on:
  pull_request:
  push:
    branches: [main]

jobs:
  plan:                                  # PR gate: plan every changed root
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: terraform -chdir=envs/dev init -input=false
      - run: terraform -chdir=envs/dev plan -input=false -var-file=dev.tfvars -out=tfplan
      - run: terraform -chdir=envs/dev show -json tfplan > tfplan.json
      - run: gh pr comment ${{ github.event.pull_request.number }} --body-file tfplan.json

  apply-dev:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: dev
    steps:
      - uses: actions/checkout@v4
      - run: terraform -chdir=envs/dev init -input=false
      - run: terraform -chdir=envs/dev apply -input=false -auto-approve

  apply-staging:
    needs: [apply-dev]                   # dev must succeed first
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - run: terraform -chdir=envs/staging init -input=false
      - run: terraform -chdir=envs/staging apply -input=false -auto-approve

  apply-prod:
    needs: [apply-staging]               # staging success precedes production
    runs-on: ubuntu-latest
    environment: production               # required reviewers = the approval gate
    steps:
      - uses: actions/checkout@v4
      - run: terraform -chdir=envs/prod init -input=false
      - run: terraform -chdir=envs/prod plan -input=false -var-file=prod.tfvars -out=tfplan
      - run: terraform -chdir=envs/prod apply -input=false -auto-approve tfplan

Three design decisions carry the weight. needs: encodes order — production cannot start until dev and staging applied, and each stage checks out the same SHA, so the code that passed review is the code that ships; no re-planning from a dirty laptop. GitHub environments supply the gate — environment: production with required reviewers turns approval into an auditable event instead of a Slack message. Prod re-plans against prod state and applies that plan, as established earlier; the staging result is evidence, not a substitute.

Now blast radius. Production often owns resources the lower tiers do not: a larger cluster, a KMS key with a customer-managed policy, a public WAF. Two ways to express that, and only one holds up:

• Separate configuration paths (right): the resources live under envs/prod/ — or in a module only that root calls. They plan, review, and destroy like everything else.
• terraform plan -target=... (stopgap): targeting reshapes the graph — dependencies get pulled in, sibling resources silently drop out of the plan — so the result is a partial view of the world that happily writes partial state. Reserve -target for surgical repairs you intend to clean up, never for routine promotion.

Close the loop on access: production credentials should exist only inside apply-prod, issued via OIDC to a role that laptops cannot assume. Then the pipeline is not a convention but the sole path — a local terraform apply fails authentication, which is exactly the blast-radius control interviews mean when they ask how you protect production. Rollback becomes boring: re-run the pipeline pinned to the previous commit; state versioning (S3 versioning on the bucket) plus the retained tfplan.json artifacts show what changed and give you terraform state pull snapshots to recover from.

One scheduled job completes the model: drift detection. A nightly terraform plan -detailed-exitcode against production, with no commit to justify any change, turns 'someone edited the console' into a red build by morning instead of a surprise at the next deploy. Plan detects, the promotion applies — the two halves of the workflow keep each other honest.

With ordering and gating settled, the remaining questions move back into the configuration — starting with how one root scales a resource from a single dev instance to six production instances without another hand-written conditional, which is where The count Meta-argument picks up.