Skip to content
advanced Phase 15 · Drift, Collaboration, and Enterprise Capstone

Team Collaboration Guardrails

Adding ownership files, required checks, and access scopes to shared repositories.

1h
0 problems
Topic Progress 0%

Routing Reviews to the Team That Owns the Blast Radius

A pull request modifies modules/network/main.tf and envs/prod/network.tfvars. The first available reviewer approves it in eleven minutes — they own the billing module and cannot name a CIDR range. The merge is technically reviewed and effectively unreviewed. CODEOWNERS is the mechanism that ends this, and it works because it answers one question before the review request is even sent: who is accountable for the blast radius this directory represents?

# .github/CODEOWNERS — patterns behave like gitignore; the LAST match wins
*                             @platform-guild
/.github/                     @platform-guild @security-team
/modules/network/             @network-team
/modules/kubernetes/          @compute-team
/modules/database/            @data-platform
/environments/prod/           @sre-oncall
*.tfvars                      @platform-guild

Design the file around directories, not people's calendars. A directory pattern routes to the team that can actually evaluate the change: the network team reviewing a route table change will ask about BGP sessions, while a well-meaning engineer from another domain rubber-stamps it. Keep ownership granularity at the module level — one owner per module directory — because * fallback owners end up approving everything and become a queue. Two properties catch teams out: the last matching pattern wins, so a specific rule must come after the general one, and a file matching no pattern falls back to * (or to nobody, if you omit the fallback — decide deliberately).

Protect the file itself. Changes to CODEOWNERS must require approval from the platform team — otherwise anyone can add their handle to a pattern and then self-approve their own infrastructure changes. The same meta-rule applies to the workflow files and branch-protection settings: the guardrails are configuration, and whoever edits the guardrails owns the security of the system.

Practical details decide whether the file helps or annoys. When a pull request touches two owned directories — modules/network/ and envs/prod/ — both owner groups must approve; multiple owners raise the bar, they do not let one team vote twice. Name a secondary handle per directory (a team channel qualifies) so one person on leave cannot hold the merge queue hostage. And expect an adjustment period: engineers used to quick informal approvals will complain that routing adds latency. The counter is measurement — median time-to-first-review typically falls, because requests stop landing in inboxes that ignore them.

CODEOWNERS governs the repository; it does not govern Terraform Cloud. Keep the two layers distinct:

Layer Controls Example permission
CODEOWNERS + branch protection who may merge a change to version control network team must approve modules/network/ PRs
Terraform Cloud workspace permissions who may run and apply against an environment apply on prod workspaces restricted to SRE
Variable set scopes who can read production credentials prod credential set attached only to prod workspaces

A merged pull request and an applied plan are different events, and an organization where every merged change auto-applies to production has quietly deleted the second layer. Give reviewers a response expectation — one working day, escalation after — because ownership without responsiveness turns into a queue where PRs wait on a two-person team. And keep the changes themselves small; the next chapter covers the checks and merge rules that make that possible, starting with what must be green before any of it lands.

The interview framing is direct: "how do you ensure the right people review infrastructure changes?" Answer in three layers — directory ownership routes the review, required checks make the review deterministic, and workspace permissions keep merge rights separate from apply rights. Candidates who mention only code review have missed that production is not changed by merging.

Required Checks: Nothing Lands Without a Green Plan and Policy

Branch protection turns your pipeline from advice into law. With required status checks configured, a red or missing job makes merging physically impossible — no amount of deadline pressure overrides it. Configure it as code (the GitHub provider, gh api, or your platform's equivalent) so the guardrails themselves go through review:

# enforced via terraform's github_branch_protection or gh api
required_status_checks:
  strict: true
  contexts:
    - "static / fmt-validate"
    - "unit-plan / terraform test"
    - "policy / conftest"
    - "plan / staging"
    - "cost-estimate"
enforce_admins: true
required_pull_request_reviews:
  required_approving_review_count: 1

Each context is a contract with the CI workflow, and two failure modes come from forgetting that. A renamed job is a silent outage: plan / staging becomes plan (staging), branch protection still waits for the old name, no workflow reports it, and every pull request sits pending until someone opens the settings page. Version job names like an API, and update protection config in the same pull request that renames them. The second trap is strict: true being off: a check that ran against yesterday's tip proves nothing about today's main. Two merged PRs can interact in ways neither plan saw — require the branch to be up to date, or use a merge queue that re-plans the merged result, or your green checkmarks certify a codebase that no longer exists.

The cost-estimate context deserves a note: it behaves like any other required check. Attaching a monthly dollar delta to every plan is the mechanism from Cost Estimation on Plans, and making that context required means an accidental instance-class upgrade blocks the merge exactly the way a broken plan does — price becomes another red or green signal reviewers cannot skip.

The plan job itself needs the exit-code mapping from Layering Tests in the Pipeline — -detailed-exitcode returning 2 for "changes present" must count as success — and should post the diff where reviewers read it:

  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: terraform init -input=false
      - run: |
          terraform plan -input=false -out=tfplan -detailed-exitcode
          code=$?
          [ "$code" -eq 0 ] || [ "$code" -eq 2 ]
      - run: terraform show -json tfplan > plan.json
      - uses: actions/upload-artifact@v4
        with:
          name: plan-json
          path: plan.json

One security rule while posting comments: never check out pull-request code inside a pull_request_target workflow that holds a write-capable token. That combination is the classic GitHub supply-chain vulnerability; untrusted code runs with your credentials. Run the plan with a read-only token, or trigger comment posting from a separate trusted workflow that consumes the uploaded artifact. Treat a flaky required check with the same seriousness as a broken one — a job that fails one run in twenty trains engineers to click rerun until green, which is the exact behavior a required check was supposed to prevent. Fix the flake in the same week it appears.

The policy job runs on the plan output, not on the source:

terraform plan -input=false -out=tfplan -detailed-exitcode || [ $? -eq 2 ]
terraform show -json tfplan > plan.json
conftest test -p policy/ plan.json

This is where the standard from policy-as-code review becomes deterministic: the same rule evaluates all four hundred monthly pull requests with the same rigor, instead of depending on whichever reviewer was assigned.

Finally, give drift a place in the same dashboard. A scheduled job runs a refresh-only plan nightly and opens an issue — or fails a weekly required context — when out-of-band changes appear. Drift that surfaces in a separate report gets ignored; drift that competes with your merge gate for attention gets fixed. The merge queue also earns its keep here: by batching and serializing landings, it means your apply runs — serialized against the state lock described next — never queue behind an engineer's forgotten laptop run.

Credential Scopes, State Locks, and the Case for Small Branches

Two engineers merge clean-looking pull requests an hour apart. The first apply succeeds; the second fails with Error acquiring the state lock: Lock Info: ID: ... — or, in the worst configuration, succeeds concurrently and overwrites the first's state updates. Neither engineer did anything unusual. The guardrails that prevent this are variable scoping, locking discipline, and merge hygiene, and they are the last collaboration layer before governance.

Credential scoping separates who can see what by scope, not by trust:

Scope Visibility Example contents
terraform (plain) shown in run logs and diffs allowed_zones = ["us-east-1a"], instance_count = 3
env (sensitive) hidden after write, redacted in logs API tokens, DB passwords
Workspace-level variables one workspace only environment-specific overrides
Variable set → workspaces list only the attached workspaces prod credentials attached to prod workspaces, nothing else

The rule that matters: production credentials exist only in variable sets attached to production workspaces. An engineer with write on a dev workspace must be unable to read prod secrets, and a compromised CI job for dev must have no path to prod credentials. Prefer short-lived federation over static secrets entirely: OIDC tokens scoped by subject claim — repo:org/repo:ref:refs/heads/main — mean the CI role is only assumable from your repository's main branch, with credentials that expire in an hour instead of leaking forever.

State locking is the concurrency control you already have and probably misread. The backend lock (a DynamoDB LockID entry for S3, a blob lease for Azure) makes the second concurrent writer fail fast instead of interleaving. Plans also acquire the lock by default while refreshing — which is why a read-only reporting job against a busy environment runs with -lock=false, and why your CI should serialize applies through the platform's run queue (one active run per workspace) rather than letting four pipelines race. When a run dies mid-apply, the lock survives its owner: force-unlock only after confirming the owning run was cancelled in the UI, because force-unlocking while an apply is genuinely alive invites two writers to the same state — the one failure mode that silently corrupts everything downstream.

Merge hygiene is the cultural half, and infrastructure repos punish violations harder than application repos. In application code, a long-lived branch conflicts in source files. In Terraform, the branch conflicts in source and goes stale against state: a branch written Monday plans clean, two pull requests merge Tuesday and Wednesday, and by Friday your plan output describes a world that no longer exists — while merge conflicts in the .tf files are only half the problem, since resolving them does not refresh your plan. The practices that keep this under control:

  • Target merge within one working day; re-plan on every rebase, and treat a stale plan as a failed review.
  • Keep module changes additive — adding an attribute or a resource is safe to land under traffic; restructuring addresses belongs in its own pull request with a moved block.
  • Split "change the module" from "adopt the new module version" so a consumer's upgrade PR stays a one-line version bump.
  • Cap diffs near 400 changed lines; past that, reviewer attention collapses and the required approval becomes a formality.

Scoping keeps credentials away from the wrong eyes, locking keeps two honest runs from corrupting each other, and small merges keep plans meaningful. Together they form the everyday layer of the practice — Enterprise IaC Operating Model turns these pieces into an organization.

Quiz

1. An engineer edits .github/CODEOWNERS to add their own team to /modules/network/ and then approves their own network change. Which control prevents this?

Question 1 options

2. A job is renamed from `plan / staging` to `plan (staging)` and every pull request now waits forever on a check that never reports. What happened?

Question 2 options

3. How should production credentials be scoped in a shared Terraform platform?

Question 3 options

4. Two runs start against the same remote state at the same moment. What outcome do you want?

Question 4 options

Flashcards

Question

How do CODEOWNERS patterns resolve, and what must protect the file itself?

Answer

Patterns behave like gitignore — the LAST matching pattern wins, with * as fallback — and directory rules route each pull request to the team accountable for that blast radius. The file itself must require platform-team approval to change, otherwise anyone can add their handle and self-approve infrastructure.

Question

What do strict required status checks buy you, and what breaks when a job is renamed?

Answer

strict: true (or a merge queue) forces the plan to run against the current tip, so a green check certifies the code that will actually merge rather than yesterday's base. Renaming a job orphans the old context name in branch protection, leaving every pull request pending until the settings are edited.

Question

Difference between terraform-scoped and env-scoped variables, and where production credentials go?

Answer

terraform variables are plain and appear in run logs; env variables are sensitive, hidden after write. Production credentials belong in a sensitive variable set attached only to production workspaces — ideally replaced entirely by short-lived OIDC federation scoped to repo and branch.

Question

What happens on a concurrent second write to locked state, and when is force-unlock acceptable?

Answer

The second writer fails immediately with Error acquiring the state lock and the Lock Info block. Force-unlock is acceptable only after confirming the owning run was cancelled — force-unlocking while an apply is genuinely alive invites two writers to one state file, the failure mode that silently corrupts everything downstream.

Question

Why do long-lived branches hurt Terraform repositories more than application repositories?

Answer

They conflict in source files like any branch, but they also go stale against state: two merges change real resources between your branch and the tip, so your plan output describes a world that no longer exists. Target merges within a working day and re-plan on every rebase.

Revision Notes

Key Takeaways

  • 1. CODEOWNERS routes each pull request to the team that can judge its blast radius; directory patterns with last-match-wins semantics, protected by a platform-approval rule for the guardrail files themselves
  • 2. Required checks are literal context strings under strict up-to-date enforcement — a green status must certify the code that will actually merge, and job renames are contract changes
  • 3. Production credentials live only in sensitive variable sets attached to production workspaces; OIDC federation shrinks exposure to an hour and a branch
  • 4. Merge rights and apply rights are separate layers: repo review gates what lands in git, workspace permissions gate what reaches an environment
  • 5. Small, frequently merged branches keep plans meaningful — in infrastructure, stale plans against moved state are the silent half of every merge conflict

Interview Tips

  • • "How do you make sure the right people review infrastructure changes?" — answer in three layers: CODEOWNERS routes by directory, required checks make review deterministic, workspace permissions separate merging from applying
  • • "A required check stopped reporting and merges are frozen — what do you do?" — explain that contexts are literal strings, the rename orphaned the old one, and the fix is updating protection settings; then say you version job names to prevent recurrence
  • • "How do you keep production credentials safe in CI?" — sensitive variable set scoped to prod workspaces, OIDC with repo-and-branch subject claims, no static keys; add that a dev compromise must have no path to prod
  • • "Two applies hit the state at once — what happens?" — second fails on the lock, CI queues one run per workspace, and force-unlock is a last resort verified against a cancelled run — show you know the difference between a lock and a merge

Cheat Sheet

Team Collaboration Guardrails

CODEOWNERS (last match wins):

*                      @platform-guild
/.github/              @platform-guild @security-team
/modules/network/      @network-team
/environments/prod/    @sre-oncall

Protect the guardrails: changes to CODEOWNERS, workflows, and branch-protection settings require platform approval.

Required checks (enforced as code):

required_status_checks:
  strict: true          # plan must run against the current tip
  contexts:
    - "static / fmt-validate"
    - "unit-plan / terraform test"
    - "policy / conftest"
    - "plan / staging"
    - "cost-estimate"
enforce_admins: true
required_pull_request_reviews:
  required_approving_review_count: 1

Job names are a contract — rename and update protection in the same PR. Never check out PR code in a pull_request_target workflow holding a write token.

Credential scopes: plain terraform vars visible in logs → sensitive env vars hidden → variable set attached only to prod workspaces → OIDC subject repo:org/repo:ref:refs/heads/main for short-lived tokens.

State locking: DynamoDB LockID / blob lease makes the second writer fail fast; CI runs one apply per workspace; force-unlock only after confirming the owning run was cancelled.

Merge hygiene: merge within a day, re-plan on rebase, keep module changes additive (use moved for renames), split module change from version adoption, cap diffs ~400 lines.

Continue Learning