Skip to content
advanced Phase 15 · Drift, Collaboration, and Enterprise Capstone

Enterprise IaC Operating Model

Combining modules, pipelines, policy, state, and cost controls into a lasting practice.

1h 30m
0 problems
Topic Progress 0%

Who Publishes, Who Approves, Who Owns State: The Operating Model

You are the first platform engineer at a company of two hundred. Ten teams already keep their own .tf files; four of them applied directly from laptops. The question "who owns this?" has no answer for any resource in production. Before tooling, before pipelines, the enterprise practice starts with a written operating model — because every infrastructure change has three owners (module author, reviewer, state owner), and enterprise failure is when one of the three is assumed by nobody. This topic builds on Team Collaboration Guardrails; where that topic gave a repository its guardrails, this one gives the organization its map.

Write the ownership table first, one row per concern:

Concern Publishes Approves Owns operationally
Base modules & registry platform / IaC guild module RFCs (design review) module versions, deprecation windows
State backends & locking platform any backend change lock recovery, state access audit
Pipelines & policy packs platform + security workflow and policy changes runner spend, policy violation trend
Environment composition each service team their own PRs (via CODEOWNERS) their resources after apply
Production applies service team requests SRE on-call (apply permission) incident response, rollback
Cost standards & thresholds FinOps escalations beyond budget monthly cost report

Read the table as a workflow: a service team composes an environment from published modules, the platform's required checks and policy evaluate it, and production apply rights sit with the people holding the pager. When a role in that flow has no name attached, write one — an unowned cell in this table is where your next incident comes from.

Then define the module lifecycle, because "who publishes modules" is really a versioning question:

module "network" {
  source  = "app.terraform.io/corp/network/aws"
  version = "~> 2.3"   # pinned to minor; upgrades arrive as reviewed PRs
}

Modules follow semantic versioning with a published changelog. A breaking change — a renamed variable, a changed resource address — requires a major bump, migration notes, and a deprecation window long enough for every consumer to move (two quarters is a workable default). Consumers upgrade through pull requests, which ride the exact same required checks as any other change; a bot that opens version-bump PRs keeps the fleet from drifting two majors behind. Environments compose the same pinned module versions: envs/dev, envs/staging, and envs/prod are directories, not forks, and promotion is bumping a version in staging and then in prod — never editing prod's module call directly.

State ownership belongs to the platform team, unambiguously. Backends live in a platform-controlled account; no human holds direct read access to production state; pipelines reach it through short-lived federated credentials; and access logs go to a bucket no pipeline can write. State is the one artifact where "shared for convenience" is a security incident waiting to happen, since it lists every resource and often carries sensitive outputs.

Interviewers ask this as "how would you roll out Terraform across a large org?" Lead with the ownership map, then the module contracts, then the pipeline — candidates who start with a tool comparison have answered a different question. With ownership defined, the next chapter installs the two controls that enforce it continuously: policy as code and hardened state.

Policy as Code and Hardened State: The Two Controls Everything Rests On

Your team merges roughly four hundred pull requests a month. Ask reviewers to remember three tagging rules, two encryption standards, and a forbidden instance list, and compliance will be probabilistic — maybe eighty percent, varying by who reviewed which directory that week. Policy as code evaluates the plan with the same rule on every pull request, in seconds, and the answer does not depend on reviewer seniority:

terraform plan -input=false -out=tfplan
terraform show -json tfplan > plan.json
conftest test -p policy/ plan.json
package terraform.tags

deny[msg] {
  rc := input.resource_changes[_]
  rc.mode == "managed"
  rc.change.actions[_] == "create"
  not rc.change.after.tags["Environment"]
  msg := sprintf("%s must carry an Environment tag", [rc.address])
}

Note what the rule reads: resource_changes in the plan JSON, so the decision happens before anything exists. Watch for computed attributes — provider-generated fields show up under after_unknown, and a rule asserting on them fails for every resource. Sentinel inside Terraform Cloud, OPA/conftest in your own CI, and the platform's native policy engine all reduce to the same contract: plan JSON in, exit code out.

Roll policy out in tiers, because enforcing a standard your modules cannot satisfy produces only exceptions:

Tier Behavior Example
Deny blocks the merge public S3 ACL, unencrypted volumes, 0.0.0.0/0 on port 22
Warn annotated on the PR, merge allowed instance type outside the approved list, missing Owner tag
Monitor counted only, reviewed monthly tags present but non-canonical, deprecated instance families

Run monitor for a month, fix the modules so compliant-by-default is the easy path, then promote the rule to deny. The metric to watch is the deny trend: a rule that fires forever is a standard nobody can meet, and the correct response is changing the module — not granting waivers. When a genuine exception is unavoidable, record it as code with an expiry date — an entry in the exceptions file naming the resource, the reason, and the date it lapses — so waivers are visible in review and die on schedule instead of becoming permanent shadow policy.

Policy governs changes; the other half of the foundation governs truth. Remote state with locking, encryption, and access logs underpins every other control — policy, drift detection, cost estimation, and review all read state, and anyone who can write state can redirect resources to an address of their choosing, which is functionally infrastructure admin access. Harden it accordingly:

resource "aws_s3_bucket" "state" {
  bucket = "corp-tfstate-prod"
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.state.arn
    }
  }
}

resource "aws_dynamodb_table" "locks" {
  name         = "corp-tfstate-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

Four properties make that backend an enterprise control rather than a bucket. Versioning recovers from a botched state push. KMS encryption with a key policy limiting decrypt to the pipeline role means a leaked state file is unreadable elsewhere. Bucket policy denies non-TLS transport and blocks the pipeline role from deleting versions. Access logging goes to a separate account the pipeline cannot reach, so tampering leaves evidence. Add a cross-cloud snapshot of state only if outage recovery requires rebuilding there — and remember the locks: the DynamoDB LockID is what turns a concurrent second apply into a fast, loud failure instead of a silent merge of two realities.

The Quarterly Review: Versions, Drift, and Cost Trends That Keep a Practice Alive

A practice that is not measured decays in a predictable order: modules fall two majors behind, drift accumulates until nobody trusts a plan, and costs creep upward in increments no single review catches. The corrective is a fixed review cadence with named artifacts and trigger thresholds — the quarterly review is where the operating model from the first chapter proves it is alive.

Cadence Artifact Owner Trigger and action
Weekly drift report (refresh-only plans) platform any production drift → ticket within 24h
Monthly module & provider version report platform >1 minor behind → upgrade PR; ≥1 major → scheduled migration
Monthly cost trend per environment FinOps >10% month-over-month unexplained → owning team explains or reverts
Quarterly policy violation trends security recurring denial → fix the module, or change the rule
Quarterly test coverage + mean time to merge platform coverage below 80% of modules → remediation sprint
Quarterly state hygiene: orphaned states, key rotation, access logs platform orphaned state → archive; key at rotation age → rotate

Each row is a report that already exists in your pipeline — drift detection, version scanning, cost estimation on plans, policy counts — so the cadence costs assembly time, not new tooling. The thresholds are the discipline: "we look at drift weekly" produces nothing; "production drift older than 24 hours opens a ticket" produces a queue that shrinks. Run the review as a meeting with an agenda and a named scribe producing decisions, not as a dashboard link people promise to open — the artifact that survives is the note saying which threshold was crossed and who took the action item.

Upgrade management deserves special care because provider majors are where practices break. Pin versions deliberately and commit the lockfile, generated with hashes for every platform your CI runs on:

terraform providers lock \
  -platform=linux_amd64 \
  -platform=darwin_arm64 \
  -platform=windows_amd64

Then stage every provider major bump through the same promotion path as code: one week soaking in dev, a few days in staging, production inside a change window, with rollback defined as reverting the version constraint and the lockfile in one pull request. Skipping the multi-platform lock is how a Windows developer and a Linux runner end up verifying different provider binaries — a discrepancy your integrity checking was supposed to prevent.

The final quarter-round question is health, and it is asked of the whole system built across this roadmap: are module contracts holding (how many consumer PRs still edit raw provider resources?), is the pipeline earning its cost (feedback time, red-gate response time), is drift flat, are policy denials trending down, is cost per environment explainable? A practice stays healthy when the numbers that define healthy are written down and reviewed on a schedule someone owns.

For interviews, this capstone is also your answer template for "how would you run Terraform at an organization?" Deliver it in dependency order: state backend with locking and access logs first, since everything reads it; then published module contracts; then the layered pipeline with required checks; then policy as code enforcing what review cannot; then ownership — CODEOWNERS, apply permissions, variable scopes; finally the metric cadence that keeps it from decaying. Six answers, each one enabling the next, none of them a tool comparison — that ordering is the difference between describing Terraform and operating it.

Quiz

1. Four hundred pull requests a month rely on reviewers remembering tagging and encryption standards. What closes the consistency gap?

Question 1 options

2. Where should write access to production Terraform state live?

Question 2 options

3. The quarterly report shows a module three major versions behind, drift up 40 percent, and cost up 12 percent month-over-month unexplained. What is the first move?

Question 3 options

4. A service team wants to modify modules/network. Who approves the change, and under what rules?

Question 4 options

Flashcards

Question

Who are the three owners of every infrastructure change, and why does naming them matter?

Answer

The module author (what the change does), the reviewer (whether it should happen), and the state owner (who controls the record of what exists). Enterprise failure is when one of the three is assumed by nobody — most IaC incidents trace back to an empty cell in that table.

Question

What are the three policy tiers and how does a rule graduate between them?

Answer

Monitor (counted only, reviewed monthly), Warn (annotated on the PR, merge allowed), Deny (blocks the merge). Run a new rule as monitor for about a month, fix the modules so compliance is the default path, then promote it to deny — enforcing a standard nobody can meet only produces waivers.

Question

What properties make a remote state backend an enterprise control rather than just a bucket?

Answer

Versioning to recover from a botched state push, KMS encryption whose key policy limits decrypt to the pipeline role, a bucket policy denying non-TLS transport, access logging to an account the pipeline cannot write, plus a lock table for concurrent writers. State write is infra-admin access, so its controls are the foundation every other control reads from.

Question

How are module and provider versions managed in an enterprise practice?

Answer

Semantic versioning with a changelog, consumers pinned with version = "~> 2.3", breaking changes shipped as majors with a two-quarter deprecation window, upgrades delivered as reviewed pull requests, and .terraform.lock.hcl committed with hashes for every CI platform via terraform providers lock.

Question

What does the quarterly review examine, and what makes it produce results?

Answer

Module and provider version lag, drift report trends, cost per environment, policy violation counts, test coverage and merge time, plus state hygiene (orphaned states, key rotation). It works only as a meeting with an agenda, thresholds, and a scribe producing named decisions — a dashboard link nobody opens changes nothing.

Revision Notes

Key Takeaways

  • 1. Every change has three owners — module author, reviewer, state owner — and the operating model exists so no cell in that table is empty
  • 2. Policy as code converts standards from reviewer memory into deterministic plan-JSON evaluation, rolled out monitor → warn → deny with exceptions that expire
  • 3. Remote state with locking, KMS encryption, versioning, and access logs underpins everything: write access to state is infrastructure admin access
  • 4. Modules follow semver with pinned consumers, multi-platform lockfiles, and deprecation windows; upgrades ride the same gates as any other change
  • 5. Weekly drift, monthly versions and cost, quarterly policy and coverage — thresholds with named owners are what keep the practice from decaying quietly

Interview Tips

  • • "How would you set up Terraform for a 200-engineer company?" — answer in dependency order: state backend with locking and logs, published module contracts, layered pipeline with required checks, policy as code, ownership map, then metric cadence; tool names last, if at all
  • • "Who should be able to change production state?" — the pipeline role via short-lived federation, humans by documented break-glass; justify it by stating that state write lets you repoint any resource
  • • "How do you enforce standards across hundreds of PRs?" — tiered policy on plan JSON: monitor for a month, fix modules so compliant is default, promote to deny; mention exceptions as code with expiry dates
  • • "What do you review quarterly?" — versions, drift, cost, policy trends, coverage, state hygiene, each with a threshold and an owner; show that the cadence produces named decisions, not dashboards

Cheat Sheet

Enterprise IaC Operating Model

Ownership table (write it first):

Concern Publishes Approves Owns operationally
Base modules platform guild module RFCs versions, deprecation windows
State backends platform backend changes lock recovery, access audit
Pipelines & policy platform + security workflow/policy changes runner spend, violation trend
Environments service teams their PRs via CODEOWNERS resources after apply
Production applies team requests SRE (apply permission) incidents, rollback

Module lifecycle: semver + changelog; consumers pin version = "~> 2.3"; breaking = major + migration notes + two-quarter deprecation window; upgrades land as PRs through the same gates; lockfile committed for all CI platforms.

Policy tiers: monitor → warn → deny, promoted after a month of data and module fixes; exceptions are code with expiry dates.

Hardened state: S3 versioning + SSE-KMS (key policy: pipeline role only) + deny-insecure-transport + access logs elsewhere + DynamoDB LockID; humans reach state by break-glass only.

Review cadence:

Cadence Artifact Trigger
Weekly drift report prod drift > 24h → ticket
Monthly version report / cost trend >1 minor behind → PR; >10% MoM → explain
Quarterly policy trends / coverage / state hygiene coverage < 80% → remediation sprint

Provider majors: pin, lock for every platform, soak dev a week, staging days, prod in a change window; rollback = revert constraint + lockfile.

Continue Learning