Layering Tests in the Pipeline

A pull request touches one module in a repository of sixty. How much of your pipeline should run on it? If the answer is "everything," median feedback stretches past twenty minutes and reviewers stop waiting for the plan to appear. If the answer is "fmt and hope," you learn about a broken count expression when an apply fails in staging overnight. The workable answer is a cost ladder: order every check by what it costs to run, run the cheap ones everywhere, and ration the expensive ones.

The previous topic built a harness that applies real infrastructure, asserts on it, and destroys it — Terratest Harness in Go. The question now is where that expensive run belongs. The test pyramid for infrastructure inverts the usual intuition about what a "unit" is: there is no pure function to call. Your unit is a plan, because a plan exercises the dependency graph, expressions, variable validations, and the provider schema without touching a cloud API.

Two numbers in that table drive the entire design.

Static checks are free, so they run on every commit with no exceptions. fmt -check -recursive plus init -backend=false && validate finish in under thirty seconds and need no credentials, no backend, and no state. Teams that gate only on plans still spend minutes of runner time before discovering a typo in a variable reference.

terraform fmt -check -recursive
terraform init -backend=false -input=false
terraform validate
tflint --recursive

Integration runs are the only layer with real cost, so they get a budget instead of a habit. A suite that creates a VPC, a database, and a load balancer runs about fifteen minutes; the infrastructure costs a few dollars per execution and the runner adds minutes on top. Six parallel jobs nightly is roughly ninety hosted-runner minutes a day — around forty-five dollars a month before cloud spend. Multiply that by five environments and the pipeline itself becomes a line item someone will challenge.

The rule that keeps this honest: every check that does not need cloud credentials runs on every commit; everything that provisions runs on merges and schedules. Credential-free checks are also the only ones that safely run on pull requests from forks, where secrets are withheld — which is exactly why the plan-unit layer is your external contribution gate.

Design against two failure modes. The first is a wall: ten required checks at four minutes each, serialized, so nobody merges before lunch. Keep total pull-request feedback under five minutes by running layers in parallel and testing only directories the diff touched — a paths filter or a changed-directory matrix cuts a monorepo run from sixty modules to three. The second is a check nobody reads: a nightly job that has been red for a month is decoration. Alert when a gate fails, and treat a broken gate as a bug in the pipeline itself, with the same urgency as a failing service.

Feedback cost is a design input, not an accident. The next chapter takes the middle of the ladder apart: gating a pull request with plan tests that provision nothing.

A plan against a real account is the most informative check you have, and it is the wrong gate for a pull request. It needs credentials, forked pull requests never receive secrets, and a plan against a live account reflects drift and pre-existing resources — not just your diff. The gate instead is terraform test running in plan mode against mock providers: it validates the wiring of the module under test in about a minute with no cloud access at all.

name: ci
on:
  pull_request:
jobs:
  static:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: terraform fmt -check -recursive
      - run: terraform init -backend=false -input=false
      - run: terraform validate
  unit-plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_wrapper: false
      - run: terraform init -backend=false -input=false
      - run: terraform test

terraform test discovers every .tftest.hcl file beside the configuration. With command = plan and a mock_provider block, the run builds the full graph and evaluates every expression against fabricated computed attributes — no credentials, no API calls, no cost. Assertions on planned values catch the mistakes reviewers miss: a count that evaluates to zero, a tag that never propagates, a validation rule that silently does not fire on the path you changed.

The credential-free property is the part that matters most. pull_request workflows from forks run without secrets; any job requiring AWS_ROLE_ARN either fails or tempts someone into pasting keys into the workflow file. Plan-unit tests never ask for secrets, so the identical gate serves internal and external contributors.

What belongs in this layer? Anything a reviewer would otherwise verify by squinting at a diff:

run "rejects_invalid_name" {
  command     = plan
  variables {
    name = "Invalid Name!"
  }
  expect_failures = [var.name]
}

run "creates_bucket_when_enabled" {
  command = plan
  assert {
    condition     = length(aws_s3_bucket.this) == 1
    error_message = "versioning must produce exactly one bucket"
  }
}

These two assertions took seconds to write and they fail deterministically on every pull request that breaks them — the same guarantee you get from a unit test in application code.

Add two more jobs to the same pull request — a plan review and a policy check:

terraform plan -input=false -out=tfplan -detailed-exitcode
terraform show -json tfplan > plan.json
infracost breakdown --path plan.json --format table
conftest test -p policy/ plan.json

-detailed-exitcode returns 0 for "no changes" and 2 for "changes present." A naive CI step treats 2 as failure and blocks every pull request that legitimately adds a resource. Map it explicitly: exit 0 and 2 pass, anything else fails.

Now the merge rule. Require status checks by exact job name — static, unit-plan, policy, plan / staging — so a red or missing job blocks the merge. Two sharp edges bite teams here. Renaming a job leaves the old context in branch protection, and the required check then sits pending forever until someone edits the settings; treat job names as a published contract. And a check that ran against an outdated commit proves nothing about the current tip: enable "require branches to be up to date" (strict: true) or a merge queue that re-runs the plan against the merged result before landing.

Cache provider binaries with TF_PLUGIN_CACHE_DIR rather than caching the whole .terraform directory — a stale .terraform snapshot can resurrect state metadata from a previous run, while the plugin cache is content-addressed against the committed .terraform.lock.hcl.

Every layer so far provably works without a cloud account. What planning cannot prove — that the provider actually accepts your arguments, that the resource reaches the state you asserted, that quotas allow four more load balancers — is the subject of the final layer: scheduled applies in a sandbox.

A plan proves your configuration is internally consistent. It cannot prove that AWS accepts kms_key_id on that resource, that IAM policies propagate fast enough for your assertion, or that the account has quota left for another load balancer. Those questions need real applies — which is why integration tests belong on merges to main and on a scheduled nightly run, never on every pull request.

on:
  schedule:
    - cron: "0 3 * * *"
  workflow_dispatch:
jobs:
  integration:
    strategy:
      fail-fast: false
      matrix:
        module: [network, database, compute, ingress]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: go test -timeout 45m ./...
        env:
          AWS_REGION: us-east-1

Three properties of that job are deliberate. fail-fast: false keeps the other modules running when one fails, so a single broken suite does not hide three passing ones. The matrix isolates blast radius: each run creates its own resources under a unique name suffix, so two modules apply in parallel without colliding on a bucket name. The 45-minute timeout bounds cost — a stuck Destroy cannot bill you all night.

Budget the top of the ladder explicitly. Nightly applies run in a dedicated sandbox account with a hard budget alert and a janitor: a scheduled job that deletes everything older than 24 hours that no active run claims. Failed destroys are the most common source of surprise infrastructure bills. defer destroy() in Terratest runs even when an assertion fails, but a killed runner skips it entirely — only the janitor catches that case.

Decide what is allowed to live at the top of the ladder, because everything here is expensive. Keep only tests that verify behavior no plan can observe: IAM policy attachment actually taking effect, a security group rule reachable from outside, a database accepting connections with the generated credentials, a provider rejecting an invalid argument combination. Logic that merely decides what gets created — counts, tags, name formatting, validation rules — belongs in the plan-unit layer where it runs for free on every pull request. When a nightly failure turns out to be a wiring bug, that is a smell: the test was parked one rung too high.

Flakiness needs a policy, not patience. A test that fails intermittently gets quarantined: tagged in the matrix, excluded from the required set, linked to an issue with a last-checked date. Quarantine is a loan. Anything older than two weeks is either fixed or deleted, because a permanently red test teaches everyone to ignore red — and a suite everyone ignores is worse than no suite, since it still burns runner money.

The final rule is the one interviewers remember: promote every recurring production fix into a test in the same pull request that fixes it. Coverage gaps do not announce themselves on a dashboard; they surface as incidents. If a security group stood open to 0.0.0.0/0 for three days because no test asserted its CIDR, the fix pull request carries both the corrected .tf and a .tftest.hcl case that fails the moment anyone widens the rule again. Add a policy rule on the plan JSON too, so the standard holds for modules nobody remembered to test.

Track coverage the only way that means something for infrastructure: the percentage of modules with at least one test file, and the count of assertions per module. A module with a test file and zero assertions is a test that never fails. Report both numbers monthly; a falling module count is an early warning that the platform is growing faster than its safety net.

That completes the ladder — static, plan-unit, plan-review, integration — each running where its cost is justified. The next topic applies the same layering instinct to changes that never come through your pipeline at all: Detecting Configuration Drift.