Partial Backend Configuration

Here is the tension partial configuration resolves. The backend block accepts only literals — no variables, no expressions, nothing dynamic, as Configuring a Remote Backend established. But one repository serves many environments, and dev and prod must never write to the same state object. You do not want engineers editing the committed block to point at their own stack; you want the block to describe how the team stores state and something else to describe which state this run uses. The answer is to under-specify the block on purpose and complete it at init time:

terraform {
  backend "s3" {
    # bucket, key, and dynamodb_table omitted on purpose
    region  = "eu-west-1"
    encrypt = true
  }
}

This is partial backend configuration: the arguments that never vary for your team are committed, and the ones that vary are left out. Run terraform init against it and Terraform discovers the missing required settings, prompting one per line:

backend.s3.bucket:
backend.s3.key:

The prompt is Terraform admitting the configuration is incomplete — fine for a one-off experiment, unacceptable as a workflow. In CI there is no terminal: init blocks against a closed stdin or aborts, and a pipeline whose state location depends on interactive input is one nobody can reproduce. A related flag matters here: terraform init -input=false turns every prompt into a hard failure, which is exactly what you want in automation — the job should refuse to guess a bucket name rather than hang for the full runner timeout. Two completion mechanisms exist, and teams usually arrive at them in order: first a wrapper script assembling flags, then — once the flag list outgrows one line and appears in three repositories — the per-environment file the wrapper simply points at.

Partial configuration composes with workspaces in a way worth deciding early. Workspaces multiply environments inside one backend configuration: the S3 backend automatically prefixes non-default workspace state as env:/<workspace>/<key> inside the object key, so terraform workspace select staging against the same init produces a distinct state path with zero extra configuration. That pattern is attractive for ephemeral preview stacks and dangerous for anything long-lived, because a typo'd workspace name creates a new state rather than failing loudly, and the surprise appears as an empty plan, not an error.

The first mechanism is the -backend-config flag, taking key=value pairs:

terraform init \
  -backend-config="bucket=acme-tfstate-prod" \
  -backend-config="key=prod/network/terraform.tfstate" \
  -backend-config="dynamodb_table=tfstate-locks"

Every argument on the command line is merged over whatever the block contained, producing the effective backend configuration for that init. One consequence surprises people who have only run full configuration: with a partial block, init is doing double duty as configuration time. Terraform persists the merged result in .terraform/terraform.tfstate — the file holding this directory's backend connection metadata, including any credentials you passed. That is a primary reason the .terraform directory must never be committed. Saved plan files capture the same backend configuration at plan time, so applying a saved plan writes to the backend the plan recorded — a deliberate safeguard, and a reason a saved plan can outlive time-limited credentials.

All of which argues for making the invocation boring and file-driven rather than a wall of flags. That is the next chapter's subject: the config file, and how it keeps both secrets and sanity out of the committed block.

-backend-config accepts a file path as well as key=value pairs, and the file contains nothing but backend arguments in HCL:

# env/prod/backend.hcl — arguments only, no terraform block
bucket         = "acme-tfstate-prod"
key            = "prod/network/terraform.tfstate"
dynamodb_table = "tfstate-locks"
kms_key_id     = "arn:aws:kms:eu-west-1:111122223333:key/abcd-1234"

terraform init -backend-config=env/prod/backend.hcl

The division of labor is now clean. The committed backend block says how the team stores state (S3, region, encryption on) — code review's job. The per-environment file says which state — the environment owner's job. If the file holds nothing more sensitive than a bucket name and a key path, it can be committed too; many teams version the boring files (env/dev, env/staging) and gitignore only the ones carrying account-specific secrets, such as a kms_key_id for a shared-services account or any argument naming infrastructure a compromised repository could reach.

CI wiring becomes declarative and reviewable in the same pull request as the configuration it serves:

- name: Init against the prod backend
  run: terraform init -backend-config=env/${ENVIRONMENT}/backend.hcl
  env:
    ENVIRONMENT: prod

No prompting, no long command lines, and the file the pipeline used is a commit — reviewing a change to prod's state location looks exactly like reviewing any other change.

The scale economics explain why files win over flags once a team passes about three environments. A wrapper carrying six flags per environment needs three code paths; a directory of files needs three files and no code. The failure mode shifts from "the script branched wrong" to "the file says the wrong bucket" — and the second failure is visible in a diff, greppable across repositories, and diffable between environments (diff env/staging/backend.hcl env/prod/backend.hcl tells you in one line whether staging and prod differ only in their key prefix, or also in encryption settings). That last property — environment parity as a reviewable artifact — is the quiet reason platform teams adopt this pattern before they have any compliance requirement pushing them toward it. When an auditor later asks "how do you know prod's state is encrypted and staging's is not?", the answer is a file comparison, not a tribal recollection.

The rules that keep the pattern healthy:

• Files contain backend arguments only — not a terraform { block, not provider config, not interpolated expressions. The literal-only rule still applies inside them; they are values, not code.
• One file per state location, named by environment, not by person. env/dev/backend.hcl, not alice-sandbox.hcl; the directory layout is the documentation of which states exist.
• Never mix files between stacks. Pointing the app stack's init at the network stack's file makes the app plan against the network's state, and nothing validates that the file matches the directory you are standing in.

Workspaces interact with S3 in one specific way worth knowing: the backend automatically prefixes non-default workspace state under env:/<workspace>/ inside the object key. One backend configuration therefore serves every workspace of one stack — useful for ephemeral preview environments. Long-lived environments (dev/staging/prod) are better served by explicit files, because explicitness is a safety property when the thing being varied is production's address.

Partial configuration did not relax an earlier rule: changing backend settings requires re-running terraform init, and init may offer to migrate state. What partial configuration changed is how easily those settings vary — a new bucket per environment, a lock-table rename — which means how often this situation arises. The decision table is small and worth memorizing:

-reconfigure says "accept these settings and keep going": it overwrites the stored backend configuration without attempting any migration. That is correct after a colleague already migrated, or when repointing at a store that was populated another way. -migrate-state forces the copy flow even if init would not have prompted — the right choice the first time a stack moves buckets. Each wrong choice fails memorably. -reconfigure when a copy was needed leaves the new backend empty, and the next plan proposes creating the entire estate over live infrastructure. -migrate-state when a copy was not wanted writes a second copy of state to a new location, leaving two stores that each believe they are authoritative until a human reconciles them — the infrastructure analog of a split-brain database.

-backend=false deserves its own line because it solves a real CI pattern: lint and validate jobs that need the provider plugin directory but should never touch remote state. It skips backend initialization entirely, so the job cannot lock, unlock, or read a state it was never meant to see.

One more scenario sits on this table: migrating away from partial configuration entirely — a team that hard-coded its per-environment files after standardizing on one account per environment. The move is the same ceremony in reverse: replace the partial block with a full one, run init -reconfigure (the state is already in place under the same bucket and key; only the completion mechanism changed), and delete the files. The reason -reconfigure and not a plain init is worth stating plainly: init sees different backend settings than last time and will offer migration, and accepting a migration prompt you do not need is how two state copies come into existence. Any time the state's physical location is unchanged, -reconfigure is the honest answer to init's question.

Every backend change ends with the same verification sequence — this is the habit that turns backend surgery from a white-knuckle operation into a checklist:

terraform init -backend-config=env/prod/backend.hcl   # accept any prompt deliberately
terraform state list | head -20                       # populated? the resources you expect?
terraform plan                                        # no surprise diff, above all no mass-create

If state list comes back empty against a stack you know is running, stop and re-run init with the other flag — you are looking at the wrong state, and apply is the last thing to run. With partial configuration in place you can store state deliberately and complete backend configuration without editing shared files. The next topic begins the state-operations half of this phase: the read-only commands that make every later mutation safer, starting with Listing and Showing State Entries.