S3 Bucket and DynamoDB Locking Backend

The default state backend for AWS estates is not magic — it is two managed services each doing what they are good at. S3 stores the state object with durability and versioning; DynamoDB provides the conditional-write primitive that locking needs. Together:

terraform {
  backend "s3" {
    bucket         = "acme-tfstate-prod"
    key            = "network/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "tfstate-locks"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:eu-west-1:111122223333:key/abcd-1234"
  }
}

S3's part is simple: bucket plus key identify one object per stack, so network/terraform.tfstate and app/terraform.tfstate coexist in one bucket. region selects the endpoint. encrypt = true makes every write server-side encrypted, and kms_key_id upgrades that from the platform default key to one you control — the difference matters for compliance and for the off-switch story in Configuring a Remote Backend.

DynamoDB's part deserves the detail, because interviews and incidents both go there. The dynamodb_table argument names a table whose partition key must be a string attribute called LockID — this is not configurable, and a table with any other key schema fails at init with a validation error. When an operation needs the state, the backend performs a conditional write:

PutItem LockID = "acme-tfstate-prod/network/terraform.tfstate"
  Condition: attribute_not_exists(LockID)
  Item: LockID, Info = { Operation: "OperationTypeApply",
                          Who: "1.2.3.4", Path: "/home/dev/infra",
                          Created: "2026-03-11T09:14:22Z" }

If the item does not exist, the write succeeds and you hold the lock; if it exists, DynamoDB rejects the request and Terraform surfaces the conflict:

Error acquiring the state lock

Error message: ConditionalCheckFailedException: The operation attempted to
edit a unique item over an item that already exists

Lock Info:
  ID:        8f4e2c9a-...
  Operation: OperationTypeApply
  Who:       10.0.4.12
  Created:   2026-03-11T09:14:22Z

Read that error the right way: it is not a failure of Terraform, it is the lock working — someone else is mid-apply, and the Lock Info block tells you who, from where, and since when. On completion the backend deletes the item with a condition that the stored LockID still matches, so you can never release a lock you do not hold.

Two extension points round out the model. Workspaces map onto the bucket as key prefixes: each non-default workspace stores its state under env:/<workspace>/<key>, so terraform workspace new staging gets its own state location without touching the backend block — an alternative to one backend per environment that some teams accept for low-stakes stacks. And the lock's Who field records the caller's identity as the AWS SDK reports it — the IAM principal behind the credentials — which is how a 2 a.m. page gets answered with a name instead of a guess; pair it with Created to judge whether a lock is stale or a colleague is simply mid-apply. One more flag belongs here: -lock=false disables locking for a run, which is tolerable for read-only investigation and never acceptable before an apply, since it means reading state that may be mid-write.

One nuance keeps the mental model honest: the lock item holds no state data. It is a mutex record. Losing it (manual deletion) does not lose state — it removes mutual exclusion, which is a different and recoverable problem covered in State Lock Breakage and Recovery. The state object itself lives in S3, where its protection story is the next chapter's subject.

Anyone can create an S3 bucket; almost nobody regrets configuring these four things on day one, and every one of them has rescued a real team:

Versioning. The failure it answers: a state overwrite — a botched migration, a destructive state rm, a bug in a wrapper script — writes a bad current version, and without versioning the previous version is simply gone. With versioning enabled, recovery is a console operation: open the object's versions, pick the version from before the incident, and either copy it over the current version or point Terraform at it. The discipline that makes this work is not enabling lifecycle expiry on noncurrent versions too aggressively; a 30-90 day noncurrent retention is a reasonable floor for state.

Encryption with a customer-managed key. The backend's kms_key_id directs writes at your key; enable automatic key rotation on that key and nothing in the workflow changes. The review question this answers: can the storage operator read your state? With a CMK under your key policy, decryption requires an explicit grant — and destroying or disabling the key renders state unreadable to everyone, including an attacker holding bucket access. Remember the pairing from the previous topic: writers need encrypt/decrypt on the key, so key policy and bucket policy are one access model.

Public access blocks and a TLS-only policy. aws_s3_bucket_public_access_block with all four settings true, plus a bucket policy that (a) denies aws:SecureTransport = false so state never travels over plain HTTP, and (b) denies non-KMS-encrypted puts. Neither costs anything; both are the first two lines of any security review of your state store.

Lifecycle and resilience refinements. Two additions earn their place as the store matures: a lifecycle rule expiring noncurrent versions after a conservative window (90 days is common) and aborting incomplete multipart uploads, and — for estates with a regional-risk appetite — cross-region replication of the bucket so state survives a regional event. Cost engineering has a place too: S3 Bucket Keys reduce the KMS request cost of frequent state writes without changing the security model, because the key policy still governs decryption.

The bootstrap problem. The bucket must exist before the backend can use it — so which state tracks the bucket itself? The standard answer: a tiny bootstrap configuration, applied once by hand, containing only the state bucket and lock table, with local state accepted for that single run because it creates nothing but the store itself:

# bootstrap/main.tf — applied once, local state is acceptable here
resource "aws_s3_bucket" "state" {
  bucket        = "acme-tfstate-prod"
  force_destroy = false
}
resource "aws_dynamodb_table" "locks" {
  name         = "tfstate-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

Note force_destroy = false — a versioned bucket holding objects cannot be destroyed by Terraform without it, which is a last-line guard around your only copy of history. The bootstrap repo is small, changes rarely, and deserves the same review rigor as everything downstream of it: it defines the floor the whole estate stands on. Once the bucket exists, a worthwhile finishing move is migrating the bootstrap's own state into it — add a backend block with a dedicated bootstrap/terraform.tfstate key and run terraform init -migrate-state — so even the store's definition has a durable home.

The lock table itself is almost trivial — and its details are exactly where production issues hide.

Schema and billing. hash_key = "LockID" with a string (S) attribute, on-demand billing (PAY_PER_REQUEST). Locks are millisecond-scale items; provisioning capacity is pure overhead. Nothing else is required: no sort key, no GSIs, no TTL — a lock that is never cleaned up by its owner is cleaned up by the operator after diagnosis, deliberately, not automatically.

Permissions. The state plane is a separate permission surface from the deploy plane, as established earlier. A CI role that applies infrastructure needs, at minimum:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
  "Resource": "arn:aws:s3:::acme-tfstate-prod/network/*"
},
{
  "Effect": "Allow",
  "Action": ["s3:ListBucket"],
  "Resource": "arn:aws:s3:::acme-tfstate-prod"
},
{
  "Effect": "Allow",
  "Action": ["dynamodb:PutItem", "dynamodb:GetItem", "dynamodb:DeleteItem"],
  "Resource": "arn:aws:dynamodb:eu-west-1:111122223333:table/tfstate-locks"
}

Scoping GetObject to the stack's key prefix means one role per environment can be genuinely least-privilege — the pipeline for network cannot even read app's state.

Failure modes worth recognizing on sight. Lost update without a lock: if dynamodb_table is removed from the backend config, Terraform warns but proceeds — locking silently off, and the cross-machine races return. Stale lock: an interrupted apply (Ctrl-C, killed CI job, VPN drop) leaves the lock item behind; every subsequent operation fails with ConditionalCheckFailedException until you diagnose and remove it. Versioning false confidence: versioning recovers overwrites, not concurrency — two writers still race, versioning just preserves the casualties. Lock-table name drift: the partial-configuration pattern of the next topic exists largely so per-environment table names can vary without editing shared code, and a typo'd table name shows up as a lock error rather than a helpful "table not found."

One human check belongs in every unlock procedure: confirm in the CI system that the job holding the lock has actually finished before forcing it. Skipping that check is precisely how two applies run concurrently — the race the lock exists to prevent. Teams write the confirmation into the incident runbook next to the unlock command, and the runbook earns its keep the first time the "killed" job turns out to have been retrying in the background.

Scale and latency. State operations are whole-object round-trips: the backend fetches the entire state object and writes the entire object back, so a state grown to tens of megabytes makes every plan and apply slow in ways no parallelism flag fixes. The durable answer is structural — split stacks by blast radius, which shrinks both the state and the review scope — and it is the same refactor that makes teams' pipelines bearable at scale.

Audit and forensics. Because every lock carries who-and-when and every state write is an S3 API call, the observability story is unusually good if you turn it on: S3 access logging plus CloudTrail data events on the state object answer the questions incident reviews actually ask — when was state last modified, by which principal, from which address. Data-event logging is cheap relative to that answer; enable it on the state bucket specifically, not account-wide.

Ergonomics. Keep one bucket, prefix keys by stack and environment (prod/network/terraform.tfstate), and let the per-stack CI roles' resource ARNs mirror that layout — the naming becomes the access model. Enable bucket logging so state access leaves an audit trail. And put the bucket's versioning, encryption, and public-access settings under drift detection like any other resource, because the controls protecting state deserve the same continuous verification as the workloads it tracks. When the store is this well understood, the remaining backend concerns — varying these settings per environment without editing the committed block — are squarely a partial-configuration problem: Partial Backend Configuration.