Declarative Infrastructure as Code Principles

A team hands you a wiki page to stand up a staging environment: create the VPC, then two subnets, then the security group, then the IAM role, then the RDS instance, with the exact aws CLI commands and the order they must run in. You execute them one evening and it works. Six weeks later a colleague repeats the process, misses step eight (the subnet route table association), and spends a day debugging a "broken" database that is actually fine but unreachable from the app tier.

That failure was not caused by carelessness. It was caused by the tool asking the operator to encode the procedure correctly every time. Declarative infrastructure as code inverts the deal: you describe the end state you want, and the tool computes the steps needed to get from the current state to that end state. You never write "first do X, then Y"; you write down what should exist, and Terraform works out that Y must happen before X, or that neither needs to happen at all because the object is already there.

Consider the same job in both styles:

# imperative: the order IS the logic, and every step must be re-checked by hand
aws ec2 create-vpc --cidr-block 10.0.0.0/16
aws ec2 create-subnet --vpc-id vpc-0abc --cidr-block 10.0.1.0/24
aws ec2 create-internet-gateway ...
# step 8 that got skipped: associate the route table with the subnet

# declarative: the objects, their properties, and the references between them
resource "aws_vpc" "staging" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "app" {
  vpc_id            = aws_vpc.staging.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "eu-west-1a"
}

The vpc_id = aws_vpc.staging.id line is doing two jobs at once. It supplies a value the subnet API needs, and it tells the engine that the subnet depends on the VPC, so the dependency ordering you used to memorize is derived from the configuration instead of the wiki. Miss nothing: the order comes out of the data flow.

Notice what the declarative file does not contain: timestamps, sequencing comments, or "if this fails, try that" recovery notes. The configuration is a specification of facts about the world, and facts have no order. The engine derives order because the world does — a subnet genuinely cannot exist before the VPC it belongs to, and the configuration states that relationship as a data reference rather than a numbered step.

One consequence deserves its own sentence because it surprises everyone on day one: a declarative tool cannot act from files alone. To compute the delta it needs a third input — its state, a record mapping the objects in your configuration to the real objects in the cloud, along with their last-known attributes. That state file is why the tool can say "the VPC already exists, only the subnet is missing", and it is also why teams must later argue about where state lives and who can touch it. You will meet that argument in earnest later in the roadmap; the principle to carry away now is that declarative execution is a three-way comparison — configuration, state, and reality — performed on every run.

Three properties fall out of this model, and they are what interviewers probe. First, idempotence: running the declarative tool against an environment it already built is safe and usually does nothing — terraform apply on converged infrastructure reports "No changes". Scripts achieve idempotence only by hand, with if guards around every step. Second, a single plan of record: the configuration files are the answer to "what does our infrastructure look like", not a console screenshot or someone's shell history. Third, computation at run time: the tool must read the current state of the world before acting, which means it can also tell you what would change before it changes anything — the terraform plan you will meet in The init, plan, and apply Workflow.

The honest trade-off: declarative tools are worse at one-time data migrations and procedural work (running a SQL script inside a database). They are for the lifecycle of infrastructure objects — create, read, update, delete — and knowing that boundary is part of using them well.

In 2017 a major US carrier deleted a peering configuration and took down large parts of its network for hours. The root-cause class is ancient: a human ran a change against production that no one else saw, reviewed, or could roll back. The principle worth internalizing is simple — every infrastructure change should travel through the same pipeline as application code: a branch, a pull request, an approval, an automated plan, and a recorded apply.

Put the infrastructure configuration in a git repository and you inherit an audit trail that scripts running from laptops can never provide. git blame on the line cidr_blocks = ["10.0.0.0/16"] tells you who opened that network, when, and — via the linked pull request — who approved it. The review itself gets materially better, because reviewers no longer read a shell script and imagine its effects; they read the diff Terraform produced:

  # aws_security_group.app will be created
+ resource "aws_security_group" "app" {
+     ingress {
+         cidr_blocks = ["0.0.0.0/0"]   # <- a reviewer catches this in seconds
+         from_port   = 22
+         to_port     = 22
+     }
+ }
Plan: 4 to add, 0 to change, 0 to destroy.

The standard wiring is: your CI system runs terraform plan on every pull request and posts that output where reviewers already are. Nothing applies until the plan is approved and the merge happens. At merge time CI runs terraform apply against a saved plan artifact, so what was reviewed is bit-for-bit what was applied — no re-planning between review and execution.

Reproducibility is the second payoff. Because the environment is fully described by a commit, your dev, staging, and production stacks are the same configuration with different variable values, not three divergent sets of hand-run commands. Two engineers can check out the same commit, run terraform apply in two separate accounts, and get matching infrastructure. When staging misbehaves, "does production have the same configuration?" becomes a git diff question instead of an archaeology project.

The same commit then promotes through environments without ever being edited. What differs between dev, staging, and production is input: a variable file, a backend selection, a workspace — never a fork of the configuration itself. That property is what makes "production is running what staging tested" a checkable claim instead of a hopeful one, and it kills the ancient failure mode where the production configuration drifted from staging because someone hot-fixed one and not the other.

The record outlives the humans, too. Six months after an incident you can answer "when did port 8080 open to the internet, who merged it, and who approved?" with git log -S "8080" -- <path>, and the pull request shows the plan diff exactly as CI produced it, because pipelines archive plan output as a build artifact. Compare that with reconstructing the same timeline from shell history on a jump host, and the argument for never letting production be reachable outside this pipeline makes itself.

Two caveats keep this honest. First, the repository holds structure, not secrets: credentials come from environment variables, TF_VAR_ prefixed variables, or a secret manager — never from a committed terraform.tfvars. Second, state (covered later in the roadmap) contains every computed value including secrets, so the audit story only holds if the state backend is protected too. When an interviewer asks "how do you make infrastructure changes auditable", the complete answer is: version control plus mandatory plan review plus an apply step that cannot run outside the pipeline.

A "snowflake" is any piece of infrastructure that cannot be recreated because its real configuration lives in one person's memory and shell history. The telltale signs: the runbook says "call Priya", the production web server has three patches that staging does not, and a security group has an extra rule nobody can explain. Snowflakes form whenever systems are mutated directly — via console clicks or one-off CLI commands — instead of through the configuration.

The declarative model attacks snowflakes from two directions: rebuildability and detection.

Rebuildability comes from treating servers as disposable carriers of behavior rather than precious artifacts. Bake the OS image, application, and patches into an immutable machine image (traditionally with Packer), then let infrastructure code create instances from that image. If a machine is misbehaving or needs a patch, you do not SSH in and fix it — you build a new image, bump the image reference in configuration, and apply. Terraform replaces the instance. Every difference between environments is now either in the image recipe or in the configuration, both of which live in version control. There is no third place for truth to hide, which is exactly why environments become reproducible on demand.

Detection is where the declared/actual comparison pays off again. Before every operation Terraform refreshes its record of reality from the cloud APIs and compares three things: the configuration (what you want), the state (what it last knew), and the actual remote object (what exists now). If someone edits a security group rule in the AWS console, the next plan surfaces it:

terraform plan -refresh-only
Note: Objects have changed outside of Terraform

  ~ security_group_id = "sg-0c123" -> "sg-0c999" # ingress rule added out-of-band

The -refresh-only flag (Terraform 1.x) scopes the run to exactly this job: report divergence without proposing configuration edits, which you then either incorporate or reject deliberately. If you decide the manual change should stay, apply the refreshed record back to state with terraform apply -refresh-only, then codify the new reality in configuration so the next plan converges on it. If the manual change was unauthorized, delete it at the console or let your next normal apply revert it — the point is that you chose, on the record, rather than discovering the surprise during an incident. Teams that take drift seriously run this on a schedule — nightly in CI, with the diff diffed against an alert threshold — because drift that is measured continuously stays small, and drift that is measured never grows teeth.

This is something no forward-only script can do. A provisioning script that runs once and exits knows nothing about the state of the world afterward; a declarative tool compares against reality on every single run, so drift has a short shelf life.

One practical warning: drift detection only covers the objects Terraform manages. It will not notice the cron job someone added on the host or the bucket policy changed by hand in another account. Drift detection is a scalpel, not a net — pair it with image immutability and restricted console access, and the snowflake has nowhere left to form.

You now have the mental model; Installing and Verifying Terraform starts turning it into a working binary on your machine.