Data Source Lookups

Some values your configuration needs already exist and are not yours to manage: the latest Ubuntu AMI in AWS's account, the VPC somebody else built, your account ID for ARN construction, the available zones in the region. A data block asks Terraform to look one up:

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "logs" {
  bucket = "acme-logs-${data.aws_caller_identity.current.account_id}"
}

The shape is deliberately parallel to resource: two labels — the data source type from the provider, and your local name — forming an address (data.aws_caller_identity.current) that every other expression traverses through the data. scope. The differences are all about lifecycle. A resource is something Terraform creates, updates, and destroys; a data source is something Terraform reads — during plan and refresh — and records the result in state so references can use it. There is nothing to apply, nothing to destroy, and no drift on the object itself; what can change is the result of the query, and Terraform re-runs it on every plan, so a lookup that starts returning different values shows up as a diff in whatever consumed it. That last property is the quietly important one: data sources are live queries, not one-time snapshots.

Because results land in state, they are inspectable exactly like resource attributes — terraform state list shows data.aws_ami.ubuntu alongside managed objects, and terraform show -json includes the recorded result set. Which raises the obvious question of judgment: when does something become a data source versus a resource? The rule that keeps configurations honest: if Terraform should ensure it exists and track its lifecycle, it is a resource; if it must merely be read, it is a data source. Looking up the VPC that a different stack manages is a data source; if this directory is responsible for the VPC, it is a resource — reading your own managed objects through data sources instead of references is a real anti-pattern (it hides dependencies and invites inconsistency), and interviewers ask about it directly.

One timing subtlety completes the model: data source reads participate in the dependency graph. A data source that references a resource in its configuration — say, filter { name = "vpc-id" values = [aws_vpc.main.id] } — is read after that resource's attributes are known. On a first apply where the VPC is being created in the same run, the read is deferred until apply does — the data source's values arrive as (known after apply) placeholders in the plan, the same unknown-value machinery you met in the expressions topic. References order reads; reads order consumers.

The provider catalog also shapes what a data source can be. Providers expose three flavors of read: single-object lookups with required identifiers (data "aws_subnet" "chosen" { id = "subnet-0aaa" }), filtered lookups that must resolve to exactly one match, and listings that enumerate collections — all sharing the same read-at-plan lifecycle. Whatever the flavor, the contract holds: Terraform has no way to create what a data source asks for; if the object is missing, the run fails. That is why the discipline of choosing between resource and data is worth the paragraph it took above — every wrong choice is either a hidden lifecycle (data where resource belonged) or a hidden dependency (resource where data belonged), and both surface later, in production, expensively.

Lookup data sources answer a precise question: given these filters, give me the one matching object. The canonical example — finding the current generation base image — shows every moving part:

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["099720109477"]   # Canonical's account ID

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

most_recent = true tells Terraform how to break ties: the query legitimately matches many AMIs (every patch release in the family), and this flag selects the newest by creation date. Without it, multiple matches are an error — "Your query returned no results. Please adjust your search criteria and try again." is the no-match version, and multiple results without most_recent is its mirror. Learn to read that error as information: your filter matched nothing, which almost always means a stale family name (-focal- after Canonical moved on), a wrong owner, or a region where the image does not exist — the lookup has precisely one job and refuses to guess outside its contract.

The consumed result is just attributes, and the idiom that follows is the pipeline into a resource: ami = data.aws_ami.ubuntu.id. Because the lookup re-runs every plan, most_recent = true also carries a behavioral consequence worth stating before it surprises you: whenever Canonical publishes a new image, your next plan proposes replacing the instance that used the old AMI. Many teams deliberately run most_recent only in dev and pin explicit AMI IDs in production — the trade-off between always-current and always-stable is a policy decision, not a technical default.

Two patterns handle conditionality. When a lookup may legitimately find nothing and that is acceptable, give the data source a count and reference the indexed result behind the same condition:

data "aws_ami" "custom" {
  count = var.use_custom_ami ? 1 : 0
  owners = [data.aws_caller_identity.current.account_id]
  filter { name = "name" values = ["app-*"] }
}

ami = var.use_custom_ami ? data.aws_ami.custom[0].id : data.aws_ami.ubuntu.id

count on a data source changes its address to data.aws_ami.custom[0], exactly as with resources — the meta-argument behaves identically in both block types, which is one more reason Attributes, References, and Meta-arguments insisted on the general rule. The other pattern, for values that are never conditional but vary per environment, is a simple lookup with no tie-breaking flag: data "aws_route53_zone" "primary" { name = "${var.domain}." private_zone = false } — one zone by name, erroring loudly if it does not exist.

Filter mechanics are worth one more pass because they are where real debugging happens. Filter blocks are ANDed — each filter must match — while the values list inside one filter is ORed, so "name matches pattern X and (region eu-west-1a or eu-west-1b)" is two blocks, not one. Providers also accept direct attributes where a filter would work (image_id = "ami-123" is the precise case), and every lookup exposes the matched object's full attribute set — data.aws_ami.ubuntu.name, .creation_date, .architecture — which is exactly what a precondition consumes when you need to assert a property of what you found ("the selected AMI must be x86_64") before anything launches from it.

Lookups answer one-object questions; listing data sources answer collection questions: give me all the subnets in this VPC, all the availability zones in this region, all the peering connections matching these tags. Their shape is the same two-label block, but their result is a collection you project into other expressions:

data "aws_subnets" "app" {
  filter {
    name   = "vpc-id"
    values = [aws_vpc.main.id]
  }
}

resource "aws_db_subnet_group" "app" {
  subnet_ids = data.aws_subnets.app.ids
}

The result exposes .ids (and .arns), a list that feeds subnet_ids-style arguments directly, or [for id in data.aws_subnets.app.ids : id] when you need to transform it. The enumeration semantics matter: listing data sources make no promise about which elements exist ahead of time, so they are the correct tool for "whatever is there when this runs" — subnets created by another stack, zones the region offers, tags applied by an unrelated process. They make no promise about order either: treat .ids as an unordered collection and wrap it in sort() whenever a downstream consumer (an array index in a script, a hash-based comparison) would misbehave on reshuffled results. The classic failure to avoid is hardcoding a list of subnet IDs that a peer team keeps resizing; the listing lookup makes the configuration follow reality, and because the query re-runs every plan, a changed result set shows up as a reviewable diff before anything applies.

Now the failure mode this whole topic has been circling: a data source lookup that fails aborts the entire plan. No resources are created, updated, or destroyed; the run stops with the lookup's error — "Your query returned no results", "Invalid VPC ID", an assume-role failure. Interviewers love this because it cuts both ways. On the one hand it is exactly the early-surfacing behavior you want: a missing prerequisite — the VPC that was supposed to exist, the AMI nobody built, the assume-role that stopped working — fails your plan loudly instead of letting a half-configured apply proceed. On the other hand, it couples your plan's ability to run to the existence of things outside your state, so an unplanned deletion by another team turns your next deploy into a lookup error rather than a diff. The engineering responses are familiar: assert prerequisites in CI with a plan before the change window, make genuinely-optional lookups conditional with count, and treat "data source failed" as a missing-prerequisite incident rather than a Terraform bug. The diagnosis order that speeds these incidents up: read which data source the error names, reproduce its query with the provider's own CLI (aws ec2 describe-images with your filters pasted in), and check whether the object moved, vanished, or whether credentials/region changed — nine times in ten the query, not Terraform, has stopped matching the world. One last distinction that settles most design questions: data sources belong to things you do not own; things you own should be resources or references. When both options exist — a lookup of an object in your own state versus a plain reference to it — the reference wins every time: it is cheaper (no API read), more honest (the dependency is visible), and impossible to silently drift. With reads covered, the next topic is the graph those references build: Implicit and Explicit Dependencies.