Detecting Configuration Drift

Compare declared infrastructure with what the provider reports

Drift is a mismatch that needs an owner, not just a red plan

Terraform manages resources by comparing configuration, state, and the objects reported by provider APIs. Configuration describes desired values. State records Terraform's relationship to remote objects and the last known attributes. During refresh, the provider reads live attributes and updates Terraform's view. Drift occurs when reality changes outside the expected Terraform workflow, such as a console edit, a manual deletion, an autoscaling action, or an external controller changing an attribute.

A normal plan refreshes state and compares the refreshed view with configuration. It may then propose changes that bring the live resource back to the declared configuration. A refresh-only plan focuses on reconciling state with remote reality and showing differences without proposing ordinary configuration changes. This is useful for detection because an operator can inspect what changed before deciding whether to update configuration or restore the declared value. It does not itself explain who made the change or whether it was authorized.

For example, an engineer may increase an instance size in the cloud console during an incident. A later refresh-only plan can show that the remote size differs from the recorded value. The team must decide whether this was a temporary emergency action or the new desired capacity. If the change is intentional, update HCL and review the resulting plan. If it is unauthorized, a normal plan can propose reverting it. Applying a refresh-only plan updates state observations; it does not make a manual change part of configuration or prevent a later normal apply from reverting it.

Drift detection is not always a simple equality check. Some cloud services adjust computed values, controllers change replica counts, and autoscaling changes capacity by design. A provider may normalize a value or report a default that differs from the written configuration. Terraform has lifecycle controls such as ignore_changes for attributes intentionally managed elsewhere, but broad suppression can hide real changes. Define ownership for each changing field: Terraform, an autoscaler, an operator, or a cloud service. Make that boundary explicit.

A drift report should identify the resource address, attribute, observed value category, environment, and likely owner without exposing secrets. Cloud audit logs may identify a principal or API call, but the Terraform plan usually cannot. Correlate plan output with cloud audit events, deployment history, and incident records. If the resource was deleted manually, Terraform may propose recreation; if an attribute changed, it may propose an update or replacement. Review the action before apply because a ?drift fix? can affect availability or data.

Run a scheduled check against the same backend and workspace that owns the resources. A developer's local state may be stale, use different credentials, or target another account. A central CI workflow or managed Terraform platform provides consistent state locking, identity, and report storage. Keep drift jobs read-only where possible and do not automatically apply a correction just because a scheduled run found a difference.

The cadence depends on risk and change rate. A production network or identity boundary may warrant frequent checks; a rarely changed sandbox may use a daily or weekly scan. Faster detection narrows the window for unauthorized change but adds API calls and alert volume. Start with a representative schedule, measure useful findings, and tune noisy resources. Ensure alerts route to a team that can act rather than a mailbox nobody monitors.

In an interview, distinguish drift detection from configuration validation and explain the refresh-only behavior. Then describe how you identify the actor and choose between updating configuration and reverting reality. That demonstrates detection is the beginning of an operational decision, not a command to auto-apply everything.

A refresh-only plan has limits: it can detect only objects and attributes the provider refreshes and Terraform tracks. External changes that are invisible to the provider schema may not appear. A plan can also fail because credentials are missing, APIs are throttled, or a provider cannot read a resource. Treat failed refresh as an unknown state, not as a clean drift result. Report infrastructure errors separately from ?no drift? so the monitoring system cannot confuse lack of evidence with evidence of no change.

Schedule drift checks against the correct state and identity

A useful scan must run where the authoritative state lives

A local command is useful for investigating a resource, but ongoing detection should use the authoritative backend and workspace. Remote state locking prevents a scheduled refresh from colliding with a concurrent apply. The job should identify its workspace, backend, account, and region in its report. If those values are implicit in a developer shell, the scheduled scan can silently inspect the wrong environment or fail to refresh production resources.

A read-only cloud identity is preferable for detection. The provider must be able to inspect the managed attributes, but a drift scan does not need broad mutation permissions. Separate read and apply roles when the platform supports it. Use temporary credentials and narrowly scoped trust policies. If a scan requires additional permission for one resource type, add only the needed read action and document why. Do not solve a permissions error by granting administrator access to the drift job.

A typical automated cycle initializes the configured workspace, validates the provider setup, and generates a refresh-only plan. Store the result as a protected artifact or summarize it in the platform's run UI. Exit status alone may not distinguish drift from a provider read error; parse structured output or use the platform's drift-detection feature to classify outcomes. Do not print full state or environment secrets to logs. A report should include commit, workspace, scan time, and whether the comparison completed successfully.

Choose a cadence that balances discovery with alert quality. A nightly scan may be enough for a stable internal tool; a high-risk production boundary may need more frequent checks. Provider API quotas and plan duration matter when many workspaces run at once. Stagger schedules or use platform queues to avoid a burst of reads. Start with a dry reporting period to learn expected changes, then make high-confidence drift alerting operationally required.

Suppress expected churn narrowly. If an autoscaler owns desired capacity, Terraform should not continuously fight it. A lifecycle ignore_changes rule can communicate that Terraform intentionally ignores an attribute, but it also reduces detection for that field. Another option is to model the autoscaler as the owner and alert on its policy or bounds rather than on each capacity adjustment. Document which system owns the value and how the team verifies that it stays within safe limits.

Detecting drift from a manual edit does not automatically make that edit unauthorized. An incident responder may have increased capacity to mitigate an outage. The report should support a decision: record the change as the new desired configuration, revert it after the incident, or import a resource that Terraform should now manage. Add the responsible principal and incident link when available. Avoid applying a normal plan until that decision is made; it may reverse a recovery change unexpectedly.

A scheduled plan may include changes unrelated to drift, such as a provider upgrade or configuration change merged since the last apply. Compare the plan with the current configuration revision and separate planned code changes from remote differences. A drift alert should not blame an operator for a deliberate new commit. Record the source revision and last successful apply so responders can classify the cause accurately.

Test the detection job in a nonproduction workspace. Make a controlled console change, verify that the report identifies it, and confirm no apply occurs automatically. Then revoke a read permission or simulate an API error and make sure the job reports an incomplete scan rather than ?no drift.? Test the clean case too. These checks validate both detection and failure reporting.

A mature alert contains a resource address, action summary, owner, and link to the run or cloud audit record. It should not contain credentials or sensitive attribute values. Define severity by asset and impact: a public access change or encryption removal is more urgent than a harmless tag update. Route alerts to service owners and include escalation for unacknowledged high-risk findings.

Separate expected automation from unauthorized drift

Drift alerts are strongest when ownership is explicit

Several systems may legitimately update cloud objects. Autoscalers change capacity, managed services rotate endpoints, certificate automation renews values, and security tooling may attach policies. If Terraform also declares those attributes as fixed, every scheduled plan can report a conflict. Decide which system owns each attribute and model the boundary. Terraform may own the autoscaler configuration while the autoscaler owns live replica count. A refresh-only plan can still update state, but a normal plan should not fight a value intentionally controlled elsewhere.

Lifecycle ignore_changes should be used deliberately. It can prevent Terraform from proposing a change to an externally managed attribute, but it also hides differences in that field from ordinary plan output. Keep the ignored attribute list narrow and add an explanation. If the external controller has its own policy or bounds, monitor those instead. Ignoring an entire resource or broad set of attributes can hide unauthorized changes to security-sensitive configuration.

Provider refresh quality affects detection. Some APIs do not expose every field, and providers may treat defaults or computed attributes differently from the cloud console. A provider upgrade can change state representation and produce apparent drift. Before applying a plan that shows widespread changes after an upgrade, inspect provider changelogs and test in a representative environment. A scanner should distinguish provider schema changes from a person changing infrastructure.

Drift can also reveal configuration debt. A resource may have been changed manually because the HCL could not express the required setting or because a module lacks an input. Repeatedly reverting it does not solve the underlying problem. Update the module, document the desired state, and migrate safely. If a resource was created manually, decide whether to import it into Terraform or leave it with a clearly identified owner. Competing management leads to repeated drift and accidental replacement.

A drift job is not a security scanner. It can reveal a changed public access setting or IAM policy, but it only knows the attributes in Terraform state and the provider schema. A static scanner evaluates source patterns; plan policy evaluates proposed changes; cloud posture tooling can inspect broader live resources. Use the relevant control for resources outside Terraform and avoid claiming that a clean Terraform refresh proves the entire account is unchanged.

The report should make its uncertainty visible. If a scan did not complete all resources, state that. If a provider read failed, distinguish it from a detected change. If an attribute is intentionally ignored, note that it is outside Terraform's drift coverage. This is more useful than a binary green indicator that hides blind spots. Keep evidence long enough for investigation but restrict state and plan access.

For remediation, choose from three paths. Update HCL when the live value is the new desired state. Reapply configuration when the change is unauthorized or temporary. Import or refactor state when Terraform's ownership model is wrong. Each path has different risk, and the plan should be reviewed before mutation. Record the decision so future responders do not repeat the same investigation.

In an interview, discuss a manual production edit during an incident. Explain how a scheduled refresh-only plan detects it, how audit logs identify the actor, and why you do not auto-apply the next plan. Then describe the chosen source of truth and how you prevent repeated drift. This answer combines Terraform mechanics with operational judgment.

Use Remediating Detected Drift after an alert to decide whether configuration should change or reality should be restored. For provider read errors and state coordination, review Remote Backend Setup. Detection is valuable only if the state is authoritative and the responsible team can make a safe decision.

Flashcards

Question

What is infrastructure drift?

Click to reveal answer

Answer

A mismatch between Terraform configuration or recorded state and the live object reported by the provider.

Question

What does a refresh-only plan do?

Click to reveal answer

Answer

Shows or reconciles state observations from live infrastructure without proposing ordinary desired-configuration changes.

Question

What should an API error mean for drift detection?

Click to reveal answer

Answer

The scan is incomplete; it must not be reported as a clean no-drift result.

Question

Why use ignore_changes narrowly?

Click to reveal answer

Answer

It can avoid fighting an external controller but also reduces Terraform visibility into the ignored attribute.

Question

What are the main drift response choices?

Click to reveal answer

Answer

Update configuration, revert the live change, or correct Terraform ownership through import or refactoring.

Revision Notes

Key Takeaways

1. Drift compares declared or recorded infrastructure with values refreshed from live providers.
2. A refresh-only plan helps identify remote differences without applying ordinary configuration changes.
3. Scheduled scans need authoritative state, read permissions, and clear failure reporting.
4. Expected changes from autoscaling or external controllers need explicit ownership and narrow exclusions.
5. An alert requires an intent decision before HCL changes, revert, or state ownership changes.

Interview Tips

• Explain the difference between a normal plan and a refresh-only plan.
• Describe how you distinguish a manual edit from a provider read error or a recent code change.
• Mention how autoscaling ownership and ignore_changes affect drift visibility.
• Give a safe remediation path that does not automatically undo an incident response.

Cheat Sheet