Remediating Detected Drift

A drift finding is evidence to investigate, not an instruction to apply

When Terraform reports a difference between configuration, state, and a live object, first establish what changed and why. An engineer may have made an emergency console edit, an autoscaler may have changed capacity, a deployment controller may own a field, or an unauthorized actor may have modified a security setting. Compare the plan with cloud audit logs, the last successful apply, incident records, and application behavior. Do not automatically apply until you understand whether doing so would undo a useful recovery action or destroy data.

If the live value is now the intended design, update HCL or the owning module input so future plans agree with reality. Review the resulting plan as a normal infrastructure change. It may show no remote change after refresh, but the code change is still necessary to make the desired value reproducible in another environment. Add rationale, ownership, and migration steps. Do not update a variable only in a workspace UI if the value belongs in versioned configuration; hidden settings make future review difficult.

If the live change was unauthorized or temporary, restore the declared configuration through a normal plan and apply after reviewing the action. A plan might update a setting in place, replace a resource, or recreate one that someone deleted. Verify downtime, data retention, dependencies, backups, and rollback before applying. Restoring a security group rule may be urgent; replacing a database during business hours may require a maintenance window. The word drift does not imply that the remediation is safe.

If an object exists outside Terraform and should become managed, define its resource configuration and adopt it into the correct state using the supported import workflow. Import binds a remote object to a Terraform address; it does not automatically write a complete desired configuration or prove the configuration matches every live attribute. After import, generate a plan and reconcile differences. A plan that proposes to replace an imported production resource is a signal to stop and refine configuration before apply.

If another controller intentionally owns an attribute, decide whether Terraform should manage the controller configuration or ignore a narrow field. For example, Terraform can manage an autoscaling policy while the autoscaler manages current replica count. Document ownership and monitor the external controller bounds. Broad ignore rules can hide real changes and are a poor substitute for clear responsibility.

State operations require care. Removing an address from state stops Terraform tracking that binding but does not delete the remote object. Moving a state address changes Terraform identity mapping and must correspond to a deliberate refactor. Importing the same object into two states can cause both configurations to fight over it. Use a backup and review the plan after any state change. Never edit state JSON directly unless following a documented recovery procedure.

A safe remediation record includes the detected difference, actor or likely cause, chosen source of truth, affected resource, reviewed plan, approver, and follow-up. For temporary emergency changes, add an owner and deadline to reconcile configuration. This prevents a console action from becoming a permanent undocumented workflow.

In an interview, explain three paths: record the live change in configuration, revert the live change, or change Terraform ownership through import or refactoring. Give an example of why you inspect the plan before applying. That shows you understand drift as a governance problem as well as a state comparison.

If the resource no longer belongs in Terraform, remove the configuration and choose deliberately whether Terraform should destroy it or relinquish management. A destroy action may be correct for an ephemeral resource but dangerous for retained data. State removal leaves the remote object in place, so assign its future owner and monitoring path. Do not confuse deleting an HCL block, removing a state binding, and deleting a cloud object; they are distinct operations.

Reconcile a manual production edit with a reviewed change

Imagine an on-call engineer increased a database instance size during an incident. A scheduled refresh-only plan now shows a mismatch. First confirm the cloud account, resource address, and live size. Check the incident timeline and audit log for the actor and API event. Determine whether the larger size restored service and whether it remains necessary. A drift alert alone cannot answer those questions.

If the larger size is needed permanently, update the Terraform variable or module input that controls it. Keep the production value in the approved configuration source. Generate a normal plan and confirm it does not propose an unexpected replacement, storage change, or unrelated update. Review the monthly cost delta and capacity metrics. Merge through the ordinary approval workflow so the next environment or recovery run reproduces the intended size.

If the increase was temporary, decide when it is safe to restore the prior size. Confirm current load, memory pressure, maintenance constraints, and rollback options. The normal plan may propose an in-place resize, but provider behavior and service constraints vary. Schedule the change if it can affect availability. Apply through the protected workflow and verify actual capacity and application health. Record the incident action and follow-up so the temporary exception does not remain indefinitely.

If the engineer created a new resource rather than editing a managed one, decide whether it should be adopted. Write configuration that reflects the live object, select the correct Terraform address, and use the import mechanism supported by the installed Terraform version. Then plan. Terraform may show differences because defaults, omitted attributes, or provider-computed values are not represented as expected. Refine configuration until the plan is understood. Do not apply a replacement merely to force an imported resource into a guessed shape.

If another service owns current capacity, move ownership to a clear boundary. Terraform can create the autoscaling group and its minimum and maximum limits while a runtime autoscaler changes the desired count. Configure Terraform not to overwrite the live count if that is the intended design, and separately alert when the count exceeds safe bounds. Keep the ignored attribute list limited. A team should be able to answer which controller changes capacity and how it is audited.

For a deleted resource, determine whether Terraform should recreate it or whether deletion was an approved decommission. A regular plan may propose creation, but recreating a database, key, or bucket can create a new empty object rather than recover prior data. Inspect backups, retention, references, and deletion protection. If the resource is intentionally retired, remove or alter configuration through a reviewed decommission plan. If it must be restored, follow the service recovery procedure instead of assuming a create action restores its contents.

After remediation, rerun the drift scan and a normal plan. The plan should be clean for the reconciled resource, and the scheduled workflow should show a successful read. If it remains noisy, inspect provider behavior, ignored fields, or an external controller rather than repeatedly applying the same change. Add a regression check or module improvement if the same drift recurs.

A useful incident record captures who changed the resource, why, when configuration was reconciled, and which plan was applied. Preserve the plan and approval evidence in the protected deployment system. Avoid copying secret plan values into an incident ticket. Link the cloud audit event and Terraform run so future responders can distinguish approved emergency actions from unauthorized modifications.

Practice this workflow in a disposable environment: make a controlled change outside Terraform, run a refresh-only plan, record the resulting difference, update configuration, and generate a normal plan. Verify that the plan expresses the intended correction and does not replace unrelated resources. Then test the separate case where the remote object is intentionally removed from management. This rehearsal teaches the difference between reconciling a value and changing ownership.

Remediation is a lifecycle decision

Updating configuration is appropriate when the live state represents a new desired outcome. It keeps infrastructure reproducible and makes future plans converge. Reapplying configuration is appropriate when live state diverged without authorization or the change was temporary. Import or state refactoring is appropriate when Terraform ownership is incomplete or incorrect. These choices are not interchangeable: a wrong choice can undo an incident fix, create a duplicate object, or replace a production service.

Importing an existing object reduces manual ownership but creates responsibility for its future lifecycle. Before adoption, identify dependencies, sensitive attributes, deletion protection, and current drift. A resource with complex immutable settings may be safer to manage through a staged migration than to import and immediately reconcile. Use a test environment to learn provider defaults. After import, review every planned action and confirm the address maps to the intended object ID.

State removal can be appropriate when Terraform should stop managing an object but the object should remain. It is not a deletion. Write down the new owner, configuration system, and monitoring path before removing the binding. Otherwise, the resource becomes an orphan that can drift unnoticed. If a resource should be deleted, remove or alter the configuration and review the destroy plan. State operations should have a backup and a second reviewer for production.

External controllers create a source-of-truth question. Terraform may own the controller policy while the controller owns live values. If both systems try to enforce the same field, plans oscillate. If neither owns it, unexpected changes go unobserved. Define the boundary, use a narrow lifecycle exception when needed, and monitor the external system. Revisit that choice when the controller is removed or the architecture changes.

Automation can speed remediation but should use different rules for different risk classes. A low-risk tag correction in a disposable environment might be auto-applied. A database replacement, public-access change, key deletion, or production identity update should require review. If a bot opens a remediation pull request, include the detected values and evidence, but do not embed secrets. Keep apply behind the same trust and approval requirements as a human-authored change.

Repeated drift alerts may signal process failure rather than one-time operator action. Look for console permissions that bypass the pipeline, an autoscaler conflict, missing module inputs, a provider bug, or a stale state boundary. Reduce direct mutation permissions where practical, but retain a documented emergency path. The goal is a clear source of truth with safe exceptions, not an unrealistic prohibition on every out-of-band operation.

Measure remediation quality by time to classify, time to reconcile, recurring drift rate, and whether the fix preserved service objectives. A low count of alerts can mean good control or poor visibility. Track failed scans and intentionally ignored attributes so the team understands coverage. Review temporary exceptions and remove them when the incident or migration ends.

In an interview, describe the decision tree and a safe production example. State what evidence you gather, which plan actions you inspect, and when import or state removal is appropriate. Mention that state removal does not delete an object and import does not write configuration automatically. Those distinctions prevent common and expensive mistakes.

A mature workflow treats drift as a signal to improve the system. If operators repeatedly make the same manual correction, add a module input or change the rollout process. If a pipeline is too slow during incidents, improve its emergency path while keeping identity and approval controls. If a controller legitimately owns a field, encode that boundary and monitor its limits. Remediation should reduce future ambiguity rather than merely make one plan green.

Use Detecting Configuration Drift to build a trustworthy alert, and State Moved and Removed Resources for address changes and relinquishing management. A safe path leaves one accountable owner for each remote object and important attribute.