Structuring Plan and Apply Jobs

Separate fast validation from privileged work

A Terraform pipeline should make the proposed infrastructure change inspectable before it can be applied. Start with formatting, initialization, validation, and static checks that can run without cloud mutation credentials. Then create a plan in a controlled environment, publish a reviewable summary, and apply only after the required authorization. The stage boundary is not merely a speed optimization; it separates code quality checks from operations that can change shared infrastructure.

Run terraform fmt -check and terraform validate early so style and configuration errors fail quickly. Initialization must use the expected backend and provider versions. Pin provider constraints and commit the dependency lock file so different stages do not silently plan with different provider selections. Validate the workspace and target before creating a plan; an incorrect environment selection can make a valid configuration dangerous.

Apply the reviewed plan

When the pipeline saves a binary plan with terraform plan -out, the apply stage can use that exact plan file. This preserves the actions that reviewers saw instead of recalculating after approval. A plan is tied to configuration, state, provider behavior, and time-sensitive data sources. It may become stale if state changes or the plan's lock expires. Handle a stale-plan error by creating a new plan and obtaining review again; do not silently regenerate and apply a different plan under the old approval.

Transfer the plan as a protected pipeline artifact with a short retention period and tightly scoped access. A saved plan can contain sensitive values even when CLI output redacts them. Avoid publishing the binary plan to a public artifact store or attaching it to a broadly visible log. Publish a readable summary for reviewers and retain the full plan only where the apply job can retrieve it securely.

Stages may run on separate workers. Ensure the artifact identifies the exact commit, workspace, backend, and provider lock used to create it. The apply job should verify those identities before executing. Do not allow an artifact from one branch or environment to be reused for another. Plan files are executable change intent and should receive controls similar to deployment packages.

Control apply authority

Apply should run only from a protected branch or an explicit approval gate. The approval should bind to the plan artifact and commit under review. Limit who can approve production changes, and make the applying identity least-privileged for the target environment. Use a state lock to prevent concurrent mutations. If a pipeline is canceled during apply, inspect state and provider outcomes before retrying; cancellation does not guarantee no resources changed.

Separate credentials for validation, planning, and applying where practical. Some plans need read access to refresh remote state, but they should not receive broad mutation permissions merely because apply later needs them. In multi-account workflows, bind credentials to the environment and workspace selected by the reviewed change. Keep credentials out of plan summaries and artifact metadata.

Design repeatable jobs

Make each stage report its inputs and outputs clearly. A failed validation should show the configuration error; a policy failure should identify the rule; an apply failure should link to state and provider diagnostics without exposing secrets. Preserve logs for audit under the team's retention policy and record who approved and executed a plan. A successful pipeline should leave enough evidence to answer which commit changed infrastructure and what plan was applied.

Do not split a pipeline into so many jobs that ownership and artifact provenance become unclear. A compact validate-plan-review-apply flow is often enough. Add separate stages for policy, cost, integration tests, or environment promotion when they provide a meaningful gate. Every gate should consume the same reviewed intent or explain why it must produce a new plan.

Create a protected production workflow

Build a pipeline with a pull-request validation job and a protected apply workflow. On a pull request, run formatting, initialization with the repository's backend configuration, validation, static scanning, and a plan for the target workspace. Publish a redacted human-readable summary and preserve the binary plan as a restricted artifact. Require the plan status and policy checks before merge. Do not run production apply from an untrusted fork or a branch that can modify the workflow and grant itself credentials.

After merge, retrieve the exact plan created from the reviewed commit only if the state and plan remain valid. Verify the commit sha, workspace, backend key, and provider lock. Require an authorized production approval tied to the plan. Then use terraform apply with the saved plan file. If the plan is stale, fail closed, create a new plan, and repeat review. A changed plan is a changed proposal, even when the code commit is identical.

Protect artifacts and state

Treat the plan artifact as sensitive. Configure access for only the review and apply identities, encrypt storage, limit retention, and avoid printing the binary plan or secret-bearing values to logs. The readable summary should mask values according to policy while preserving resource addresses and action types. Protect the state backend with encryption, access controls, versioning, and locking; CLI redaction does not remove values from state or plan files.

Configure artifact provenance. The plan stage should emit a manifest with commit, workspace, backend identity, Terraform version, provider lock hash, and artifact checksum. The apply job verifies this manifest before use. A cross-environment artifact mismatch should stop the job. Use a unique artifact name per run rather than a mutable latest-plan path that can race with another pull request.

Handle concurrency and cancellation

State locking protects Terraform operations against concurrent state writes, but pipeline concurrency controls improve clarity. Serialize applies per workspace and allow independent workspaces to proceed. If an apply is canceled or times out, do not assume it rolled back. Check state, provider events, and resource health before retrying. A partial apply may have completed some operations; the next plan must reconcile actual infrastructure.

Set timeouts that reflect realistic provider operations, and preserve diagnostic output when jobs fail. Use a separate cleanup or recovery procedure rather than automatically running destroy after an apply error. On successful apply, publish outputs only when they are safe and necessary; mark sensitive outputs and keep them out of general logs.

Verify the gate

Test a no-op plan, a normal resource change, a policy rejection, a stale plan, an unauthorized approval, and a canceled apply. Confirm that a changed artifact cannot bypass review and that a failed apply does not trigger a blind retry. Review the pipeline as code: protect workflow files, require ownership review for changes to credential or branch conditions, and audit execution logs. The workflow is reliable when a reviewer can identify the exact proposed change and the platform can prove the same intent was applied.

Optimize for provenance, not job count

Separate plan and apply to provide review and authorization, but preserve the connection between the reviewed proposal and the executed operation. Replanning after approval may produce different actions because state or external data changed. If a new plan is required, treat it as a new review artifact. A clear provenance chain includes commit, workspace, backend, provider lock, plan checksum, approver, and apply result.

Binary plan artifacts can contain sensitive data, so artifact storage is part of the security design. Restrict access, use short retention, and publish only a redacted summary broadly. Protect the state backend independently. Marking a value sensitive changes display behavior, not encryption or access policy.

Balance access and safety

Pull-request plans improve review but may require cloud read access. Do not expose production credentials to untrusted contributions. Options include plans from trusted branches, isolated preview accounts, static validation without cloud access, or a platform that runs plans under scoped identities. Choose based on the value of a real plan and the threat model. Apply permissions should be narrower and separately approved.

State locks prevent conflicting Terraform writes, while pipeline concurrency controls make scheduling and user feedback clearer. Both are needed in different places. If a job is canceled during apply, reconcile actual state rather than assuming atomic rollback. Design error messages and recovery ownership before an incident occurs.

In an interview, explain the validate-plan-review-apply lifecycle, why the exact plan artifact matters, how credentials and artifacts are protected, and what happens when a plan goes stale. Mention protected branches, manual approval, workspace serialization, and apply recovery. This shows the pipeline treats infrastructure changes as reviewed, attributable operations rather than a command run after every merge.

Artifact provenance matters when jobs run on different workers or approval takes hours. Bind the plan to a checksum and include the source commit, backend identity, workspace, Terraform CLI version, provider lock hash, and creation time. The apply stage should reject a missing or mismatched manifest. Do not use a mutable artifact alias such as latest, which can point to a different pull request's plan. Expire artifacts after the review and apply window, while retaining a redacted audit record under the organization's normal change-retention policy.

Approval design should match impact. A development workspace can use automatic apply after merge, while production may require a separate authorized approver and an environment protection rule. The author should not be able to modify the workflow and approve their own expanded permissions in the same change. Protect the pipeline definition, backend configuration, and role trust policy with code-owner review. Keep the apply identity scoped to the target account and workspace.

Consider plan drift and external data sources. A saved plan is not an evergreen deployment artifact; state changes, provider changes, and expiring plan locks can invalidate it. If the apply job rejects it, create a new plan and obtain approval for the new actions. Do not silently refresh the plan after approval. A no-op plan is also valuable evidence: it can reveal drift or confirm that a prior change is fully reconciled. Define how drift is separated from the pull request's intended change.