Injecting Secrets from External Stores

Retrieve secrets through an identity boundary, not a committed variable

Terraform needs credentials, but configuration should not contain long-lived secrets

Terraform providers authenticate to cloud APIs and other services. Prefer workload identity or short-lived role credentials supplied by the execution platform over static access keys stored in HCL. A CI job can exchange its identity token for a temporary cloud role session; a developer can use the approved local credential chain. Restrict the role to the environment and actions required by the plan or apply. Terraform configuration should describe which provider to use, not contain a reusable administrator password.

Sometimes a managed resource needs an application secret, such as a database password or API token. A provider data source can read a named value from AWS Secrets Manager, Vault, or another secret system. The data source is a read operation. Terraform tries to read it during planning, though it can defer the read until apply when an argument depends on a value that is not known yet. The secret store controls retrieval permissions and rotation, but once Terraform reads the value, the next question is where Terraform and the provider persist it.

A sensitive variable or output tells Terraform to redact the value in normal CLI or UI output. It is a display label, not encryption and not a guarantee that the value is absent from state. A data source result or managed resource argument can still be written to plan or state files depending on the configuration and provider schema. Anyone who can read those artifacts may be able to recover the secret. Marking the variable sensitive reduces accidental disclosure in output; it does not replace state protection.

Modern Terraform also supports ephemeral values and provider-defined ephemeral resources. Ephemeral data is available during an operation but omitted from plan and state files, subject to where Terraform allows the value to flow. A provider may expose an ephemeral resource for temporary credentials, and some managed resource arguments are write-only so the value can be sent during apply without being stored. These features are useful for passwords, tokens, and temporary connections when the provider supports them. Confirm the resource and argument support in the installed Terraform and provider versions; a sensitive data source does not become ephemeral just because it is marked sensitive.

Use the narrowest mechanism that fits the secret. For provider authentication, use the platform credential chain or an ephemeral input when supported. For a secret that must be configured into a remote service, prefer a write-only resource argument or a provider-supported secret reference. If a value must persist because the remote resource exposes it as readable state, Terraform cannot make that lifecycle secret-free; evaluate whether another system should configure it, and secure state accordingly.

A secret's source and destination are separate trust boundaries. A data source may require permission to read a secret, while the resource provider needs permission to configure the target. Avoid giving one role access to every secret in an account. Scope reads to the specific secret path and use separate roles for planning and applying when possible. Do not output the secret for debugging or pass it through broad module outputs. Child modules should accept a narrow sensitive input or ephemeral value and avoid echoing it in outputs.

Plan and state protection is mandatory whenever sensitive data can be stored. Use a remote backend with encryption in transit and at rest, state locking, access control, and audit logs. Restrict plan artifacts and JSON output, shorten retention, and avoid committing them to version control. CI logs and pull request comments should contain a resource address and safe error category rather than the secret value. A state file is a sensitive data store even if the console redacts every secret.

Secret rotation affects Terraform behavior. If a secret is rotated outside Terraform, a data source may read the new value on a later plan, potentially causing a downstream resource update. If the resource provider cannot read back the configured secret, Terraform may not know whether the remote value changed. Use an explicit version or rotation trigger when necessary, and avoid forcing a replacement just to push a new password unless the service requires that lifecycle. Define how rotation is tested and how consumers reload the credential.

In an interview, distinguish authentication credentials from application secrets, then explain sensitive versus ephemeral. Sensitive redacts display; ephemeral omits supported values from plan and state. Close with backend protection and least-privilege identity. Those distinctions matter more than naming a particular vault.

Environment variables set by a secret-aware platform can keep literal values out of configuration files, but they do not automatically keep the values out of state or logs after Terraform consumes them. Use environment injection for provider credentials when supported, and use ephemeral inputs or write-only arguments for values that must not persist. Inspect the provider behavior rather than assuming the source channel controls every later copy.

Pass only the needed secret to the resource and protect every artifact

Follow the value through plan, apply, state, and logs

Start by identifying the consumer. A cloud provider may only need a temporary credential to call its API; the application itself may need a database password; a deployment resource may need a third-party token. Do not retrieve a secret into Terraform if the provider can authenticate through workload identity or if the application can retrieve it directly at runtime. Every Terraform data flow creates another opportunity for state, plan, or log exposure.

If Terraform must retrieve a value, read a single secret by stable identifier and use a role limited to that path. Avoid listing every secret and filtering in HCL, because that broadens access and can expose names or values unnecessarily. Keep the secret's value out of outputs, local debugging, and generated files. If a child module needs it, pass only that value through a sensitive input, and use ephemeral flow when the Terraform version and module interface support it. Do not duplicate the value in several resources unless each consumer truly requires it.

Inspect the saved plan and state behavior in a sandbox before using a secret in production. A data source read may be visible in the plan, or a resource argument may be recorded in state. Sensitive display marks can mask output while the underlying artifact still contains the value. If a value is persisted, verify remote backend encryption, state access permissions, audit logging, backup retention, and artifact controls. Never upload a plan JSON document to an unrestricted pull request just because a normal plan command redacts it.

When the provider offers an ephemeral resource or a write-only argument, check its exact lifecycle contract. Ephemeral values are available only in allowed contexts and are not kept in state or plan files. Write-only arguments may require a companion version field so Terraform knows when to resend a value that it cannot read back. If the remote API cannot report the current secret, Terraform cannot compare the actual secret with configuration. Rotation therefore needs an explicit signal and a tested update procedure.

A practical workflow for a database credential has distinct phases. The secret manager owns generation and rotation. Terraform configures network access and the database instance. A supported write-only password field or ephemeral path delivers the new value during an operation. The application retrieves or receives the credential through a separate secure mechanism. Health checks verify that both the database and application accepted the rotation. Avoid printing the value during tests; assert only that authentication succeeds and the version changed as intended.

CI identity should not come from a long-lived secret stored in repository variables if federation is available. Bind the job identity to the repository, branch, and environment. Use a read-only role for plan when feasible and a narrowly scoped apply role behind approval. Do not expose secrets to arbitrary pull request workflows from forks. A plan stage may need read permissions and secret retrieval, but a formatting or static scanning stage should not receive cloud credentials at all.

Secret rotation can race an apply. If the version changes after the plan, the saved plan may contain an old value or may need to be regenerated. Define whether rotation is coordinated by Terraform or by the secret manager and how the deployment consumes the new version. Avoid allowing a scheduled Terraform run to roll back a newer manually rotated value. A version identifier can make the intended secret generation explicit without publishing the secret itself.

Failure messages need care. A provider may include a request payload or response body in an error. Review logging behavior and sanitize output in wrappers before publishing it to CI. Log the secret identifier, resource address, and correlation ID only if those details are safe. Do not print environment variables for debugging. If a secret is accidentally committed or exposed in a plan artifact, rotate it immediately and follow the incident response process; removing the text from a later commit does not invalidate the credential.

In an interview, trace one secret end to end: source identity, retrieval permission, Terraform value semantics, resource consumption, state persistence, rotation, and log protection. Explain which stage owns the secret and why. This shows you understand that ?stored in a vault? does not make every downstream copy safe.

A reviewer can ask whether Terraform needs the plaintext at all. Some cloud resources can receive a reference to a managed secret and retrieve it at runtime, avoiding secret material in Terraform. That may change the application identity and availability requirements, but it often reduces exposure. Choose based on the service's supported integration and operational needs, not merely convenience during initial provisioning.

Choose between Terraform delivery and runtime secret retrieval

A secret store is only one part of the design

Terraform-delivered secrets are useful when infrastructure creation must establish an initial credential or configure a service that has no native secret reference. Runtime retrieval is often preferable when applications can authenticate to a secret manager with workload identity and fetch values as needed. Runtime retrieval avoids putting plaintext into Terraform plans and state, but it adds a dependency on the secret service, network access, identity policy, caching, and failure behavior. A service that cannot reach its secret store may fail to start or refresh credentials.

Choose where rotation occurs. A secret manager can rotate a value independently, but consumers must detect and reload it. Terraform can set a version or trigger a resource update, but then rotation becomes coupled to an infrastructure run. A write-only argument protects persistence but does not magically reconcile a remote value that the provider cannot read. Define the authority, trigger, rollout order, and rollback behavior. Test overlapping credentials if the service supports zero-downtime rotation.

Secret value handling has a cost. Passing the value through Terraform can simplify bootstrap but expands the sensitive data boundary to the runner, provider, state backend, saved plans, and any system that can read those artifacts. Runtime retrieval shifts the boundary to the application identity, secret manager, and network path. Map the new risks rather than declaring one design universally safer. Use short-lived credentials where possible and make access auditable.

Terraform's sensitive flag is useful but easy to misunderstand. It redacts values in normal outputs; it does not encrypt a local state file, restrict state readers, or ensure a provider does not store the value. Ephemeral variables, ephemeral provider resources, and write-only arguments address persistence for supported flows, but they have restrictions and depend on version and provider support. Pin compatible versions and test before adopting the feature in a critical pipeline.

Data sources can be deferred until apply if their inputs depend on values unknown during plan. This affects review because the secret or returned attributes may not be available in the plan stage. If a policy or resource argument depends on that value, test the complete run sequence and understand which phase reads the data. Do not assume every secret is resolved before reviewers approve a plan. Make the dependency and its security implications visible in the workflow.

Avoid placing secrets in Terraform outputs, even sensitive outputs, unless a consumer truly needs an output contract and the state access is appropriately restricted. A root output is often surfaced in CI or workspace UI. Prefer passing a value directly to its consumer and returning a nonsecret identifier or version. For cross-module use, an ephemeral child output can avoid persistence where the language supports it; do not attempt to expose an ephemeral value as a persistent root output.

Use a threat model for state access. Separate backend permissions by environment, encrypt backups, enable audit trails, and review who can download state. Remote storage can prevent local persistence in ordinary operations, but plan files and logs still need protection. Terraform's locking prevents conflicting state writes; it is not a confidentiality control. Treat state recovery and secret rotation as separate procedures.

In an interview, compare Terraform injection with direct runtime retrieval. Discuss bootstrap needs, blast radius, rotation, and availability. Then state that sensitive is redaction while ephemeral is non-persistence in supported contexts. Avoid claiming that a vault alone eliminates exposure. The quality of the identity and artifact controls determines where copies can appear.

For a deployment review, document the secret owner, retrieval principal, consumer, persistence behavior, rotation trigger, and recovery path. Verify the plan/state behavior with a disposable credential and inspect the actual artifacts under controlled access. Revoke the test credential afterward. This evidence is stronger than assuming the resource provider treats a field as secret.

Read HashiCorp's sensitive data guidance for sensitive and ephemeral values, and the data source documentation for when data reads can happen during planning or apply. Provider-specific secret support must be checked in that provider's resource documentation.

Flashcards

Question

What does sensitive=true do?

Click to reveal answer

Answer

Redacts a value from normal display; it does not guarantee the value is absent from state or plans.

Question

What is an ephemeral Terraform value for?

Click to reveal answer

Answer

Passing temporary data through supported contexts without persisting it in plan or state files.

Question

Why might a secret data source read during apply instead of plan?

Click to reveal answer

Answer

Its arguments may depend on values Terraform cannot determine until resources change during the run.

Question

What should a Terraform state backend provide when secrets may persist?

Click to reveal answer

Answer

Encryption, access control, locking, and audit logging, with protected backups and plan artifacts.

Question

What is a common alternative to Terraform-delivered application secrets?

Click to reveal answer

Answer

Let the application retrieve secrets at runtime through its own least-privilege workload identity.

Revision Notes

Key Takeaways

1. Prefer short-lived workload identity over static credentials in HCL or repository variables.
2. Sensitive marks redact display, while ephemeral values and write-only arguments can avoid persistence in supported cases.
3. Data source values and resource arguments may still appear in plan or state; verify the provider behavior.
4. Protect remote state, plan artifacts, CI logs, backups, and secret retrieval roles.
5. Choose Terraform delivery or runtime retrieval based on rotation, availability, and access boundaries.

Interview Tips

• Trace a secret from its source through retrieval identity, Terraform, resource use, and state.
• Distinguish sensitive redaction from ephemeral non-persistence.
• Explain how a rotated secret reaches consumers without exposing it in CI output.
• Compare Terraform injection with runtime retrieval, including the new availability dependency.

Cheat Sheet