Terratest Harness in Go

Mocked plans and plan assertions stop at the API boundary. Terratest crosses it: a Go test calls Terraform for real, waits for the cloud to build the thing, checks it with normal Go assertions, and tears it all down — in one go test invocation. That makes it the layer where a dead AMI, a malformed policy, or a missing IAM permission finally fails a test instead of a production apply.

package test

import (
	"testing"

	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

func TestLogBucketModule(t *testing.T) {
	t.Parallel()

	uniqueID := random.UniqueId()
	bucketName := "acme-logs-" + uniqueID

	options := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: "../examples/log-bucket",
		Vars: map[string]interface{}{
			"bucket_name": bucketName,
		},
		EnvVars: map[string]string{
			"AWS_REGION": "us-east-1",
		},
	})

	defer terraform.Destroy(t, options)
	terraform.InitAndApply(t, options)

	actualARN := terraform.OutputRequired(t, options, "bucket_arn")
	assert.Equal(t, "arn:aws:s3:::"+bucketName, actualARN)
}

Read the ordering carefully, because it is the whole safety argument. defer terraform.Destroy appears before InitAndApply. Go runs deferred calls even when the test fails or panics, so a failed assertion still tears the infrastructure down. Written in the other order — apply, then defer destroy, with a fatal in between — the failure path skips the cleanup and the bucket outlives the test. defer Destroy first, always.

The pieces around it: terraform.WithDefaultRetryableErrors wraps the options so known-flaky errors (AWS throttling, eventual consistency during create) trigger retries inside Terratest's Terraform calls; InitAndApply runs terraform init then terraform apply -auto-approve in TerraformDir; OutputRequired reads one output and fails the test if it is missing, where the softer Output returns an empty string. random.UniqueId() is not decoration — real cloud resources need globally unique names, and the suffix guarantees a rerun never collides with a leaked bucket from yesterday's failed run.

Run it the way you run any Go test, and remember the flag that matters most:

go test -v -timeout 45m -run TestLogBucketModule ./test/...

-timeout is not optional. The Go test binary panics and exits when a test exceeds the timeout — default 10 minutes — and an S3 bucket plus policy plus DNS record routinely blows past that on a cold account. The panic interrupts whatever Terraform was mid-apply, and the deferred destroy may never run. Give every Terratest job an explicit, generous timeout; the next chapter's retry loops and the final chapter's leak strategies build on this guarantee.

One structural note on where tests live: TerraformDir pointed at ../examples/log-bucket is the canonical pattern — test the example that doubles as documentation, so the docs themselves are proven to deploy. Unit-level module tests point straight at the module directory with variables, exactly as Plan Assertions and Expectations did with native test files. The two frameworks coexist: native .tftest.hcl for fast mocked plans, Terratest for the real cloud.

Inside the options struct, a few fields earn their keep in real suites. RetryableTerraformErrors, which WithDefaultRetryableErrors pre-fills with known-transient messages like Error waiting for S3 Bucket and throttling errors, is a map you can extend for your provider's particular flakiness. NoColor: true keeps captured output parseable. BackendConfig, Vars, and VarFiles let one test exercise several scenarios without copy-pasting the options block — and when the module under test needs no backend at all, local state inside the example directory keeps runs isolated: a test must never take the lock on the state your staging environment uses.

Read outputs with terraform.Output when absence is acceptable and OutputRequired when it is not; for structured values, OutputList and OutputStruct unmarshal JSON-typed outputs into Go directly, so a list of ARNs arrives as a []string ready for assert.Len.

The infrastructure is up and the output says the load balancer's DNS name — but the very first GET http://alb-123.us-east-1.elb.amazonaws.com/ fails, because DNS has not propagated and target registration is still settling. This is not a bug in your module; it is how distributed systems behave. Terratest's answer is the retry module, which polls until the check passes:

package test

import (
	"fmt"
	"strings"
	"testing"
	"time"

	"github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/retry"
	"github.com/gruntwork-io/terratest/modules/terraform"
)

func TestWebEndpointServesContent(t *testing.T) {
	t.Parallel()

	options := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: "../examples/web-service",
	})

	defer terraform.Destroy(t, options)
	terraform.InitAndApply(t, options)

	albDNS := terraform.OutputRequired(t, options, "alb_dns_name")
	url := fmt.Sprintf("http://%s/health", albDNS)

	status, body := retry.DoWithRetryE(
		t,
		"GET " + url,
		30,               // max retries
		10 * time.Second, // sleep between retries
		func() (string, error) {
			status, body, err := http_helper.HttpGetE(t, url, nil)
			if err != nil {
				return "", err
			}
			if status != 200 {
				return "", fmt.Errorf("expected 200, got %d", status)
			}
			if !strings.Contains(body, "ok") {
				return "", fmt.Errorf("unexpected body: %s", body)
			}
			return "healthy", nil
		},
	)

	_ = status
	_ = body
}

retry.DoWithRetryE runs the closure, and on error waits the interval and tries again, up to the retry count; when the closure finally returns without error, the retry succeeds and the test continues. The three arguments force you to make the eventual-consistency budget explicit: how many attempts, how long between them, and what exactly "done" means. DNS for a new ALB name commonly needs 60–90 seconds; ACM validation can need many minutes — budgets you can state in code review.

Contrast with the instinctive fix, a bare time.Sleep(90 * time.Second) after apply is a guess wearing a constant's clothes.

Too short and the test is flaky — the same failure, just probabilistic. Too long and every green run pays the full tax; a ten-test suite burns fifteen minutes sleeping through successes. Worse, the sleep tells you nothing when it is not enough: the subsequent assertion still fails, with no hint that the underlying issue was timing. A retry loop is fast when the system is ready and honest when it is not — the final error carries every attempt's real reason.

Two practical details from real suites. First, the closures should use the E-suffixed helpers (HttpGetE, not HttpGet), because the non-E variants call t.Fatal on error — and Fatal from a non-test goroutine or deep inside a retry closure gives you a failure without retries. Returning the error is what lets retry work at all. Second, retries are not only for HTTP: aws.WaitUntilS3BucketExistsE, DNS lookups via net.LookupHost, and ssh.CheckSshCommandE against a fresh instance are all eventual-consistency fronts, and they all take the same explicit budget. What sleeps remain good for is the small, deterministic pause between a create and its dependent read — and even those you replace with retry the moment they appear in a flaky test's history.

For readers who prefer fewer moving parts, Terratest ships purpose-built wrappers over the same loop: http_helper.HttpGetWithRetryE takes the expected status and retries inside, and aws.WaitForS3BucketToExistE and similar waiters encode cloud-specific readiness. They are the retry pattern pre-applied; reach for raw retry when the readiness condition is custom — a health endpoint's body, an SSH command's exit code, a file appearing on a mounted volume. Either way, test exactly the condition the next assertion depends on. A retry loop checking status == 200 followed by an assertion checking the body is two budgets for one wait; checking both inside the closure collapses them into one honest failure message.

A test suite that creates real money is a suite with operational duties. Three of them are non-negotiable, and each maps to a specific line of Go or a specific CI flag.

Duty one: outlive the work. The default go test timeout is 10 minutes, and exceeding it panics the test binary mid-apply. The panic path is precisely where deferred destroys are least reliable and real resources are most vulnerable — Terraform was interrupted with the load balancer half-created. Set the timeout to dominate the worst case:

# Makefile-style invocation for the integration lane
go test -v -timeout 60m -run 'Test.*Integration' ./test/...

60 minutes is not optimism; it is apply (5–10 min) + eventual consistency (1–5 min) + retries + destroy (3–8 min), multiplied by whatever concurrency the runner allows. When a test must cap its own region, Terratest offers retry.DoWithRetryAndTimeout, which puts a wall-clock bound around the retry loop itself — the Go test timeout is the backstop, not the scalpel.

Duty two: guarantee the teardown. The skeleton from the first chapter — defer terraform.Destroy before InitAndApply — covers assertion failures and panics. What it cannot cover is a killed CI runner, an OOM-killed test binary, or a SIGKILL from a hard job timeout. Design for that residue rather than pretending it away:

The janitor line earns its place in interviews: it proves you have actually found the $4,000 NAT gateway someone's aborted test run left running for a month, and that your answer was systemic (an isolated test account with an expiry tag), not heroic (a person remembering).

Duty three: parallelize deliberately. t.Parallel() across tests in the same package lets the suite overlap applies, but real clouds impose rate limits — IAM policy propagation, AWS API throttling, per-region vCPU quotas — so the honest configuration is moderate parallelism plus WithDefaultRetryableErrors to ride out throttles:

func TestBucket(t *testing.T)  { t.Parallel(); /* ... */ }
func TestQueue(t *testing.T)   { t.Parallel(); /* ... */ }

// CI: run only what the change touches, with room for the slow lane
go test -v -timeout 45m -parallel 4 ./test/...

Cost control follows the same shape: integration tests live in their own account with budgets and janitors, run nightly or on merge — not on every keystroke — because the mocked plan tests from Mocking Providers in Tests already gate the fast feedback loop. That layering, and where a new test belongs in it, is the subject of the next topic, Layering Tests in the Pipeline. The one-sentence version for the interview room: Terratest proves the cloud accepts and serves what you declared — apply, retry until truly healthy, assert in Go, destroy always, and assume every guarantee will one day be violated by a runner dying, so budget timeouts and janitors accordingly.

One measurable habit ties it together: log the apply and destroy durations per test. Teams that chart them find the slowest test first — usually an unused DNS zone waiting on propagation — and trimming that one wait can halve the nightly lane's wall-clock time, which in turn makes people willing to run the lane more often. Coverage in infrastructure tests is bought with minutes, so spending them where the flakiness actually lives is the whole game.