Finding and Preventing Goroutine Leaks

A service runs clean at 14 goroutines after startup. A week of traffic later, memory has crept up and the count reads 48,000 — none of them burning CPU, none of them in your code's obvious loops, all of them parked in channel operations and system calls. The process still answers requests, just slower, bigger, and one OOM away from a page. This is a goroutine leak: work that can never finish, occupying memory and runtime resources forever because nothing in the program is capable of waking it.

Unlike a heap leak, a goroutine leak is inspectable — the goroutines are still there, with full stack traces. Start with the count:

fmt.Println(runtime.NumGoroutine()) // call it before and after the operation you suspect

Run the suspect operation repeatedly; if the count never returns to baseline, dump the stacks and read them:

buf := make([]byte, 1<<20)
n := runtime.Stack(buf, true) // all goroutines
fmt.Printf("%s\n", buf[:n])

// or via pprof, which aggregates and is friendlier in production:
// curl localhost:6060/debug/pprof/goroutine?debug=2

A profile output groups identical stacks and shows counts — "4200 goroutines blocked in select" tells you the shape instantly. Reading a leaked stack is a skill: find the frame that is your code, then the frame below it that shows why it waits. The vocabulary of blocked frames maps almost one-to-one to fixes:

The debugging workflow is mechanical from there: dump stacks, group by frame, identify the waiting contract each group is stuck on, and find the code that owed them a wakeup — the producer that exited without closing, the consumer that stopped reading, the WaitGroup whose counter never hit zero. The stack tells you what is waiting; the missing wakeup tells you who owes it. The rest of this topic is about not owing it in the first place.

Most channel-related leaks are variants of four shapes. Each has a canonical cause and fix.

1. Send with no receiver. A goroutine sends to a channel whose consumer exited early — the classic request-scoped worker whose handler stopped reading:

results := make(chan Result)

go func() {
	for _, item := range items {
		results <- process(item) // handler returned after first error: parked forever
	}
}()

The consumer's return closed the door; the producer still knocks. Fix: give the send an escape route, watching the cancellation signal:

select {
case results <- process(item):
case <-ctx.Done():
	return ctx.Err()
}

2. Receive with no sender and no close. for v := range ch where the producer died before closing — a panic in the producer skips its defer close only if the defer itself was never registered, and more commonly the producer returns early on an error path that bypasses close. Fix: defer close(out) on the producer side, registered before any work that might return early (close ownership), so every exit path marks the stream done.

3. Sender blocked on a full buffer that nobody drains. The buffered-channel variant of shape 1: make(chan T, 64) looks safe until the consumer quits; the sender parks at slot 65 and never moves. The buffer only postpones the leak. Fix: same as shape 1 — select the send against ctx.Done() — and audit who drains before sizing buffers large.

4. Waiting for a count that never lands. A goroutine in wg.Wait() while a sibling exited without Done — usually because the sibling returned on an error before its defer wg.Done() was registered (Add after launch, Done not deferred), or because one of N workers was cancelled and skipped its work entirely. Fix: defer wg.Done() as the first statement in the goroutine, Add before launch, and never gate a WaitGroup on workers that can fail before they start. The context-flavored sibling: a goroutine selecting on ctx.Done() of a context whose cancel function was dropped without being called — the leak that defer cancel() discipline (cancellation) prevents.

A cross-cutting rule covers all four: the goroutine that stops participating must not be the one someone else is blocked on. When a consumer gives up, it owes senders an unblock (drain in a goroutine, or cancellation the senders watch). When a producer gives up, it owes consumers a close. Shutdown is a debt both sides must pay, and shape analysis during review — who is blocked on this channel, and what exactly wakes them? — finds the unpaid ones before runtime does.

The four shapes share a root cause: a goroutine with a blocking operation and no exit condition. Prevention is making the exit condition structurally unavoidable.

Every goroutine gets a done path, named in the launch site:

func worker(ctx context.Context, in <-chan Job, out chan<- Result) error {
	for {
		select {
		case j, ok := <-in:
			if !ok {
				return nil // normal end
			}
			select {
			case out <- do(j):
			case <-ctx.Done():
				return ctx.Err() // even the SEND watches cancellation
			}
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

The nested select is the detail that matters: the outer one escapes waiting for input; without the inner one, a worker that receives fine but blocks pushing to a full output still ignores cancellation completely. Walk the goroutine's stack in your head — every line where it can park needs a case in some select, a context-aware library call, or a guarantee that the other side will unblock it.

Tickers and timers need explicit shutdown. time.NewTicker never stops on its own; a goroutine leaking a ticker leaks its timer and its channel forever:

ticker := time.NewTicker(time.Second)
defer ticker.Stop()

Beware the related one-shot patterns: time.After in a loop allocates a timer per iteration (modern Go can garbage-collect unreferenced timers, but stopped-and-drained explicit timers remain the pattern that behaves identically across all supported versions), and a time.AfterFunc callback that sends to an abandoned channel parks a runtime timer goroutine. Where the runtime cannot help — anything blocked on your channel — cancellation is entirely on you.

Retire the goroutine, not just the signal. Closing done tells workers to stop; it does not prove they stopped. Shutdown sequences that must not leak need the join: cancel, then wait for workers to exit before the process does — a WaitGroup alongside the context, or errgroup's g.Wait() which already couples both. main returning without joining races the last workers' cleanup against process exit; on a graceful-shutdown path, the sequence is: stop accepting → cancel contexts → wait for workers → close resources, in that order.

Mechanize the check. Discipline decays; tests do not. The goleak package snapshots goroutines at test start and fails the test on ones left behind:

func TestMain(m *testing.M) {
	goleak.VerifyTestMain(m)
}

Every leaked goroutine then shows up as a failing test with its full stack — turning a production-week-long creep into a CI failure on the exact PR that introduced it. The cheaper approximation, counting runtime.NumGoroutine() before and after a test body with a small settle window, catches gross regressions without a dependency. Either way, the target is the same one the profile table started with: after work completes, the goroutine count returns to baseline — because every goroutine the program started has been given a way to finish.