Graceful Shutdown and Connection Drain

The orchestrator sends SIGTERM during a rollout. The service's main has no signal handling, so the Go runtime's default action fires: the process dies immediately. Three hundred requests in flight are cut mid-response — clients see connection reset, the load balancer retries some of them, and the ones that were non-idempotent (a payment capture, a cart update) get double-submitted by the retry. The fix is not exotic; it is a sequence, and the sequence has an order.

Start with the signal. signal.NotifyContext cancels a context when SIGTERM or SIGINT arrives, which lets shutdown flow through the same cancellation mechanism as everything else:

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           mux(),
		ReadHeaderTimeout: 5 * time.Second,
		IdleTimeout:       60 * time.Second,
	}

	errCh := make(chan error, 1)
	go func() { errCh <- srv.ListenAndServe() }()

	select {
	case err := <-errCh:
		if !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	case <-ctx.Done():
		log.Println("shutdown requested, draining")
	}

	stop() // stop intercepting; further signals use default behavior

	shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
	defer cancel()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("forced shutdown: %v", err)
	}
	log.Println("server stopped")
}

Shutdown does exactly two things: it closes all listeners immediately (so no new connections arrive), then it polls open connections until they finish their current request or the context expires. Idle keep-alive connections are closed right away; a connection mid-request is given until the deadline. The distinction matters — Close() kills everything now, Shutdown() waits, and the timeout converts waiting into a bounded worst case.

The order inside main is deliberate. stop() before draining restores default signal handling, so a second SIGTERM — the operator who decides the drain is stuck — hard-kills the process instead of queuing behind the first. And errors.Is(err, http.ErrServerClosed) is how a clean shutdown differentiates itself from a real bind failure: ListenAndServe always returns an error, and mistaking the expected one for a crash makes the service log a fatal error on every healthy deploy.

The deadline needs an arithmetic justification, which is the third piece: the drain budget must exceed your slowest request but stay under the orchestrator's kill grace period (Kubernetes' terminationGracePeriodSeconds, minus the time the platform needs for its own steps). A 30-second grace inside a 40-second kill window leaves room for both. If Shutdown returns a context-deadline error, work is still in flight — log it loudly, because the following Close() is about to cut those connections. The pattern for coordinating the same drain across goroutines beyond HTTP appears in errgroup; the HTTP half is what we have just seen.

srv.Shutdown covers plain HTTP request handlers and nothing else. The moment your service has background workers, message consumers, WebSocket sessions, or in-flight database work, the sequence needs more pieces — each one an explicit wait with its own deadline.

Background workers. A goroutine consuming from a channel and a ticker flushing every 30 seconds do not stop when the HTTP server does. Give every worker the shutdown context and wait for it:

func worker(ctx context.Context, jobs <-chan Job) error {
	for {
		select {
		case <-ctx.Done():
			return nil // drain is requested: finish gracefully
		case j := <-jobs:
			if err := process(j); err != nil {
				return fmt.Errorf("job %d: %w", j.ID, err)
			}
		}
	}
}

Two rules separate correct from nearly-correct: a worker must check ctx.Done() before blocking forever on its channel (a plain <-jobs with no select never wakes), and something must wait for it — a sync.WaitGroup, or an errgroup that also propagates the first worker error. Cancelling without waiting is just a slower crash: main returns, the runtime tears down remaining goroutines mid-write, and the flush that was about to happen never does.

Hijacked connections and WebSockets. Shutdown does not track connections whose handler has called Hijack — chat sessions, gRPC streams, and long-poll handlers are invisible to it. The server believes it has drained while those sockets remain open. The fix is bookkeeping:

var inflight sync.WaitGroup

func wsHandler(w http.ResponseWriter, r *http.Request) {
	conn, _, err := hj.Hijack()
	if err != nil {
		return
	}
	inflight.Add(1)
	go func() {
		defer inflight.Done()
		defer conn.Close()
		serveSession(conn)
	}()
}

// in main, after srv.Shutdown:
inflight.Wait()

Every hijacking handler registers before it starts its session and deregisters when the session ends; shutdown waits on the same WaitGroup after Shutdown returns. Whether sessions should be closed gracefully (send a goodbye frame, wait for acknowledgements) or terminated after a shorter grace is a product decision, but it must be a decision, not an accident.

Database pools and message consumers. db.Close() after the workers stop — closing first turns every in-flight query into an error. Consumers stop taking new messages, finish the current one, and nack or leave unacked anything unfinished so another instance can pick it up; a consumer that commits half a batch during shutdown recreates the truncation problem at the storage layer.

Putting the pieces in one goroutine makes the order explicit:

// shutdown sequence, once, on SIGTERM:
readiness.Store(false)     // 1. stop routing new traffic (see below)
_ = srv.Shutdown(drainCtx) // 2. drain HTTP
workersCancel()            // 3. cancel worker contexts
wg.Wait()                  // 4. wait for workers and hijacked sessions
db.Close()                 // 5. close resources last

The readiness flip in step 1 deserves its own paragraph because it is the step most often missing: cutting the listener stops new connections, but the load balancer keeps sending requests until health checks notice. A readiness endpoint that returns 503 as soon as shutdown starts gives the routing layer a few seconds to divert traffic while existing requests finish — the coordination between drain and routing is the subject of Metrics, Logs, and Traces as well, and the two halves only work as one mechanism.

A graceful shutdown that has never been tested is a hypothesis. The hard parts are budgets — how long each phase may take and how they nest — and the discipline of demonstrating the behavior before an incident demonstrates it for you.

The server needs timeouts even when shutdown works. An http.Server with no ReadHeaderTimeout accepts a client that opens a connection and dribbles headers forever; WriteTimeout never firing lets a slow reader pin a handler. These limits interact with the drain: a request that started just before SIGTERM may run up to WriteTimeout, so the drain budget must exceed it.

srv := &http.Server{
	Addr:              ":8080",
	Handler:           mux(),
	ReadHeaderTimeout: 5 * time.Second,
	ReadTimeout:       15 * time.Second,
	WriteTimeout:      30 * time.Second,
	IdleTimeout:       60 * time.Second,
}

ReadTimeout/WriteTimeout cover whole phases; ReadHeaderTimeout alone fixes the slowloris case without clobbering long downloads — for file transfers, drop WriteTimeout and bound the handler with its own context deadline instead.

Budget arithmetic, written down:

slowest request (≤ WriteTimeout)   10s
worker drain                       5s
shutdown drain context            25s   ← total your code allows
orchestrator kill grace           40s   ← platform hard stop
second-signal escape hatch        operator / tini sends SIGKILL

The invariant: your drain deadline must be less than the platform's kill, or the platform becomes your timeout and your logs show an unexplained SIGKILL instead of a controlled forced shutdown message. When they are equal, the race decides — and the race is never on your side.

Readiness is the difference between draining and dropping. Liveness (/healthz) answers "is this process functional" and should stay 200 until the very end; readiness (/readyz) answers "should traffic be routed here" and must flip to 503 the instant shutdown begins. Kubernetes removes the pod from endpoints on a readiness failure — but only if something flips it, so tie the endpoint to the same atomic.Bool the shutdown sequence sets:

var ready atomic.Bool

mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
	if !ready.Load() {
		http.Error(w, "draining", http.StatusServiceUnavailable)
		return
	}
	fmt.Fprintln(w, "ok")
})

With a preStop hook or a small sleep before Shutdown, the flip propagates through the load balancer before the listener closes — closing the listener first still races the health-check interval.

Prove it with a test. Start the server with httptest.NewServer, begin a request whose handler blocks on a channel, call Shutdown in a goroutine, and assert that the in-flight request completes while a new connection is refused:

started := make(chan struct{})
release := make(chan struct{})
// handler: close(started); <-release; write 200
// test: launch request, wait <-started, go srv.Shutdown(ctx),
//       then expect new GET to fail, close(release),
//       assert first request returned 200.

That test encodes the entire contract in fifty lines: old requests finish, new connections are rejected, and the process exits only after the first assertion holds. Add a race-detector run (go test -race, see The Race Detector) because shutdown bugs are data races in disguise — the readiness flip, the worker wait, and the connection counters all touch shared state. A drain that works under -race in CI is the closest thing to a guarantee you get before the first real deploy.