HTTP Clients, Timeouts, and Retries

The payment webhook handler calls http.Get(ledgerURL + "/entries/" + id) and forwards the result. One day the ledger service accepts connections but never responds. Your handler has no timeout, so its goroutine waits forever; requests pile up behind it; the load balancer keeps sending more. The call that had no deadline becomes the outage — for your service, not the ledger's.

http.Get uses http.DefaultClient, whose Timeout is zero: unlimited. The fix is a client you own, with timeouts layered by phase, because "the request hung" can mean five different hangs:

var client = &http.Client{
	Timeout: 10 * time.Second, // ceiling for the WHOLE call: dial -> TLS -> headers -> body
	Transport: &http.Transport{
		DialContext: (&net.Dialer{
			Timeout:   3 * time.Second,
			KeepAlive: 30 * time.Second,
		}).DialContext,
		TLSHandshakeTimeout:   5 * time.Second,
		ResponseHeaderTimeout: 5 * time.Second,
		IdleConnTimeout:       90 * time.Second,
		MaxIdleConns:          100,
		MaxIdleConnsPerHost:   20,
	},
}

The layering answers different questions. DialContext.Timeout bounds TCP establishment — a black-hole network fails in 3 seconds instead of hanging. TLSHandshakeTimeout catches a peer that accepts TCP but stalls during certificates. ResponseHeaderTimeout catches a server that connects and reads your request but never answers headers — the exact "accepted but silent" failure that motivated all of this. Timeout on the client is the backstop covering everything, including reading the response body, which none of the transport-level knobs see.

Timeouts are only half the control; the other half is cancellation. Derive each call's context from the inbound request so that a client who disconnects does not leave your service fetching on their behalf:

ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()

req, err := http.NewRequestWithContext(ctx, http.MethodGet, ledgerURL+"/entries/"+id, nil)
if err != nil {
	return fmt.Errorf("build request: %w", err)
}
resp, err := client.Do(req)
if err != nil {
	if errors.Is(err, context.DeadlineExceeded) {
		return ErrUpstreamTimeout // your 504, distinct from a 4xx passthrough
	}
	return fmt.Errorf("call ledger: %w", err)
}

NewRequestWithContext makes the context part of the request, so cancellation propagates through dial, TLS, headers, and body reads. Two deadlines coexist on purpose: the per-call 2 seconds is the budget this operation can afford, while the client's 10-second Timeout is the absolute ceiling for anything using that client. When both exist, the tighter one wins — and the errors still tell you which fired (error identity with errors.Is). What no timeout can fix is an upstream that fails transiently. That is a different problem with its own rules.

A teammate wraps every downstream call in a retry loop "for resilience." The ledger's POST /charges starts timing out under load, so your service dutifully retries — and every retry that actually succeeded server-side creates a second charge. The resilience feature becomes a duplicate-payment incident, and the retry policy is the culprit.

Retry safety has two axes: was the failure transient (would another attempt plausibly succeed) and is the request safe to repeat (does a second attempt duplicate an effect). Only when both hold should you retry.

func doJSON(ctx context.Context, method, url string, body []byte, out any) error {
	var lastErr error
	for attempt := 0; attempt < 3; attempt++ {
		if attempt > 0 {
			wait := time.Duration(1<<attempt) * 100 * time.Millisecond // 200ms, 400ms
			wait += time.Duration(rand.Int63n(int64(wait / 2)))       // jitter
			select {
			case <-time.After(wait):
			case <-ctx.Done():
				return ctx.Err() // caller's budget outranks the retry schedule
			}
		}

		var rd io.Reader
		if body != nil {
			rd = bytes.NewReader(body) // rewindable: safe for repeats
		}
		req, err := http.NewRequestWithContext(ctx, method, url, rd)
		if err != nil {
			return err
		}
		resp, err := client.Do(req)
		if err != nil {
			lastErr = err
			if !isTransient(err) { // DNS failure, invalid URL: deterministic
				return err
			}
			continue
		}

		if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {
			_, _ = io.Copy(io.Discard, resp.Body) // drain so the connection can be reused
			resp.Body.Close()
			lastErr = fmt.Errorf("upstream %d", resp.StatusCode)
			continue
		}
		defer resp.Body.Close()
		if resp.StatusCode >= 400 {
			return fmt.Errorf("upstream %s: %w", resp.Status, ErrClientFault) // 4xx: not transient
		}
		return json.NewDecoder(resp.Body).Decode(out)
	}
	return fmt.Errorf("gave up after 3 attempts: %w", lastErr)
}

Every clause encodes a rule. Exponential backoff with jitter stops a thundering herd of clients from stampeding a recovering upstream in lockstep. The select on ctx.Done() means retries never outlive the caller's patience — without it, three attempts times upstream latency blows past the inbound deadline. Rebuilding the body from a []byte each attempt matters because the reader from attempt one is consumed; retrying with a half-read stream sends a truncated request.

What counts as transient: connection resets and timeouts before a response arrived, 429, and 5xx. What never does: 4xx (the request itself is wrong), and — for non-idempotent methods — anything after a response may have been processed. For POST endpoints that must survive retries, the upstream contract needs an idempotency key header so a repeated charge with the same key returns the original result. The client sends the key; the server deduplicates. Neither side alone is enough.

Bounded attempts, jittered backoff, context supremacy, and an honest classification of failures: that is the whole policy. The last piece of the outbound story is what happens to the connections these calls leave behind.

A service dials a downstream on every request and closes each connection after use. Latency graphs show a sawtooth: smooth while idle connections warm up, then a spike every time they expire. The fix is not faster networking — it is letting the transport keep connections alive, which means reusing one client and closing bodies properly.

http.Client is designed to be shared. Behind it, the Transport maintains a pool of idle connections keyed by host, and every reuse skips TCP, TLS, and the HTTP handshake. Three habits break the pool:

Creating a client per request. http.Client{} with a fresh Transport each call starts with zero idle connections — you pay the full handshake every time and abandon the old pool. One client (or one per distinct configuration) lives for the process.

Mutating http.DefaultClient. It is global state shared with every library that did not bring its own client — a package that sets DefaultClient.Timeout = 5s changes timeouts for your entire process, and the reverse: a DefaultTransport you tune globally affects third-party calls that expected defaults. Own a client; leave the defaults untouched (middleware's version of this rule: config at wiring, not globals).

Abandoning response bodies. A response you never close keeps its connection out of the idle pool forever — the classic slow leak that ends as "too many open files." Closing alone is not enough for error paths: the connection is only reusable once the body is read to EOF.

func fetchStatus(ctx context.Context, url string) (int, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
		return 0, err
	}
	resp, err := client.Do(req)
	if err != nil {
		return 0, err
	}
	defer resp.Body.Close()

	// consume before close, or the connection dies with the body
	_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<20))
	return resp.StatusCode, nil
}

The pool has a size, and the defaults are quietly small: MaxIdleConnsPerHost defaults to 2. A service doing 500 concurrent calls to one host keeps at most two idle connections; the rest are closed on release and re-dialed on the next burst. Setting it near your real concurrency (as in the client from the first chapter) is usually the single highest-leverage latency change for chatty downstreams. MaxIdleConns caps the global total across all hosts; IdleConnTimeout decides how long an unused connection survives.

You cannot tune what you cannot see: the transport exposes httptrace hooks and the Transport's ResetStats/Stats methods show pool hits, misses, and wait counts — the client-side cousin of the database pool metrics you will meet in Connection Pool Tuning. And since outbound calls are where distributed latency accumulates, the honest way to prove these settings worked is to test the client against a controllable server (httptest) rather than by watching production graphs after the fact.