Mutexes, RWMutex, and Atomics

An endpoint tracks concurrent requests in inFlight so a load balancer can shed traffic. Under load the dashboard reads 47 while ps shows 180 connections. The code is three lines:

type Gauge struct{ n int }

func (g *Gauge) Incr() { g.n++ }
func (g *Gauge) Decr() { g.n-- }

g.n++ is not an operation — it is three: read the value, add one, write it back. Two goroutines can both read 47, both write 48, and one increment evaporates. No bug in the arithmetic; the bug is interleaving without mutual exclusion. The race detector names it precisely (The Race Detector), and the repair is Go's lock:

type Gauge struct {
	mu sync.Mutex
	n  int
}

func (g *Gauge) Incr() {
	g.mu.Lock()
	g.n++
	g.mu.Unlock()
}

func (g *Gauge) Decr() {
	g.mu.Lock()
	g.n--
	g.mu.Unlock()
}

func (g *Gauge) Get() int {
	g.mu.Lock()
	defer g.mu.Unlock()
	return g.n
}

The critical section — read, add, write — now executes as a unit; a second goroutine's Lock blocks until the first Unlocks, and the lock also supplies the memory ordering the happens-before relation demands: unlock-then-lock makes the first section's writes visible to the second.

Four disciplines keep locks correct. Every access goes through the lock, reads included — the reader that "only reads" a value mid-write is exactly how the gauge drifted. Lock and unlock live in one method; a Lock() here and Unlock() in the caller cannot survive an early return. The zero value is usable — var mu sync.Mutex needs no constructor, which is why the field pattern above works. Never copy a value containing a mutex: pass pointers (func (g *Gauge) receiver, not Gauge), because a copy is a different lock guarding nothing. go vet flags copies after use.

What the lock does not give you: fairness (waiters queue roughly FIFO but nothing is contractual) or protection across methods without the same mutex — Inc on one copy and Get on another coordinate nothing.

A config cache is read by every request handler and rewritten once a minute by a refresh job. One sync.Mutex serializes all readers, which looks wasteful — many reads do not actually conflict. sync.RWMutex distinguishes: any number of readers or one writer, never both.

type Cache struct {
	mu   sync.RWMutex
	vals map[string]string
}

func (c *Cache) Get(k string) (string, bool) {
	c.mu.RLock()
	defer c.mu.RUnlock()
	v, ok := c.vals[k]
	return v, ok
}

func (c *Cache) Set(k, v string) {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.vals[k] = v
}

Readers share; writers exclude everyone. For a cache read on every request with a write per minute, that ratio is exactly what RWMutex is built for.

But the intuition "more readers = faster" is where teams get hurt. RLock is not free: it performs an atomic increment on a shared counter — a cache line that every concurrent reader bounces — and it still fully excludes writers while they hold the lock. Two regimes follow:

A mutex-guarded read of a map in 30 ns under contention can lose to RWMutex only when readers genuinely overlap for meaningful time. The rule: start with Mutex, switch to RWMutex when a profile shows lock contention dominated by concurrent readers, not before. go test -bench with -mutex-profile or pprof's mutex profile makes the call empirical (Profiling with pprof).

One deadlock to know by heart: read locks are not recursive-safe when a writer is waiting. Goroutine A holds RLock, a writer calls Lock (queued, now blocking new readers), A tries a second RLock — deadlock. The documentation says it plainly: no goroutine should expect to re-acquire a read lock until the initial one is released. Design so each section takes its lock once.

Also remember what RWMutex still does not give: atomicity across the two methods — Get returning a value that a Set invalidates one line later is a logic-level staleness problem, not a race. Locks prevent interleaving; they do not make multi-step workflows atomic.

The in-flight counter never needed a lock — it needed one indivisible word operation. sync/atomic provides exactly that, and since Go 1.19 the typed wrappers make misuse hard:

type Gauge struct {
	n atomic.Int64
}

func (g *Gauge) Incr()      { g.n.Add(1) }
func (g *Gauge) Decr()      { g.n.Add(-1) }
func (g *Gauge) Get() int64 { return g.n.Load() }

Why this fixes what ++ could not: atomic.AddInt64 lowers to a single hardware instruction (lock-prefixed add on x86) — read, modify, write happen as one unit at the CPU level. No goroutine observes the middle. Atomics also supply sequentially consistent visibility, so a Load after another goroutine's Add cannot return a stale word.

Flags get the same treatment, with compare-and-swap as the interesting primitive — "do this only if nobody else already did":

var started atomic.Bool

func startOnce() {
	if started.CompareAndSwap(false, true) {
		go run() // exactly one goroutine ever enters
	}
}

CompareAndSwap(old, new) returns true only when the current value equals old, so N goroutines racing to flip the flag produce one winner. That is the hand-rolled version of one-shot initialization, and sync.Once is its stdlib sibling for functions:

var (
	cfgOnce sync.Once
	cfg     *Config
)

func config() *Config {
	cfgOnce.Do(func() { cfg = loadConfig() })
	return cfg // every caller sees the fully built value
}

Where atomics stop. Two failure classes lie outside their reach. Multi-word invariants: balance and pending must move together — two atomic updates leave a window where one changed and the other did not; only a lock covering both restores the invariant. Compound access: atomics cannot increment a map entry, push a slice, or swap a struct field while recomputing it — each atomic op is one word, and anything larger is a critical section. The line is simple: atomics protect a value; locks protect an invariant.

Why prefer the typed wrappers over atomic.AddInt64(&x, 1) on a plain int64: they prevent accidental non-atomic access (x++ compiles on the raw variable), carry their own no-copy semantics, and guarantee the 8-byte alignment that 64-bit atomics require on 32-bit platforms — a footgun that used to crash rather than miscount.

Choosing becomes mechanical: single counter or flag under contention → atomic; any state spanning two fields → mutex; read-mostly config → RWMutex after profiling says so; ownership that never needs locking → pass the value through a channel instead. The reason all four are correct — what ordering each one establishes — is the memory model itself, in Scheduling and the Happens-Before Relation.