The Saga Pattern for Transactions

Booking a trip touches three services: Flights.reserve (your booking service), Hotels.charge (payments), and Insurance.issue (partner API). The instinct is a distributed transaction: begin, prepare on all three, commit. That is two-phase commit, and it fails in production for reasons that are structural, not implementation-specific: a participant that blocks in PREPARED holds its locks until the coordinator resolves — and if the coordinator dies mid-commit, no one else can decide the outcome. A database tolerates that because the coordinator is a local, durable log; across services it means an insurance partner's connection pool, a hotel's retry logic, and your booking row all sit locked behind a process that may need manual intervention.

Add the classic failure and the argument closes:

2PC participant list includes an external partner API:
  PREPARE → partner holds the offer, returns 'prepared'
  coordinator crashes before COMMIT
  ⇒ partner holds inventory; your DB holds locks
  ⇒ every retry from either side risks double-charge or double-book

Timeouts do not save you: a prepared participant CANNOT unilaterally abort,
because another participant may already have committed.

A saga replaces the global lock with a plan: an ordered sequence of local transactions, each in one service, each committing on its own. If step 3 fails, the saga runs compensating transactions for steps 2 and 1 in reverse order — not by rolling back a database (nothing was globally rolled back), but by issuing new business operations that neutralize the earlier effects.

BookTrip saga:
  1. BookingService.reserve(seat)          LOCAL TX commit
  2. PaymentService.charge(card, 420)       LOCAL TX commit
  3. InsuranceService.issue(policy)         ✗ fails (partner 503)
  compensate: PaymentService.refund(420)    LOCAL TX commit
              BookingService.release(seat)  LOCAL TX commit
  ⇒ each step briefly held real state — the saga's cost is visible intermediate states

That last line is the essential trade, and it is the interview answer to 'how is this different from ACID?': a saga provides atomicity only at the end of the whole sequence; intermediate states are observable by other services. Between steps 2 and 3, the money is captured and the policy does not exist — if an outside reader queries at that instant, it sees a charged booking that is not yet insured. If some reader must never observe that state, you either keep that step in one local transaction (co-locate the data) or add a status field (pending → confirmed) so readers treat intermediate states as not-yet-final rather than wrong.

Practical requirements for every saga design, stated up front:

1. Idempotent steps — a timeout where the charge actually succeeded means the retry will fire again (Event-Driven Architecture Patterns).
2. A durable log of saga state — which step, what happened, so a crash of the coordinator resumes rather than guesses. The log is the process's memory; an in-memory workflow list is not a saga.
3. Timeouts as first-class outcomes — a step that never returns must be treated as failed-and-uncertain, triggering either a query-back (GET /charge/{id}) or a compensating path after a decision policy.
4. A decision on uncertainty — if refunding might double-refund, you query state first; if re-issuing might double-issue, you use idempotency keys. Compensations are ordinary operations, subject to the same at-least-once laws.

Where sagas shine: cross-service workflows with independent ownership of data. Where they do not: the moment you would rather have a lock — 'two users must not both take the last seat' is better served by a conditional local update (UPDATE inventory SET held = held + 1 WHERE seats > 0) inside one service than by a saga racing to compensate. Keep the critical check local; saga everything around it. The next chapter designs the compensations themselves.

Developers hear 'compensating transaction' and picture rollback. The distinction matters: rollback returns the system to its prior state as if nothing happened; a compensation is a new business operation that produces a state the business accepts as equivalent — and it may be lossy, delayed, or irreversible. You cannot un-send an email, un-cancel a reservation that a competitor just bought, or un-authorize a fraud flag already consumed by a model. Compensations are designed per step, and their names reveal their semantics.

Work the example: an order saga completes reserve inventory, charge card, ship label, then issue invoice fails. Compensations fire in reverse:

inverse                              NOT an inverse
────────────────────────────────     ────────────────────────────────
void invoice request                 delete the invoice row
refund charge (new transaction!)     un-charge (impossible)
release reservation                  un-reserve (someone may have taken it)

Two properties separate correct compensation design from hopeful thinking:

Compensations are themselves transactions. refund writes a new row, moves real money, and can fail. So the saga needs its own failure handling for its failure handling: if refund fails, retry with backoff forever-or-until-policy, because money is captured and cannot be silently abandoned. This is why saga coordinators persist step outcomes durably before proceeding — a crash between charge and compensation must not lose the knowledge that a charge exists.

Compensations must be idempotent and guard against duplicates. The refund will be retried; give it an idempotency key derived from the saga and step (saga-7712:charge-refund) so a duplicate refund request is a no-op at the payment provider. Where a true inverse does not exist, you compensate with a forward correction: refund rather than void, re-list the item rather than restore the cart's exact state, notify a human queue when the model's fraud flag needs clearing.

The other half of compensation is what happens when compensation itself cannot run — the ugly but real category:

Sagas also carry a semantic trap interviewers love: two sagas that touch the same resource in opposite directions can interleave and produce dirty reads of each other's intermediate states. Saga A holds a seat while saga B, seeing no seat, compensates — but A then fails and releases it, leaving B compensated for a seat that was never actually taken. The mitigation is the same as always: make the critical decision a conditional local update, keep the saga's intermediate state marked as tentative, and treat retries as new saga instances rather than re-entering a half-finished one.

Finally, name the audit value, because it is real: a saga's durable log is a complete, ordered record of every step and compensation — which is exactly what an incident review or a financial reconciliation needs. Systems that implement this well include Cadence/Temporal (durable workflows with timers and retries), AWS Step Functions (managed coordination), and lightweight DB-backed state machines. Now the structural question: keep the sequence in one place or spread it? That is orchestration versus choreography, next.

Two teams implement the same order saga. Team A runs an orchestrator: a single OrderSaga service that calls inventory, then payments, then shipping, and invokes compensations when a step fails. Team B uses choreography: each service reacts to events — InventoryReserved triggers the charge, PaymentCharged triggers shipping — and failures are handled by each consumer deciding whether to emit a compensation event. Six months later Team A complains the orchestrator is a bottleneck; Team B cannot draw their saga on a whiteboard without two markers and an eraser. Both complaints are the predictable costs of their choice.

The comparison, honestly stated:

Rules of thumb that hold up in practice: orchestrate when the sequence changes often or has strict ordering, side effects, or timers (a booking flow with 30-minute payment deadlines, retries, and a partner SLA is miserable as choreography — you need durable timers and a place to see 'saga 7712 stuck at step 3'). Choreagate when the participants are stable, the ordering is loose, and each service's reaction is self-contained (post a comment → send notification, update feed, index search — three independent reactions, no compensation required between them).

The choreography failure mode deserves a specific name: distributed coupling without a controller. When flow control is implicit, debugging means grepping five services' logs for a correlationId, and changing the sequence means coordinating deployments of services owned by other teams. Compensations are worse — the 'what undoes this?' knowledge must be encoded as events each consumer chooses to handle, and a consumer that misses the compensation event leaves the system in a stale intermediate state indefinitely. Mitigations if you choose choreography: an explicit SagaFailed/compensation contract topic, correlation IDs on every event, and a reconciliation job that detects sagas stuck in intermediate states.

The hybrid most production systems land on: orchestrate the critical path, choreograph the notifications. The orchestrator owns reserve → charge → ship because those steps are ordered and must compensate; the resulting OrderConfirmed event fans out to analytics, email, and recommendations, which need no coordination among themselves. Also note that an orchestrator can itself publish events (it is a client of the bus, not a replacement for it), so the pattern choice is per-flow, not per-system.

Interview framing: state both axes — where does flow control live and who can explain the sequence during an incident — then pick using change frequency and compensation needs, and show you know the hybrid. Contracts are the next pressure point: Schema Evolution and Compatibility is how these long-lived event chains survive independent deployments.