Event-Driven Architecture Patterns

The first design review of a new integration goes badly in a predictable way. The order service publishes SendEmail to a topic. The email service consumes it. Marketing then wants a receipt email, fraud wants a notification, and the loyalty service wants to credit points — but all three need different information, and the order service now has to know that email, fraud, and loyalty exist. Every new consumer forces a change in the producer. The fix is not a broker feature; it is a sentence: an event records what happened, not what someone should do about it.

COMMAND (instruction, couples producer to one consumer's API):
  { "action": "sendEmail", "template": "receipt", "to": "a@b.com" }

EVENT (immutable fact, describes the past in the producer's own language):
  { "eventType": "OrderPlaced.v2", "eventId": "01HX...", "occurredAt": "2026-03-04T10:22:11Z",
    "orderId": "o-7712", "customerId": "c-88", "totalCents": 4299, "currency": "USD" }

The shift buys several things at once. Discovery: any team can subscribe later without asking the order service's permission — receipts, fraud scoring, analytics, and point accrual all read the same fact. Stability: the producer publishes in its own domain vocabulary, so it changes the event when its model changes, not when a consumer's template does. Replay: because facts are immutable and timestamped, a new consumer can reprocess history (Message Queues and Brokers covers the log mechanics). Auditability: a stream of facts is a story you can query, whereas a stream of commands is a stream of obligations.

Facts also carry built-in hygiene rules that interviews probe:

• Past tense, no directions. PaymentReceived, InventoryReserved, UserRegistered — if the name contains a verb the consumer must perform, it is a command wearing an event's clothes.
• Immutable. Correcting a fact means emitting a new fact (PaymentRefunded), never editing the old one — which is precisely what makes the log replayable.
• Self-contained enough. Include identifiers, amounts, and version; consumers should not have to call back to the producer's database to interpret the event (that reintroduces the coupling you removed).
• Not a database dump. Publishing the entire row means every internal column change breaks consumers. Publish the fields the contract promises; enrich in the consumer if needed.
• Correlation and causation IDs. Carry correlationId (one business flow) and causationId (what triggered this) so you can reconstruct a chain across services during an incident.

The honest counterpoints, because event-driven has real costs: debugging is harder — a request's effects now scatter across consumers, so correlation IDs and distributed tracing are mandatory, not optional; ordering is only per key, so consumers must tolerate out-of-order arrival unless they are strictly partition-scoped; and dual-write hazards appear the moment a service writes its database and publishes an event in two separate steps (the fix is the transactional outbox, next chapter). A useful interview line: events give you loose coupling at design time and explicit work at runtime — the runtime work being idempotency, ordering tolerance, and tracing.

Get the fact/command distinction right first and most of the rest of this topic follows mechanically. The next two chapters handle the machinery: contracts that let producer and consumer evolve separately, and consumers that survive redelivery.

At 16:00 on Friday a producer deploys what looks like a harmless change: customer.name (string) becomes customer.fullName (object). The schema registry has no rule configured, so the first consumer deserializes and throws TypeError: Cannot read properties of undefined on 100% of events. The topic backs up for six hours. A schema registry exists to make that deploy impossible rather than detectable, by validating every serialized payload and every registered version against a compatibility policy before anything reaches the wire.

The compatibility modes are the core vocabulary:

Walk the direction carefully — it confuses everyone once: backward compatibility is about the consumer being new. If a new consumer can interpret old events (a superset schema with defaults for added fields), you can deploy the consumer first, then the producer, safely. Adding an optional field with a default satisfies both directions in practice and is the safest evolution path in production. Removing a required field, changing a field's type, or renaming a required field breaks something for someone — those need a new topic or a versioned event type.

The mechanics worth knowing for interviews:

producer ──serialize(schema id in envelope)──► broker
              │                                   │
              ▼                                   ▼
        registry: is this   ◄──── registry: resolve id ── consumer
        version allowed?           to its schema definition

Avro embeds a schema ID; Protobuf uses field numbers (wire-compatible renames!);
JSON Schema validates structure but costs bytes unless you use $ref discipline

Protobuf's field-number rule is a favorite interview detail: you may rename a field freely (numbers are the contract) but never reuse or renumber an existing number — a removed number must stay reserved forever. Avro's union-of-schemas reader model gives the analogous rule: add fields with defaults, never change a field's position in a union.

Deployment discipline makes the modes real. Expand/contract (parallel change): deploy consumers that read both old and new shapes → deploy producers to emit the new shape → remove support for the old shape in a later release. Skipping steps one and two is how teams break contracts even with FULL compatibility configured. Registry config also has to be per-subject: Kafka's subject-level strategy (TopicNameStrategy vs RecordNameStrategy) decides whether compatibility is enforced per topic or per event type — a fact that matters when two event types share a topic.

One more production concern: validation must run in CI, not only at runtime. The registry rejects a bad registration at publish time, but discovering it during a deploy at 16:00 Friday is nearly as bad as the outage. Register schemas against a mock broker in the pipeline, run the consumer suite against the previous producer's payloads, and the Friday scenario stays theoretical. Where contracts and long-lived workflows meet — a multi-service flow whose steps cannot upgrade independently — you are in The Saga Pattern.

At-least-once delivery means your consumer will see the same event more than once: a crash between processing and offset commit, a visibility timeout expiring mid-job, a rebalance moving a partition to a member that re-reads uncommitted records (Message Queues and Brokers has the ack mechanics). If processing PaymentReceived twice adds 4299 to a balance twice, the system's correctness now depends on a race you do not control. An idempotent consumer makes duplicate delivery invisible by giving every event a stable identity and making the effect conditional on having never seen that identity.

The canonical implementation, in SQL:

BEGIN;
INSERT INTO processed_events (event_id, processed_at, payload_hash)
VALUES ('01HX...', now(), 'ab12...');   -- fails on duplicate ⇒ COMMIT nothing
-- only if the insert succeeded:
UPDATE accounts SET balance = balance + 4299 WHERE id = 'c-88';
COMMIT;

The unique constraint on event_id is doing the work: the database's primary-key guarantee is your deduplication, and the effect and the marker commit in the same transaction — so a duplicate either finds the row (skip) or the system crashes with neither marker nor effect (retry cleanly). Key design decisions around it:

• What is the idempotency key? The natural choice is the event's immutable eventId (or Kafka's (topic, partition, offset)). Never a hash of mutable payload — two legitimately distinct events with equal payloads are not duplicates.
• How long do you keep markers? Deduplication windows must exceed maximum redelivery time; a TTL is acceptable when retention plus retry limits bound the gap (Kafka 7-day retention + max 3 retry attempts ⇒ a marker TTL of days is sufficient). Unbounded marker tables grow forever, so pick the window deliberately.
• Where does dedupe live — consumer or database? In-memory caches lose their contents on the crash that causes redelivery; only durable state (the DB constraint, or an idempotent external API) protects against that. This is the follow-up question that separates a real answer from a rehearsed one.
• Idempotency vs exactly-once. Idempotency is at-least-once transport + an effect that tolerates repetition. Kafka's exactly-once semantics (idempotent producers + transactions) covers produce-and-commit atomicity within Kafka, but the effect in your database still needs the marker if it can be reached by other paths.

The other half of the story is getting events out of the database atomically. Writing your row and publishing an event are two systems: if the commit succeeds and the publish fails, downstream never learns; if you publish first and the commit fails, downstream acts on something that did not happen. The transactional outbox solves it — insert the event into an outbox table in the same transaction as the state change, then a relay (Debezium reading the table's WAL stream, or a poller publishing and marking rows) ships it to the broker. Consumers see exactly the committed events; the producer's dual-write window disappears. Reading that same WAL stream on the consumer side (CDC) gives the mirror-image guarantee: projections are updated only from committed changes.

Interview checklist for this topic: state that at-least-once makes duplicates normal, show the INSERT ... ON CONFLICT DO NOTHING-style guard with a unique key, name the durable store for markers, mention the outbox for the producer side, and — for extra credit — note that ordering still matters: dedupe stops double-effects but not out-of-order effects, so state transitions need version checks (WHERE version = :expected) or per-key serial processing.