Schema Evolution and Compatibility

Two engineers argue about whether a change is 'compatible' and both are right, because they are measuring different directions. Compatibility is not a property of a change — it is a property of a change relative to a direction of reading. Fix the vocabulary first, because every registry, every RFC, and every interview question hangs on it:

writer schema: the schema a producer used when it serialized the record
reader schema: the schema a consumer uses when it deserializes

BACKWARD compatible: a NEW reader schema can read data written by OLD writers
   ⇒ new consumers may read old data  ⇒ deploy new consumers FIRST, producers later
FORWARD compatible:  OLD reader schemas can read data written by a NEW writer
   ⇒ old consumers survive new producers ⇒ deploy producers FIRST... or never break them
FULL compatible:     both of the above

The direction that trips everyone: backward sounds like 'old code reads new data', and it is the opposite. Anchor it with the deployment consequence — backward compatibility is what lets you ship the consumer upgrade before the producer upgrade, which is the order you can always control within one release train. Forward compatibility is the harder promise (old binaries must tolerate data they have never seen), which is why registries often default to BACKWARD and why a system that must never break slow consumers asks for FORWARD or FULL.

Work a concrete matrix, because this is where interviews test understanding:

Two subtleties that separate a strong answer from a passable one. First, JSON Schema's default additionalProperties handling decides forward compatibility for you: strict validators reject unknown keys (forward-breaking), while lenient ones drop them (forward-safe). Second, a table like this is only a hypothesis: compatibility is proven against real retained payloads — encode old data with the new schema and replay it through the new reader — never against developer intuition.

The wire format mechanics explain why any of this works at all. Avro serializes without field names in the payload — it writes fields positionally against the writer's schema, which the registry identifies by schema id in the record envelope; the consumer resolves that id, gets the writer's schema, and applies the reader's schema field-by-field to map old names/positions to new ones. Protobuf writes field numbers, so name changes are invisible on the wire, and unknown fields are preserved-but-skipped by default — that is Protobuf's built-in forward tolerance. JSON with no schema id at all gives you neither guarantee, which is exactly why untyped JSON events become a liability once consumers outside your team exist.

Interview formulation: define both directions in terms of reader-versus-writer versions, give the deployment order each permits, and name one edit that is backward-safe but forward-breaking (add-with-default). If you can explain why — id-or-field-number driven deserialization — you have answered the follow-up before it is asked.

The goal of a compatibility policy is to let teams deploy independently. That only happens if a known set of edits is boringly safe — and the rest is gated. Here is the edit list, with the reasoning, ordered from 'ship it' to 'never alone':

SAFE (backward, usually full):
  + add a NEW optional field with a default
  + add a new event type / new topic
  + widen a numeric type (int → long/decimal) where readers accept it
  + add a new enum value IF readers treat unknown values as 'other'

NEEDS A ROLLOUT PLAN:
  - remove a field          → mark deprecated first, stop reading, then remove in v2
  - narrow a type (long→int) → truncation risk for out-of-range values
  - rename a field          → additive: add new + duplicate old, migrate, drop old

NEVER IN PLACE (breaking):
  × change a field's type incompatibly (string → object, int → string)
  × reuse a removed Protobuf field number
  × reorder/reuse positions in an Avro union
  × tighten validation (making an optional field required)

Each safe row has a mechanism worth stating. Add-with-default is safe backward because the reader fills the gap, and safe forward only if old readers are configured to ignore unknown fields — check that config, do not assume it. Additive renames (add fullName, keep writing name, migrate readers, stop writing name months later) convert a breaking rename into two safe steps — the same expand/contract discipline as a database migration. Enum additions need the 'unknown → OTHER' rule because an old consumer with an exhaustive switch on PaymentMethod will throw on the new value; if your code has switch without default, your schema policy must forbid enum additions until that code is fixed.

Numbers to know for Protobuf, since it comes up constantly:

message Payment {
  reserved 3;              // old field 'discount' removed — number NEVER reused
  reserved "discount";     // name also reserved so it can't return as a new field
  int64 amount_cents = 1;  // numbers are the contract; rename freely
}

Reusing number 3 means old producers' bytes get misread as the new field's type — silent corruption, not an error. Reserving is the difference between a schema history and a landmine. Avro's equivalent trap is union position: appending a new type to a union is safe; changing the order is not.

Governance makes the list real. A registry with compatibility enforced per subject (TopicNameStrategy, RecordNameStrategy, or None) rejects unsafe registrations at push time — but the check that actually catches regressions runs in CI: compile the new schema against the previous version, replay last week's captured payloads through the new reader, and run consumer tests against producer fixtures from the previous release. Teams also need a deprecation ritual: mark the field deprecated in the schema, monitor whether any consumer still reads it (registry lookup by field or a decoder-side counter), announce removal in a changelog with a date, then remove in a versioned event. Skipping the monitoring step is how 'unused' fields turn out to have three consumers you never met.

A governance detail interviewers appreciate: compatibility is per subject, not per platform — two topics on one Kafka cluster can have different policies, and one topic carrying two event types needs RecordNameStrategy so each type's evolution is checked independently. When a change genuinely cannot be made compatible, you do not relax the policy — you version the contract, which is the final chapter's subject.

The change: customer: string becomes customer: {id, name} — a structure change that fails both directions. The business wants it in three weeks and consumers cannot all deploy in the same window. Compatibility policy says no; the deadline says yes. The resolution is that breaking changes do not go through the compatibility gate — they go through versioning, in one of three standard shapes.

Shape 1 — dual publishing (the safe default). Deploy a producer that writes the new event to orders.v2 while continuing to write orders.v1. Consumers migrate at their own pace on orders.v1 → orders.v2; when lag on v1 hits zero and the last consumer is gone, stop v1. Costs: double write volume, and every consumer must eventually switch — you are buying time, not coordination-free migration.

Shape 2 — expand/contract (parallel change), for in-place evolution. The three phases, which are the same discipline as zero-downtime database migrations:

EXPAND     1. consumers accept BOTH old and new shapes (readers tolerant,
              defaults applied) — deploy this everywhere first
           2. producers start emitting the new shape; registry enforces
              backward compatibility so old readers still work
MIGRATE    3. wait for lag/drain; monitor which readers still see old shape
CONTRACT   4. remove old-shape handling in a later release — only after
              nothing emits or requires it

The failure mode to name in an interview: skipping phase 1. That is the Friday incident — producer ships first, consumer that cannot parse it crashes, topic backs up. Expand/contract exists precisely to make every intermediate state a state some version of every component already understands.

Shape 3 — a transformation at the boundary. A compactor, a stream-processing job, or an adapter consumer converts v1 → v2 on the fly (Kafka Streams/Connect with SMTs, or a small service). Use it when dual publishing is unacceptable (volume) or when old producers will exist for months (mobile clients!). Mobile is the canonical unpublishable-producer case: your 'old version' is in customers' pockets for a year, so the API must be forward-tolerant by policy and structurally versioned (/v2/ paths, Accept headers, event typeVersion fields).

Operational checklist for the migration window:

Also decide what happens to records already in the log when a consumer encounters a schema id it cannot resolve (registry outage, subject deleted): fail fast to a deserialization DLQ with the id preserved, never skip-and-continue silently — silent skips become unexplained data gaps in someone else's dashboard weeks later.

Interview summary to deliver: classify the change; if additive, ship it behind compatibility enforcement; if breaking, version the contract with dual publish or expand/contract, monitor drain, then contract — and gate all of it in CI with payload replay. From data contracts, the roadmap turns to choosing the store itself: Relational vs NoSQL Selection.