Parallel Runs and Cutovers

Compare before moving authority

A parallel run evaluates an old and a new system against equivalent inputs before the new system becomes authoritative. The goal is to find behavioral differences while rollback is still relatively safe. It is useful for high-risk migrations such as pricing, eligibility, data transformation, or reporting where the team can compare outputs and wants evidence beyond unit tests. It is not a guarantee of correctness: both systems can share the same flawed assumption, and the test data may not represent production edge cases.

Define what is being compared. Normalize nondeterministic fields such as timestamps, generated ids, or ordering when those differences are irrelevant. Compare business meaning, invariants, and required side effects rather than raw bytes. A divergence report should identify the input cohort, old result, new result, classification, severity, and owner. Do not dismiss every mismatch as harmless; categorize it as expected policy change, known limitation, defect, or unresolved behavior.

Shadow safely

For read-only calculations, send a copy of the input to the new implementation while the old result remains visible. The shadow result must not affect the user or trigger duplicate side effects. For writes, blindly executing both systems can charge twice, send duplicate notifications, or create inconsistent records. Use a side-effect-free evaluation mode, a sanitized replay environment, or an outbox and idempotency strategy designed for the migration. Keep sensitive data protected and limit access to captured production inputs.

Parallel execution consumes additional capacity and can alter the behavior being measured. Measure its CPU, memory, database load, and external calls. Do not point a shadow system at a production provider if requests mutate state. Sample traffic deliberately when full duplication would overload resources, and state which cohorts are excluded. Preserve correlation identifiers while avoiding personal or secret data in reports.

Set a cutover gate

Before the run begins, define the observation window, representative workloads, acceptable divergence classes, latency and error thresholds, data reconciliation rules, and decision owner. A gate should be specific: for example, no unresolved financial mismatches above a defined severity during a representative business cycle. Include rare but consequential edge cases through generated or replayed scenarios rather than relying only on traffic frequency.

Investigate divergences by tracing the same input through validation, defaults, state, time zones, rounding, and external dependencies. Fix a defect or explicitly decide to change a policy. Update tests so the mismatch cannot silently return. A low aggregate mismatch rate can still hide a severe category, so report distributions and impact rather than only one percentage.

Cut over and preserve a safe return

After evidence meets the gate, move authority in stages: internal users, a small cohort, a broader cohort, then full traffic. Monitor business outcomes, latency, errors, data consistency, and support signals. Keep a rollback path and test it before the cutover. Define who can invoke it, the threshold, and whether the action is a routing change, restore, or forward repair.

Rollback becomes harder after the new system accepts writes in a different schema or owns state the old system cannot read. Use an expand-and-contract migration, compatibility period, reconciliation, or explicit point of no return. Do not promise a simple switchback if data has changed irreversibly. Once the new system is stable and authoritative, stop the duplicate run, remove temporary traffic copies, retire the old implementation, and keep the evidence and decision record for future reference.

Compare a new pricing engine

Assume a legacy pricing service is being replaced. The old system remains authoritative while the new engine calculates a shadow quote for the same cart and customer context. Use a stable correlation id to pair the results. Define normalization for fields that are expected to differ, such as computation timestamp, but preserve exact comparison for currency, tax, discount eligibility, and total. Never charge a customer from both outputs during the shadow period.

Create a divergence report with counts and samples by category: rounding mismatch, missing promotion, stale customer tier, invalid currency, or a difference that is expected because policy intentionally changed. Track severity and business exposure. A one-cent difference on a low-value item may still matter at scale or under a contractual rule. Review samples safely and restrict access to customer data.

Validate coverage and performance

Run representative traffic from different regions, currencies, customer types, and promotion states. Add generated edge cases for rounding boundaries, expired offers, empty carts, and concurrent updates. Production traffic may not include rare cases during a short window. Confirm the new engine's latency and resource cost under both normal and peak load; running it alongside the old service doubles some work and can distort the benchmark.

For an unknown outcome, decide whether to fail closed, use the old result, or hold the request for review. During shadow mode, the old result remains user-facing, but the new result still needs a classification and owner. If the old service calls an external tax provider, do not duplicate that side effect simply to compare outputs. Use a deterministic fixture or provider sandbox with the same contract behavior.

Make the cutover reversible

Define a gate before changing traffic: required coverage, maximum unresolved high-severity mismatch, error and latency limits, and a representative observation period. If the gate passes, route a small cohort to the new engine and compare outcomes with the old service where safe. Expand in steps and pause automatically or operationally when a threshold is exceeded. Keep the current implementation available while the new path establishes stability.

Test rollback before the first cohort. Verify that requests routed back to the old service still see compatible data and that idempotency prevents duplicate writes. If the new service persists a different representation, synchronize or reconcile it before switching authority. Record the moment when rollback is no longer a simple routing action and what forward-repair plan applies afterward.

Close the migration

After the new path is fully authoritative, stop shadow traffic and remove comparison code, dashboards that only served the migration, temporary credentials, and the old service. Confirm no consumer still depends on legacy output. Keep the divergence report, gate result, and decision record as evidence, and update the architecture view and runbook. Leaving two active engines indefinitely preserves the cost and ambiguity the migration intended to remove.

As an exercise, inject a known rounding mismatch and verify that the report groups it correctly. Then simulate a new-engine timeout and a rollback after a partial cohort writes data. The system should expose the failure, keep the user-facing result well-defined, and show what reconciliation is required. Parallel runs are most valuable when they reduce uncertainty before authority moves, not when they merely produce a second set of logs.

Evidence quality and cost

Parallel execution provides comparative evidence but increases compute, database, and operational load. It may not be possible to run every input twice, and sampled traffic can miss rare high-impact behavior. Combine safe production shadowing with deterministic fixtures, historical replay, and targeted edge cases. Explain how the sample was selected and which behaviors remain untested.

The comparison itself needs a domain-aware oracle. Byte-for-byte equality is inappropriate when timestamps differ, while broad normalization can hide a real defect. Define comparison fields, tolerances, severity, and business impact before looking at results. Use privacy controls for captured inputs and remove data that is no longer needed after the evaluation window.

Cutover is a data decision too

Routing traffic back is only a rollback if the old system can safely process the current state. Schema changes, new writes, sequence allocation, external side effects, and generated identifiers can make the systems diverge. Plan compatibility and reconciliation before the first cutover, and label the point at which the strategy changes from rollback to forward repair. Practice that boundary with representative data.

Avoid a cutover based only on an aggregate success percentage. Segment divergence by severity, user group, currency, tenant, or operation. A rare but severe mismatch can matter more than many low-impact differences. Let the decision owner accept residual risk explicitly and state the conditions that would pause or reverse the rollout.

In an interview, describe identical input capture, safe shadow execution, domain-aware comparison, divergence ownership, a measurable gate, staged traffic movement, and rollback constraints. Explain why duplicate side effects are dangerous and how data ownership changes the return path. Finish with cleanup: temporary comparison paths and the old system should be retired once the new path is proven. This shows a parallel run is a bounded verification phase with a planned end state.

Sampling needs deliberate limits. A random sample can estimate common behavior, but it may underrepresent a rare transaction type with severe consequences. Stratify by operation, tenant size, region, or value band when those dimensions affect results. Use deterministic replay for rare edge cases and keep a record of the input set version. If production traffic is copied, redact or tokenize sensitive fields and enforce a retention deadline for the captured data.

The cutover owner should be able to explain the go/no-go result without relying on a dashboard screenshot alone. Summarize the observation window, cohort coverage, unresolved divergence, performance impact, and known blind spots. Let the domain owner accept a bounded residual risk explicitly. If the team cannot determine whether a mismatch is safe, it is unresolved rather than harmless. A review checkpoint after the first real traffic cohort can confirm whether shadow assumptions held under actual user behavior.

Rollback planning should include the data timeline. Before writes move, routing back may be enough. During a dual-read or replication stage, verify which copy is current. After the new system becomes the source of truth, define whether the old system can consume changes, whether a reverse migration exists, and how long it would take. State the point of no return in the runbook and require an explicit decision before crossing it. A forward-fix plan is a valid recovery strategy, but it is not the same as rollback and should be staffed accordingly.