Pairwise Combination Testing

Your web app must work on 4 browsers × 3 OSes × 3 viewport classes × 2 network profiles × 3 locale settings. Someone asks for full configuration coverage: 4 × 3 × 3 × 2 × 3 = 216 configurations, at a conservative 20 minutes of setup and execution each — 72 hours of machine time per build, and that is one feature area. Every configuration tester meets this wall, and the wall is made of a multiplication that has no upper bound: add device memory (3 values) and the space becomes 648. Full coverage is not slow; it is impossible, and the technique that replaces it is the single most valuable idea in configuration testing.

The idea rests on an empirical finding from the 1990s-2000s telecom and Microsoft field studies that has held up across industries: the large majority of configuration-relevant defects are triggered by the interaction of at most two parameters. A rendering bug that appears on Firefox + dark mode, a session bug on Safari + intermittent network, a layout bug on tablet viewport + RTL locale — these are pairs. When a three-way interaction defect does appear (say Safari + low memory + large upload), it is rare enough and specific enough that risk-based picks and targeted exploration cover it better than exhaustive enumeration ever could.

Pairwise (all-pairs) testing builds a suite in which every pair of parameter values appears together in at least one test case. The arithmetic drop is dramatic:

A 216-space reducing to roughly a dozen cases is not an approximation trick — it is combinatorial structure: pairwise coverage can always be achieved in the neighborhood of the product of the two largest parameters (12 = 4 × 3 here), regardless of how many more parameters exist. That invariance is why the technique scales: adding the sixth parameter grows the suite by a few rows, not by multiplication.

The generation is a constraint-satisfaction problem — choose rows so every value-pair occurs, prefer rows that also cover new triples — and it is solved by published algorithms (greedy with backtracking, IPOG in the academic tooling). You never do it by hand beyond three parameters; you use a generator:

Microsoft PICT:   model file → dozens of rows, seconds
python AllPairs:  pip install allpairspython → matrix from a dict

What the generator hands back is a matrix — one row per test case, one column per parameter, cells are chosen values — and reading that matrix is the skill: any pair of columns, scanned down, covers all their value combinations at least once. The caveat that separates real usage from toy usage is constraints: real systems forbid combinations (iOS + Chrome; 32-bit OS + 8 GB heap expectation; 'cash on delivery' unavailable in the express lane). All generators support constraint syntax, and honoring them is not optional — a generated suite full of impossible configurations wastes the hours you just saved. The constraints themselves are worth extracting from the developers, because a spec that lists invalid combinations is a spec nobody wrote down before you asked.

The difference between a useful pairwise suite and an academic one is the quality of the model you feed the generator. A worked example with PICT's model-file syntax for the configuration space above:

// model.pict — parameters with realistic value sets and forbidden combos
Browser : Edge, Chrome, Firefox, Safari
OS      : Windows, macOS, iOS, Android
Viewport: mobile, tablet, desktop
Network : fast, slow-3g
Locale  : en-US, de-DE, ar-RTL

// constraints — combinations that cannot (or must not) occur
[OS = iOS]   => [Browser = Safari];
[OS = Android] => [Browser != Safari];
[Viewport = mobile] => [Network != fast];   // mobile always emulated on throttled link

Three modeling decisions in that file carry most of the real-world value. Value sets come from telemetry, not imagination: if analytics say Safari-mobile is 18% of traffic and desktop-Firefox is 2%, the values reflect the user base, and 'desktop' can be split by memory class if bugs cluster there. Constraints encode what developers know but nobody documented — the iOS/Safari pairing is a platform fact, and extracting these constraints in a review meeting routinely surfaces disagreements about what the product even supports. The throttled-mobile constraint shows a product decision disguised as a parameter: 'mobile always on slow-3g' was chosen to test the realistic worst case; writing it down makes that choice reviewable — someone can object and add a fast-mobile row by hand.

Running PICT produces a matrix (simplified here):

 1  Edge     Windows  mobile   slow-3g   ar-RTL
 2  Chrome   iOS      tablet   fast      de-DE
 3  Firefox  macOS    desktop  slow-3g   en-US
 4  Safari   Android  mobile   fast      de-DE
 ...  (typically 12-20 rows for this model)

Verify by inspection the property you paid for: take any two columns and confirm every value combination appears in some row — browsers × OSes, viewports × locales, and so on. Then the practical finish work that separates generated suites from generated artifacts: each row needs oracle data (what the correct behavior even is on this configuration — a rendering-diff baseline, an expected API contract) and environment provisioning (how the row's device/OS/browser actually gets provisioned — device farm, VM snapshots, or physical devices). Pairwise solves which configurations to test; the surrounding engineering still decides how the results are judged, and teams that skip the oracle step end up with fast suites that execute thoroughly and verify nothing.

Maintenance is the last chapter of real usage: value sets drift (a browser ships a new major version — update the model, regenerate, diff the matrix). Keeping the model file in the repo next to the tests, regenerated on parameter change, means the suite tracks the product instead of fossilizing — the same living-document discipline as strategies and plans, one level down.

A final production concern: oracles for visual configurations. Rendering defects are configuration-specific by nature, and a row asserting only 'no JavaScript errors' misses the misaligned grid that appears only on tablet Safari. The practical answer is screenshot baselines per configuration class — one reference image per (browser, viewport) pair, with a reviewed tolerance — or, where baselines are too brittle, a structured visual checklist the tester runs against the row's output. Both keep the generated matrix honest: coverage without a stated expectation per row is exactly how teams end up with fast suites that execute thoroughly and verify nothing.

Pairwise is a coverage floor, not a test plan — and the professional move is layering: risk-picked combinations go above the generated base, because there are always a handful of configurations where failure is unacceptable and interaction depth beyond pairs is plausible. The layering has three tiers.

Tier 1 — risk-picked triples and critical pairs (hand-authored, always run). These come from the risk workshop and defect history, not from a generator: the combination of Safari + intermittent network + a saved-card checkout session is where session-resumption bugs live, and it deserves an explicit, named test even if pairwise also happens to include a similar row. The rule of thumb: if a combination's failure would block release or corrupt data, write it by hand, give it a name, and never let a regeneration delete it. Concretely, teams mark these as 'pinned rows' appended to the generated matrix.

Tier 2 — the generated pairwise matrix (regenerated per model change). This is the breadth floor, and its job is the long tail of interactions nobody would think to write: obscure value pairs that only a generator bothers to combine (Firefox + ar-RTL + tablet). Deleting pairwise because 'we test the important combos anyway' reintroduces exactly the blind spot the field studies found — the defect you did not think to combine.

Tier 3 — targeted deep exploration (human, occasional). For the highest-risk modules, an exploratory session deliberately walks three-way and four-way interactions the two lower tiers cannot enumerate: slow network + expired session + form resubmission + a background refresh — the multi-condition pileup that production actually experiences. This tier is also where you check the pairwise assumption itself: if your last three escapes were three-way interactions, the empirical justification has weakened for your product, and deeper coverage levels (threewise, a 2-4× suite cost) deserve reconsideration.

Measurement closes the system: tag every configuration-related escape with the tiers — was the failing combination in Tier 1? covered by the Tier 2 matrix (under-tested despite coverage — an oracle gap)? or in the Tier 3 blind zone (a modeling gap — add the parameter or the pin)? Over two or three releases this produces a defensible answer to the interview question that ends this topic: 'how do you decide how much configuration testing is enough?' The answer: exhaustive is impossible; pairwise is the justified floor because most configuration defects are pair-triggered; risk picks protect the named high-impact combinations; exploration covers the pileups; and escape data tells you whether your product breaks the pair rule and needs deeper coverage levels. Every clause of that answer is checkable, which is what makes it an answer rather than an opinion — and it caps the test design techniques of this phase with their shared philosophy: each technique buys specific defect classes at a known cost, and the buyer's discipline is matching purchases to risk. The next phase applies these same economics to the defect lifecycle, starting with Defect Lifecycle States.