Estimation and Coverage Metrics

The plan asks for your estimate on testing the saved-card feature. The two bad answers are a single number produced instantly ('three days') and a number produced defensively ('hard to say, maybe a week?'). Both fail the same way: they hide the uncertainty instead of modeling it. Two techniques handle estimation honestly, and interviewers ask for both by name.

Three-point estimation forces three numbers per activity instead of one:

optimistic (O)  = 1.5 days   # everything works: build stable, data ready, no blocked deps
most likely (M) = 3.0 days   # normal friction: some retests, one environment hiccup
pessimistic (P) = 6.0 days   # realistic bad day: rework, blocked env, requirement change

PERT mean      = (O + 4M + P) / 6 = (1.5 + 12 + 6) / 6 ≈ 3.25 days
PERT stddev   = (P - O) / 6 = 0.75 days

The formula matters less than the conversation it forces. Producing O requires asking 'what does a perfect run look like?' — which surfaces blockers early (does the test environment exist?). Producing P requires asking 'what typically goes wrong here?' — which surfaces the retest loop and environment flakiness that single-point estimates systematically forget. Reporting the result as '3.25 days ± 0.75' communicates uncertainty honestly; the ± is the part stakeholders remember when planning the sprint.

Summing PERT means across the plan's rows gives the total (the 4.7-day figure from Building a Test Plan came from exactly this arithmetic). But PERT assumes you can decompose the work — and for a whole release, you often cannot. That is where historical velocity acts as the sanity check: the team's last six sprints recorded how many QA-days of planned work were actually consumed. If velocity says this squad reliably completes 4.0 QA-days per sprint and your decomposition totals 4.7, one of the two numbers is wrong — either the plan is optimistic about scope, or the velocity is unrepresentative. Experienced estimators present both numbers and the discrepancy: 'decomposition says 4.7, our history says 4.0 — I recommend trimming E2E scope or the number will trim itself.'

Use-case point estimation is the third method worth naming: score each use case by driver complexity (transaction count, interface count, algorithmic complexity), sum the weighted drivers to a use-case point total, and multiply by a productivity factor calibrated from past projects. It fits portfolio-level estimation (twelve features across a quarter) better than sprint-level work, where three-point decomposition is faster and more precise.

A closing discipline: track estimate vs actual for every cycle, not to grade anyone but to calibrate. Reviewing last quarter's estimates against actuals usually reveals one systematic bias — the tester who always estimates low because they forget environment flakiness, or the team that estimates high to build in slack. Two cycles of feedback corrections beat any formula, which is why the technique matters less than the habit of closing the loop.

For interviews, practice narrating an estimate live: given a feature brief, decompose it into test activities, assign O/M/P with reasons, sum the means, and compare the total against an assumed sprint capacity. Examiners listen for three things — that you include the retest loop in P, that you state the ± range instead of a single number, and that you name what you would cut if the total exceeded capacity. Candidates who can do that arithmetic out loud, on a whiteboard, without hedging, have demonstrated the estimation habit itself, which transfers to any methodology the company uses.

'What is your coverage?' is an ambiguous question until you name the denominator — and the two answers mean completely different things. Requirement coverage asks: what fraction of specified behavior has a verified test? Code coverage asks: what fraction of implementation lines/branches did execution touch? Teams get into trouble when a number from one column is used to imply the other.

Requirement coverage comes from the traceability link (requirement or acceptance criterion → executed, passing test):

requirement coverage = requirements with ≥1 passing executed test / total in-scope requirements × 100

For release 4.12: 24 in-scope requirements, 23 covered → 95.8% — and the uncovered one (REQ-104, admin force-expiry) is named, not averaged away. This is the number that belongs in exit criteria because it answers the business question: what did we promise that nobody checked? Its weakness is granularity: 'covered' hides whether one shallow happy-path case or nine scenarios verify the requirement, so the metric pairs best with the risk bands — a high-risk requirement's single test should not count as comfortably covered.

Code coverage comes from instrumentation during test execution — line, branch, function, and condition coverage, reported by tools like JaCoCo, Istanbul, or coverage.py:

branch coverage = branches taken / branches available × 100    (82% on the payment module)

What it proves: which code the suite executed. What it does not prove: that the suite verified anything. A test that calls a function and asserts nothing moves coverage as much as a thorough one. The famous illustration: printing every element of a list hits ~100% line coverage of the printing logic while testing none of its edge behavior (empty list, huge list, nulls). Coverage of 80% with no uncovered hot spots is a healthy suite; coverage of 95% with the missing 5% being the error-handling paths is a trap metric.

The practical policy experienced teams adopt: use code coverage as a gap-detector, not a target. Set a floor per module (say 70% lines) to force conversations about untested areas, then read the uncovered diff — the new or changed lines the suite never touched — which is far more actionable than the aggregate. Percentage targets without diff-reading get gamed: teams add assertions-free tests to push the number up, which is metric-gaming at its purest.

Both coverage views close through the same door: risk-weighted interpretation. A release reporting 100% requirement coverage and 78% branch coverage can be excellent (the uncovered branches are dead error paths) or alarming (the uncovered branches are the retry logic in the payment service). The number opens the conversation; the risk band of the uncovered area decides what it means. That is why coverage belongs in the same report as the risk matrix, never in a slide alone.

For interviews, the question 'requirement coverage vs code coverage?' is a favorite discriminator. Answer with the two definitions, one formula each, the caveat that code coverage does not prove verification, and the policy (floor plus uncovered-diff reading, no naked percentage targets). Candidates who only know code coverage from CI badges typically stop at the definition — the policy sentence is what distinguishes you.

Coverage says what you tested. The quality metrics say how well the testing worked — and two of them run the length of your career because they measure the thing everyone cares about: did defects get found before users did?

Defect escape rate (leakage) tracks where defects were found:

escape rate = defects found in production / (defects found in production + defects found pre-release) × 100

release 4.11: 3 production defects, 57 pre-release → 3/60 = 5.0%
release 4.10: 6 production defects, 44 pre-release → 6/50 = 12.0%

The trend line (12.0% → 5.0%) is the story stakeholders want: detection moved upstream. But the raw percentage invites two distortions, so pair it with context. Severity mix: three escaped typos and three escaped double-charge bugs produce the same rate and completely different meanings — always report escape rate by severity band. And denominator drift: a quiet release with six total defects can show a wild percentage on tiny numbers; annotate the count alongside the rate.

Defect detection percentage (DDP) is the same data viewed from the quality of the pre-release phase: DDP = pre-release defects / all defects × 100 (release 4.11: 95%). It has a useful sibling analysis: detection rate by phase — what percentage of each release's total defects were caught at requirements, unit, integration, system, UAT, or production. If 80% of defects consistently surface at system testing, that says your upstream levels are thin; shifting the curve left is the measurable form of 'shift-left'.

Defect density adds size normalization:

defect density = defects found per KLOC (or per story point, or per requirement)

KLOC density suits maintenance codebases with stable counting; per-requirement or per-story-point density suits agile teams where size counting is already estimated. Its use is comparative: this release at 4.2 defects/story-point against the 3.5 baseline flags something changed — new team member, new service, rushed requirement — before quality does.

Now the traps, because every published metric becomes a target, and bad targets get hit:

The interview-ready move is to refuse the vanity metric politely and offer its replacement with a one-line reason: 'raw defect counts reward finding trivial bugs; I would report severity-weighted counts plus escape rate, which measures what actually reached users.' That answer demonstrates metric maturity — the rarest QA skill — because it shows you understand both the measurement and the incentive system it creates. The dashboard you build from these metrics is what turns status reporting from spreadsheet archaeology into a live conversation, which is the final topic of this phase.