Performance Concepts and Baselines

qa-performance-concepts-baseline-percentiles -->

Your checkout API has a monitoring panel that proudly reports average response time: 180 ms, yet support keeps getting tickets saying checkout "hung for several seconds." Both statements are true, and the gap between them is the single most common performance mistake in interviews and in production: the average is the wrong summary for latency.

Latency is a distribution, and a distribution is summarized with percentiles. Take every request in the window, sort them by duration, and read the value at each position:

1,000 requests sorted ascending
   950 × 120 ms     → p50 = 120 ms, p95 = 120 ms
    45 × 800 ms     → p99 = 800 ms
     5 × 6,000 ms   → worst case, gateway timeout
average = (950×120 + 45×800 + 5×6,000) / 1,000 = 180 ms

The average is 180 ms and five real people waited six seconds. Reporting only the mean hides exactly the users who file tickets. The percentile set you should always report together:

Interviewers ask "how would you measure this API?" to see whether you reach for p95/p99 instead of the mean. Say percentiles first, then throughput, then error rate.

Throughput is the second pillar: requests completed per second (rps) or transactions per second (tps) over the window. It only means something next to latency, and Little's Law connects them:

concurrent requests in flight = throughput × latency
40 rps × 0.5 s = 20 requests in flight

That equation is a sanity check you can run in your head during a load test. If throughput stays at 40 rps but p95 climbs from 500 ms to 2 s, concurrency has quadrupled to 80 — your thread pool, database pool, or upstream is queueing, and the queue is now the bottleneck, not the CPU. It also converts user counts into a load model: 200 users who each issue a request every 5 seconds is 40 rps, not 200 concurrent.

Error rate is the third pillar and the one teams forget under load. Count HTTP 5xx, gateway timeouts, connection resets, and — critically — correctness failures: a 200 response that returns a stale price is a failed request. Timed-out requests must be counted as errors at the timeout threshold (say 6 s), never quietly dropped as "slow successes," or your error rate reads 0 % while users see failures. That is why performance runs need the same response assertions functional tests use, a point Test Reporting and Dashboards already made for functional runs: a green run with wrong answers is not green.

Hold the three metrics in one sentence when you report: p95 latency, throughput at that latency, and error rate — for example "240 rps at 310 ms p95 with 0.02 % errors." Any one of them alone is a number without meaning, and the follow-up to this topic, Designing JMeter Test Plans, is where you start producing those triples on demand.

qa-performance-concepts-baseline-test-types -->

Teams routinely run one long test at one load level and call it "performance testing." It answers exactly one question, usually the least interesting one. Each test type is defined by the shape of the load curve and the question it answers — pick the question first, then the shape.

Load is the release gate: it validates the number in your SLA. The classic mistake is testing at exactly 100 % of expected peak with zero headroom — real traffic has bursts, so test at peak plus 20–30 %.

Stress deliberately walks past the ceiling. You are watching the shape of collapse. A healthy system at 150 % load returns 429 with Retry-After and holds p95 at a degraded but finite value. An unhealthy one shows latency compounding without bound (the queue grows: remember Little's Law — latency climbs, so concurrency climbs, so the queue grows) and connections timing out. The follow-up question interviewers add: does the system recover when load returns to normal, or is it wedged? Run the stress test long enough to include that recovery ramp.

Spike is a different physics problem. Autoscalers poll every 30–60 s, TCP backlog is finite, and a database pool of 20 connections does not grow on demand. A 10× burst in 2 seconds tests whether the front door absorbs or drops: expect connection errors if the buffer is too small, or a latency cliff if every request queues behind 20 pool slots.

Soak finds the bugs no short test can. Real numbers from runs that mattered: a JVM heap growing 40 MB/hour until java.lang.OutOfMemoryError at hour 6; a connection pool leaking one socket per minute so hour 5 sees HikariPool-1 - Connection is not available, request timed out after 30000ms; a log volume filling at 1 GB/hour. Every one of those passes a 20-minute load test. If you only have budget for two tests, run load on every release and soak nightly — a soak failure at 03:00 is far cheaper than one at 09:00 in front of customers.

Notice what the table shares: every row needs a stable load generator and repeatable numbers, which is what the next topic builds — Designing JMeter Test Plans turns these four shapes into reusable thread-group configurations.

qa-performance-concepts-baseline-establishing -->

"Is checkout slower than last sprint?" is unanswerable unless someone wrote down what slower meant before the change. A baseline is a stored, trusted set of measurements — p50/p95/p99, throughput, error rate — taken on a known build under known conditions, against which every future run is compared. Without one you are comparing vibes.

The procedure that produces a trustworthy number:

1. Fix the environment. Same instance types, same DB size, same data volume. Note the commit SHA and the build number alongside the results — a baseline with no version attached is a rumor.
2. Fix the data. Seed an identical dataset (say 50,000 orders, 200,000 users) and reset it between runs. Query plans change when tables grow; a baseline taken on an empty database measures an application nobody runs.
3. Warm up, then repeat. Discard the first run (JIT compilation, connection-pool fill, cold caches), then run at least 3 times and record the spread.
4. Store the artifact, not just a number — the JTL/CSV and the generated report belong next to the trend data you already publish from Test Reporting and Dashboards.

A small baseline table for one service might look like this:

Three rules keep those numbers honest:

• Account for noise. Staging is shared. Expect roughly ±5–10 % run-to-run variation on p95; set your alert threshold above it (a common choice is baseline × 1.2), or you will spend weeks chasing the background job a colleague triggered at 14:00. If a run breaches the threshold, re-run before filing a bug — two consecutive breaches is a regression, one is weather.
• Never compare across environments. A staging box that is half the production size will produce p95 numbers 2–4× worse, and the ratio does not transfer. Either baseline production passively (real traffic percentiles from your APM) or keep staging sized proportionally and say so in the report.
• Compare like-shaped runs. Same virtual-user count, same ramp, same payload size, same think time. A baseline taken at 50 users compared against a run at 200 users measures load, not regression.

The comparison itself can be a five-line script rather than a person staring at two charts:

k6 run --summary-export new.json checkout.js          # candidate build
node -e '
  const base = require("./baseline/summary.json").metrics.http_req_duration.values;
  const cur  = require("./new.json").metrics.http_req_duration.values;
  const ratio = cur["p(95)"] / base["p(95)"];
  console.log(`p95 ${base["p(95)"]} -> ${cur["p(95)"]} (${(ratio*100).toFixed(0)}%)`);
  process.exit(ratio > 1.2 ? 1 : 0);'

That exit code is the whole point: the baseline stops being a document people consult and becomes the gate the pipeline enforces. Store the baseline artifacts per release — the summary JSON, the JTL, the commit SHA, the seed-data version — so that when someone asks "when did checkout get slow?", the answer is a bisect over stored baselines rather than archaeology.

One more placement decision matters: how much of the baseline comes from production. Staging baselines answer "did this build regress under this load model," while passive production monitoring answers "is the real p95 within the SLO the business promised." You want both — the staging gate catches the regression before release, and the production percentile confirms the promise after it. The next topic builds the first of those two instruments, turning this procedure into a repeatable JMeter plan: Designing JMeter Test Plans.