Analyzing Results and Bottlenecks

qa-performance-analysis-optimization-correlation -->

Your load run is green for eight minutes and then p95 climbs from 310 ms to 1,400 ms and stays there. The client knows when; it cannot know why. Explaining the "why" requires evidence from inside the system, captured on the same clock as the client numbers, and the discipline of collecting both together is the entire skill of performance analysis.

Start with what the client already gives you. From the k6/JMeter results you have per-sample rows: timestamp, label, duration, status. From Load Scripting with k6 you also exported thresholds and a summary. What you do not have is the server's opinion of that window. Collect it in parallel, with sampling fine enough to see a 30-second incident:

pidstat -p $(pgrep -f 'app.jar') 1 900   > server-cpu.txt   # %user/%system, per second
vmstat 1 900                              > server-mem.txt   # free, swap, context switches
iostat -xz 1 900                          > server-io.txt   # await, %util per disk

If the service runs in Kubernetes, the equivalent is one kubectl top pods per minute plus the workload's own metrics — but do not rely on dashboards alone: a Grafana panel summarizes, it does not retain the per-second resolution an incident review needs, so capture raw output alongside it.

On the database, three queries do most of the work:

-- PostgreSQL: what was actually slow during the window
SELECT query, calls, mean_exec_time, max_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time * calls DESC LIMIT 10;

-- connection saturation
SELECT state, count(*) FROM pg_stat_activity GROUP BY state;

-- was an index skipped?
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;    -- look for Seq Scan + shared read blocks

Then align the timelines. The three clocks that must agree are the load generator's, the application host's, and the database's — skew of even 30 seconds will send you hunting the wrong incident. Practical guardrails: run NTP on every box, use the same time zone in every dashboard, and mark the incident with an exact timestamp (14:20:07 UTC) when you write the report. If you have an APM or OpenTelemetry tracing, this is where spans pay for themselves: a trace started at 14:20:07 shows the request's time split by service and by database call, so the slow leg is named rather than inferred.

Reading the aligned graphs is pattern matching against a short list. CPU-bound? pidstat shows one core pinned at 100 % while others idle — a single-threaded bottleneck (JSON serialization, a lock, one event loop). Memory-bound? Look for swap activity and, on the JVM, GC logs showing pause frequency rising with the heap; the classic signature is latency stepping up in sawtooth with GC cycles. Queue-bound? Throughput holds while latency climbs — remember Little's Law: if rps is flat and p95 doubled, concurrency in flight doubled, which means something is queueing. Flat throughput plus rising latency is a queue, not a compute shortage, and the queue is where you look next: a pool, a lock, or a downstream dependency. One more: network or disk-bound systems show high await in iostat with idle CPU — often an uncached query pulling megabytes per request, or logging synchronously on the hot path.

One method warning that separates seniors from juniors: never analyze a production spike with metrics collected after it ended. Set the collection up before the run (or leave passive monitoring always-on), because the counters you needed — pool wait time, per-query latency, cache hit ratio — only exist for windows somebody was measuring.

qa-performance-analysis-optimization-signatures -->

Four bottleneck classes produce most of the p95 regressions you will ever investigate, and each leaves a signature that is hard to fake. Learning the signatures turns "the API is slow" into a ten-minute diagnosis.

The N+1 case deserves its own paragraph because it is the regression most often caught by a performance test rather than a functional one. A page that loads 20 orders and then issues SELECT * FROM line_items WHERE order_id = $1 per order turns 1 query into 21 — invisible at 50 rows, catastrophic at 20,000 rps. The database metrics show calls growing linearly while mean_exec_time stays flat, which is the tell: high call count with innocent per-call latency. Functional tests never see it because they check correctness, not volume.

The pool signature contains a counter-intuitive lesson interviewers like. The reflex "the pool is exhausted, raise maximumPoolSize" often makes things worse: a PostgreSQL server that was comfortable serving 20 connections per pod does not scale to 200 — each connection costs memory and scheduling on the DB, so total load rises and latency climbs further. Diagnose why connections are scarce (slow queries holding them, or a missing pool.release in an error path) before changing the number, and consider that the correct ceiling may belong to the database, not the app.

Two more signatures round out the list, both visible only on the server side. Thread-pool saturation: application CPU is moderate, active request threads equal the pool maximum, and the accept queue shows growing wait time — the app is not slow, it is full; fixes are more concurrency (carefully), less work per request, or shedding load early with 429s. JVM GC pressure: p95 rises in a sawtooth correlated with GC log entries ([GC pause (G1 Evacuation Pause) 204M->140M(512M), 0.21 secs]), heap usage climbing to the ceiling and then dropping — the fix is heap sizing or, more often, allocating less per request.

Fan-out has a numeric rule worth carrying: sequential downstream calls add their latencies (5 services × 60 ms p95 ≈ 300 ms of unavoidable floor), so a service-level p95 target under that floor is arithmetically impossible. Parallelize, or make the target honest.

When the signature is named and the evidence is in hand, the remaining work is communication — the subject of the last chapter here, and the part that decides whether anything gets fixed.

qa-performance-analysis-optimization-slo-report -->

A bottleneck analysis that ends with "p95 was 1,400 ms" changes nothing. Two artifacts turn measurement into action: a service level objective the team commits to, and a report that makes the fix the cheapest option on the table.

An SLO is a number, a window, and a budget. It names a service level indicator (the metric — p95 latency of POST /checkout measured at the edge), the target (under 400 ms), and the window (a rolling 30 days):

SLI:   p95 latency, POST /checkout, all production traffic
SLO:   p95 < 400 ms over a rolling 30 days
Budget: 99.9 % compliance → ~43.2 minutes of violation per 30-day month
        (99.95 % halves it to 21.6 min; 99 % allows 7.2 h)

The error budget is the shared language between QA and engineering: a healthy budget means ship freely; a spent budget means the next sprint goes to performance work. That is why the SLO belongs in the pipeline as a gate — the same threshold discipline from Load Scripting with k6, applied to production percentiles or to a release-candidate load run against the baseline.

The report itself should fit two pages. Structure it so a rushed lead can act on it without asking you a question:

## Performance findings — checkout service, build 2f91c3 (2026-09-14 load run)
1. Summary     p95 rose 310 ms → 1,400 ms at 120 rps; SLO (400 ms) breached above ~75 rps.
2. Evidence    k6 run at 120 rps, results attached; pg_stat_statements: orders_by_customer
               940 ms mean × 1,200 calls/s; EXPLAIN shows Seq Scan on line_items(order_id).
3. Root cause  Missing index after the 2026-09-10 schema migration; N+1 from OrderRepository.
4. Options     A) CREATE INDEX CONCURRENTLY — ~2 h, no code change, expect p95 < 350 ms
               B) Batch the N+1 — 2 dev-days, also cuts DB CPU ~40 %
               C) Cache page responses — masks the query cost, adds staleness risk
5. Recommendation  Do A now (this week), schedule B next sprint; re-run the baseline after both.
6. Validation  Re-run same k6 plan; gate: p(95)<400 ms, error rate < 0.5 %, exit code 99 on breach.

Notice what makes it decisive: every claim carries its artifact (run id, query, plan), each option names cost and expected effect, and there is a validation step that proves the fix — the same test that found the bug retires it. Recommendations without a re-test invite the same regression two migrations later.

One reporting rule keeps the report honest across releases: publish findings with the numbers you did not like. If the run also showed login at p95 250 ms against a 280 ms target — compliant but close — say so in a "watch list" line; selective reporting trains readers to distrust everything else in the document. And attach the raw artifacts (JTL, pg_stat_statements dump, report screenshot) as links, not attachments, so the report stays two pages while the evidence stays one click away for the engineer who disagrees.

Interviewers use this topic to test whether you can close the loop: how would you know you fixed it? Answer with the baseline comparison from Performance Concepts and Baselines — same plan, same data, threshold met, artifact stored. Once performance work is a closed loop, the roadmap moves to a different class of quality: how real people, including those using screen readers, experience the interface — starting with WCAG Accessibility Fundamentals.