Serving Trade-offs: Latency and Cost

You are asked to serve a 120M-parameter encoder at ~250 QPS average (600 QPS peak) with p99 ≤ 100 ms. "Which hardware should we use?" sounds like a taste question. It is arithmetic, and the unit that matters is not the hourly sticker price — it is cost per million requests:

$ per 1M = hourly_price / (sustained_qps × 3600) × 1,000,000 = 277.78 × hourly_price / qps

Measured options for the same model, same load profile:

Check one row: the T4 at 400 qps moves 400 × 3600 = 1.44M requests per hour, so 0.53 / 1.44 = $0.37 per million. The 16-vCPU box has the lower hourly sticker ($0.48 vs $0.53) and still costs 4.5× more per request ($1.67 vs $0.37), while its p99 already grazes the SLO. Hourly price is a decoy; per-request price at measured throughput is the decision.

Then traffic enters, and it flips answers. The same T4 at 20 qps realized traffic: 20 × 3600 = 72,000 requests/hr → 0.53 / 0.072 = $7.36 per million — worse than any CPU row, because a warm GPU bills identically whether it is serving or idle. The break-even against the CPU option that meets the SLO (a constant $1.67/1M once you autoscale instances to load) is where the always-on T4's realized cost equals it: 277.78 × 0.53 / q = 1.67 → q ≈ 88 qps. Below roughly 88 qps of steady traffic, an always-on GPU is the expensive option; above it, the GPU wins per request at every point on this table. Spiky traffic makes it worse — an autoscaler cannot save you from a cold start of 30-90 seconds (image pull plus weight load) during a burst you did not see coming.

Two mitigations, in order of preference: scale to zero for internal or batch-triggered endpoints (accept the cold start or pre-warm on a queue signal), and batch the offline work so bursts become scheduled runs rather than warm replicas.

The GPU-hours math extends to everything else you run:

• Idle serving: 1 replica × 730 h/month × $1.01 = $737/month for one A10G that serves nothing at 3 a.m.; two replicas for HA = $1,474. At 500 qps average traffic (1.31B requests/month), that HA pair costs $1,474 / 1,314 = $1.12 per million — utilization, not instance price, decides whether GPU serving is cheap.
• Training: a nightly 1.5-hour fine-tune = 45 GPU-hours/month ≈ $45. A hyperparameter sweep of 40 configs × 3 folds × 20 min = 120 fits = 40 GPU-hours ≈ $40 — and spot pricing (~70% off) drops it near $12, because a failed sweep trial is cheap to retry.

Interview framing: state your assumptions (qps, SLO, model size), show the formula, quote the per-million numbers, then name the two caveats — break-even traffic and cold-start latency. That answer shows you have run the bill before, not just the benchmark.

The capstone you just shipped ends with an endpoint and a p99 budget; the hardware decision above is where that budget starts costing money — End-to-End Capstone Project.

Serving one request at a time wastes a GPU: kernel launches and memory-bandwidth stalls dominate, so a single 4 ms forward pass leaves the device mostly idle. Group concurrent requests into one tensor and throughput climbs — this is dynamic batching (Triton, TensorFlow Serving) or continuous batching (vLLM). The cost lands on tail latency, because every request in a batch waits for the batch to finish, and someone has to wait for the batch to fill.

Measured shape for an image-classification-class model on one A10G:

(overhead = network + serialization, a flat ~5 ms.) Read every p99 row as queue wait + compute + overhead. Moving from batch 1 to 32 lifts throughput 11.7× (65 → 760 rps) while p99 triples (16 → 48 ms) — usually an excellent trade under a 100 ms budget. Batch 64 buys only +29% more throughput (760 → 980) and more than triples p99 again (48 → 169 ms). That is the shape of the curve: batch size is set by the p99 budget, not by the GPU-utilization graph, because utilization keeps climbing long after the SLO is dead.

Two waits compose the queue term, and they fail differently:

1. Fill delay — waiting for enough requests to arrive so the batch can form. You cap this directly with the server's max-queue-delay setting.
2. Saturation backlog — waiting because the server is already busy. No delay setting bounds this one; waiting grows roughly as 1/(1 − ρ) where ρ is utilization, so at 96% utilization a batch endpoint's tail explodes while its dashboard looks "efficient."

The sizing rule follows: run replicas at 60-70% of measured plateau throughput. At batch 32 the plateau is 760 rps; 0.65 × 760 ≈ 500 rps per replica, so a 600 qps peak needs two replicas — three if you require one to survive a failure with headroom. Driving one replica to 95% saves an instance and buys you the page.

# NVIDIA Triton model configuration
dynamic_batching:
  preferred_batch_size: [8, 16, 32]
  max_queue_delay_microseconds: 40000   # cap fill delay at 40 ms
  max_available_queue_size: 64          # bound the backlog per scheduler

Set the delay from the budget, not from folklore: worst-case fill delay + worst-case compute at your chosen batch + overhead must stay under the p99 SLO. With a 100 ms budget, batch-32 compute (22 ms) plus overhead (5 ms), a 40 ms delay cap leaves ~33 ms of margin for variance and network. Raise the cap to 200 ms and you have bought throughput the SLO can never collect on.

Three traps worth naming before an interviewer does:

• Client-side batching is not server-side batching. A client assembling its own groups spreads arrivals smears queueing across the stack, and you cannot bound the wait end-to-end. One coordinated server queue is the design that lets you reason about p99 at all.
• Retries amplify load. If downstream callers retry at 50 ms while your p99 is 80 ms, a saturation event triggers retries that add offered load exactly when you are near ρ = 1 — the collapse feeds itself. Use retry budgets with jitter and deadlines below your SLO.
• Batching changes per-request semantics for anything stateful. Streaming scores through a batch endpoint is fine; mixing requests that mutate server state into one batch is not.

The economics connect back directly: 760 rps on one $1.01/hr A10G is 1.01 / (760 × 3600 / 1e6) = $0.37 per million requests, while the same card unbatched at 65 rps delivers 1.01 / (65 × 3600 / 1e6) = $4.32 per million — 11.7× more expensive per request for identical hardware. Batching is the single biggest cost lever in serving, which is why the next chapter asks what happens when you also shrink the model.

If hardware and batching still miss the budget, shrink the model. Three levers — precision, distillation, pruning — each buy speed and memory for a measured accuracy cost. The discipline that keeps this honest: decide the accuracy floor first, then buy the cheapest speed that stays above it. An unmeasured 0.4 points is a gamble, not a trade.

A BERT-base-class encoder (110M parameters) on one A10G, batch 16:

(artifact size is pure arithmetic: 110M × 4 / 2 / 1 bytes; footprint includes CUDA context and batch workspace.)

What the rows say, in decision order:

• fp16 is nearly free. 2× smaller, ~1.9× faster, −0.02 points. Take it on any hardware with tensor cores unless you have a measured numerical reason not to.
• int8 post-training quantization is the real trade. 4× smaller, 2.7× faster (24 → 9 ms), −0.44 points. Quantization-aware training simulates int8 rounding during fine-tuning and recovers 0.32 of those points (−0.12 total) for the price of one extra training run — call it 3 GPU-hours ≈ $3. Against int8's $0.31 → $0.19 per-million serving cost at fixed hardware, a workload of 500 qps (1.31B requests/month) saves ~$158/month. The payback is immediate; the only question was whether −0.12 clears your floor.
• Memory sets a concurrency ceiling you feel as throughput. The 2.1 GB fp32 footprint fits ~6 concurrent batch-16 slots on a 16 GB card (leaving headroom for the runtime); the 0.7 GB int8 footprint fits ~18. 3× the concurrency on the same GPU, on top of the 2.7× per-forward speedup — instance memory is a cost constraint, not a storage detail.
• Distillation attacks the FLOPs, not the bytes. A 6-layer student trained on the teacher's soft targets runs at 8 ms fp32 (3× the baseline's speed) for −1.25 points. Compress further by quantizing the student: int8 student ≈ 4 ms p50, roughly 6× faster than baseline for about −1.5 points — acceptable only if your floor says so.

Before you shrink anything, consider routing: run a cheap model first and escalate only the hard cases.

def score(x):
    p = fast_model.predict_proba(x)[:, 1]
    out = p.copy()
    uncertain = (p > 0.35) & (p < 0.65)      # small model is confident → keep it
    if uncertain.any():
        out[uncertain] = strong_model.predict_proba(x[uncertain])[:, 1]
    return out

With 80% of traffic resolved by the fast path: mean latency = 0.8 × 6 + 0.2 × 45 = 13.8 ms versus 45 ms flat — 3.3× faster, with accuracy within 0.1 points of the strong model, because the escalated 20% contains essentially all the errors. Cascades monetize the same insight as quantization: spend accuracy-shaped compute only where accuracy is at risk.

Pruning (structured sparsity, head removal) is the third lever and the one to approach last — its speedups only materialize with hardware kernels that exploit the sparsity pattern, so an unstructured 50% prune often returns no latency at all. Measure, don't assume.

Interview framing: give all four numbers — the accuracy floor, the measured delta, the latency gain, and the dollars saved per million — then say which lever you would pull first and why. Interviewers for ML system design are checking one thing: whether every quality trade you make is quantified against a floor someone agreed to, which is exactly the skill the next round rewards: ML System Design Interviews.