Online and Batch Serving

Your team is about to put a fraud model inside the checkout path. The API's end-to-end SLO is p99 200 ms, product wants the score in the same response the customer sees, and nobody has written down a latency budget. Write the budget first, because in this decision the latency SLO, not your preferred serving stack, decides the architecture.

Add up what happens inside that 200 ms before the model gets a single millisecond:

Inference gets roughly 60 ms p99. If profiling says 180 ms, no endpoint tuning rescues the plan: you either move computation out of the request path, shrink the model, or renegotiate the SLO. Say which one you are doing out loud — that is the serving decision.

Online serving mechanics are simpler than they look. One process loads the model into memory at startup and then answers REST or gRPC calls; loading per request would add 200-500 ms and is a non-starter. gRPC over HTTP/2 with protobuf typically cuts 30-50% off serialization versus JSON REST and multiplexes many requests over one connection, while REST wins when your proxy, tracing, and on-call tooling already assume HTTP/1.1. Either choice bills you a warm replica around the clock — an endpoint idling at 5 requests per second still pays the instance's full hourly rate. That idle cost is the seed of the economics we settle in Serving Trade-offs: Latency and Cost.

Three patterns place computation relative to the request:

1. Compute at request time. The model runs on every call. Required when the score depends on request context — the items in this cart. You own the tail latency.
2. Precomputed lookup. The score or embedding is computed offline and stored by entity key; the request does one key lookup in 1-3 ms. Correct for per-user or per-product values that change slowly — an affinity score refreshed nightly is the textbook case.
3. Feature-store lookup + live model. Heavy features (30-day aggregates) are materialized offline, fetched online in a couple of milliseconds, and inference still runs synchronously. You eliminate the dominant cost — feature computation — and accept the refresh interval as your freshness bound.

The practical rule: anything derivable from data older than a few minutes belongs in the precomputed path; only genuinely request-specific state must be computed live.

Two operational details finish the online story. First, cold starts: a new replica must pull the image and load weights — 30 to 90 seconds for a mid-size model — so autoscaling cannot rescue a sudden spike; you size for peak or keep a minimum replica count warm. Second, capacity is not latency: adding replicas raises throughput but leaves per-request inference time untouched, which is why the budget above is per-request and cannot be solved with a horizontal scale-out. Spiky traffic compounds the cost problem, since every warm replica bills full price during the troughs.

Each pattern fails differently, and interviewers probe exactly there. Precompute goes stale and needs a freshness SLA; request-time compute blows the p99 budget under load; the hybrid can silently disagree with training if offline and online feature code diverge. Record which pattern a model uses, its budget, and its freshness requirements in the version you promoted from Model Registry and Versioning — the deployment gate needs those numbers to enforce later.

Every night at 01:00 your team re-scores 40 million credit accounts and writes the results back to a warehouse table. Nobody is waiting on this job — analysts read the scores at 08:00. Millisecond latency does not matter here; throughput, cost, and reproducibility do, and those three properties are what batch scoring buys you.

The arithmetic explains why batch dominates this volume. Row-wise inference at 4 ms per row would take 40,000,000 x 4 ms = 160,000 seconds, about 44 hours on one process. Vectorized inference in 4,096-row chunks at 20,000 rows per second finishes 40M / 20k = 2,000 seconds — 33 minutes, or about 9 minutes across four workers. Now spread the same work over 24 hours as an online endpoint and you need 40,000,000 / 86,400 = 463 requests per second sustained, holding a GPU hot all day for a job that takes minutes when batched. Batch is not just simpler; it turns the same work from a permanent replica into a bounded, billable run.

The three inference patterns differ along latency, traffic shape, and cost profile:

A production batch job must honor four guarantees, and each one is a lesson learned from a job that did not:

1. Idempotency. Re-running the same run leaves the same table state. Write results into a partition keyed by (run_date, model_version) and overwrite it — appending blindly means a retry doubles the row count and nobody notices until a report is wrong.
2. Lineage. Every output row carries model_version and data_snapshot_id, so a wrong score traces back to the exact artifact and dataset that produced it.
3. Checkpointing. Partition by entity ranges; when the job dies at minute 24, resume from the last completed partition instead of restarting 33 minutes of work.
4. A late-data policy. Decide before launch what happens when the upstream partition lands at 02:10: wait for it, run partial and backfill, or fail loudly. Silent partial scores are worse than a missed run.

The contract in code — an Apache Airflow-style task; the scheduler matters far less than the guarantees:

@task(retries=2, retry_delay=timedelta(minutes=10))
def score(snapshot_date: str) -> None:
    model = registry.load_production()          # pinned version, never "latest"
    for part in entity_partitions(snapshot_date):
        df = read_partition(snapshot_date, part)
        df["score"] = model.predict_proba(df[FEATURES])[:, 1]
        df["model_version"] = model.version
        overwrite_partition("credit_scores", snapshot_date, part, df)
    publish_manifest(snapshot_date, model.version)

The manifest is the piece teams skip. Downstream consumers read the manifest instead of the raw table, so they never see a half-written day, and the morning freshness check becomes trivial: does a manifest exist for today with the expected model version? — an assertion you can alert on the moment it fails.

Batch also reshapes economics. Because a retry is cheap, you can run on spot instances and burst across preemptible GPUs, dropping cost per score by an order of magnitude versus a warm online replica. What you pay instead is staleness: the score is exactly as old as the last completed run. Whether that staleness is acceptable for each feature is the question the next chapter answers.

Your recommender needs avg_transaction_value_90d. Computing it from raw events means scanning millions of rows — impossible inside a 40 ms request budget. Meanwhile marketing wants scores to reflect new activity within a minute, not tomorrow's batch. Neither a pure online nor a pure batch pattern fits, so you build the hybrid: compute heavy features offline, serve them online, refresh them incrementally.

The mechanism has three moving parts:

1. Materialize the expensive feature offline — nightly batch job writes avg_transaction_value_90d to an online store (Redis, DynamoDB, or a feature store's serving layer) keyed by entity ID.
2. Refresh incrementally where a nightly value is too old: a streaming consumer updates the aggregate as each event arrives, moving freshness from hours to seconds without ever touching the request path.
3. Fetch at request time in 1-3 ms by key, then run the model synchronously on the assembled features.

Every feature in this design needs a declared freshness SLA: the maximum acceptable age of its value. Alert when age exceeds the SLA — a stalled consumer or missing partition produces confident wrong scores, not errors, which is exactly why freshness monitoring belongs beside latency and error rate (Production Monitoring and Drift Detection formalizes the thresholds).

Streaming inference is a related but distinct pattern, and interviews often ask you to separate them. An event stream (Kafka, Pub/Sub) triggers scoring on each event or micro-batch, and the score is written to a store the application reads — the model never serves an HTTP request. Latency lands between 100 ms and 5 seconds: far too slow for a synchronous checkout call, perfect for re-ranking a feed the moment a user acts. Consumers scale with lag rather than with a replica count you guess in advance.

Consistency is where hybrids break, in three specific ways:

• Train/serve skew. Offline training computes avg_transaction_value_90d with one SQL query while the streaming updater maintains it with different logic. Fix it structurally: one shared definition — feature-store transforms or a common library used by both jobs — plus a contract test that compares offline and online values for a sample of entities on every merge.
• Point-in-time correctness. A training row must see the feature value as it was at the label time, not today's value; otherwise tomorrow's information leaks backward into last month's label and your offline metrics are fiction. Feature stores exist precisely to provide point-in-time joins against historical snapshots.
• Misses at read time. The key is absent for a cold entity. Pick the behavior before launch: a default value, a TTL-bounded fallback, or degrading to a model that does not use the feature. Whatever you choose, log the miss rate so you know how often the fallback fires.

A decision table keeps the conversation honest:

When an interviewer asks "how would you serve this model?", answer in a fixed order: state the SLO, classify each feature's freshness requirement, choose the pattern per component, then name the fallback. That sequence shows you derive the architecture from constraints instead of reaching for a favorite framework.

Serving is only half the lifecycle. Once an endpoint exists, every change to it — new features, new weights, a different image — has to reach production without breaking the guarantees above, which is the job of CI/CD for ML Pipelines.