CI/CD for ML Pipelines

Your fraud-scoring repo passes lint and 214 unit tests, so the pull request merges cleanly. Two days later production degrades: the upstream team renamed card_present to card_present_flag, your pipeline encoded every value as null, and no test caught it because in ML the data is an input, and code tests never see it. Software CI validates one artifact (the binary). ML CI must validate three: code, data, and the model that training produces from their combination. That is why an ML pipeline adds gates on top of ordinary tests instead of replacing them.

The full sequence, with what each stage actually blocks:

A GitHub Actions skeleton — the shape matters more than the vendor:

name: ml-ci
on: [push, pull_request]
jobs:
  code:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.lock && ruff check . && pytest -q
  data-check:
    needs: code
    steps:
      - uses: actions/checkout@v4
      - uses: iterative/setup-dvc@v2
      - run: dvc pull data/reference   # versioned reference snapshot
      - run: python -m checks.validate_schema data/live --reference data/reference
  smoke-train:
    needs: data-check
    runs-on: ubuntu-latest
    steps:
      - run: python train.py --subset 5000 --epochs 1 --expect-loss-below 0.80

Stage 2 is the highest-leverage minute in the pipeline. A schema or distribution check against the versioned reference snapshot catches the renamed column before you spend a GPU-hour. Express the expectations as code reviewed in pull requests — column presence and types, null_fraction < 0.05, value ranges, allowed category sets, and a population-stability check against the reference for high-impact features. Great Expectations and TFDV-style tools both reduce to exactly this: named expectations with a pass/fail result the pipeline can block on.

The smoke test (stage 3) exists to prove the training entrypoint still runs end to end with the current dependency lock: imports resolve, data loaders iterate, loss decreases, the artifact writes to disk. Its assertions stay deliberately weak — loss moved, no NaN, metric above a generous floor — because its job is catching breakage in minutes, not judging quality. Quality is stage 5's job, and quality judgments need the full dataset.

Full training should not run on every pull request: a 45-minute A10G job at 20 PRs per day burns roughly 15 GPU-hours daily for changes that 99% do not affect training. Run it on merge to main or nightly, cache the data snapshot through DVC so repeated runs skip the download, and gate it behind a run-full-train label for when an author genuinely needs it.

The pattern from Online and Batch Serving tells you what the deploy stage must produce — an endpoint whose latency and freshness guarantees hold — but nothing tells you whether the new model deserves to replace the old one. Stage 5 does, and it is the subject of the next chapter.

Stage 5 asks one question: does the candidate deserve to replace what is running? Your candidate reports AUC 0.8712 against the champion's 0.8697, and someone on the pull request writes "ship it." That comment is the bug. A 0.0015 gap means nothing until you know the noise floor, and promoting on point estimates alone ships regressions roughly as often as it ships gains. The promotion gate must be executable code, not a person eyeballing a dashboard — dashboards drift with attention, pipelines do not.

Encode four criteria, all versioned in params.yaml so changing a threshold is a reviewed diff:

1. Minimum effect size. Require a relative improvement of at least +0.3% (a business-set floor). Below that, shipping adds risk for no measurable gain — keep the champion even if the candidate is technically higher.
2. Statistical confidence. A paired bootstrap over the holdout: resample the same items with replacement 1,000 times, compute the metric difference each round, and require the 95% confidence interval to exclude zero.

import numpy as np

def delta_bootstrap(y, cand, champ, metric, n=1000, seed=42):
    rng = np.random.default_rng(seed)
    idx = np.arange(len(y))
    deltas = np.empty(n)
    for i in range(n):
        s = rng.choice(idx, len(idx), replace=True)
        deltas[i] = metric(y[s], cand[s]) - metric(y[s], champ[s])
    lo, hi = np.percentile(deltas, [2.5, 97.5])
    return float(deltas.mean()), float(lo), float(hi)

# ship only if mean delta >= MIN_EFFECT and lo > 0

3. Slice guardrails. No critical segment — country, device, merchant category — may regress by more than 1 point. A global gain that hides a 4-point loss on your largest segment is a guaranteed rollback, and aggregate metrics will never show it.
4. Non-metric gates. p99 latency within the budget from serving design, artifact size within GPU memory, and calibration error within tolerance if scores drive business thresholds.

Once the gate passes, promotion mechanics limit blast radius: the registry moves the version Staging → Production with the gate's evidence attached — metrics JSON, data snapshot ID, git SHA — then rollout proceeds shadow (log-only, zero risk, compare offline) → canary (5-10% of traffic for 30 minutes, automatic rollback if error rate or latency breaches) → full. The registry semantics come from Model Registry and Versioning; the gate is what makes the stages mean something.

None of this is reproducible without versioned inputs, which is where DVC earns its place. Git tracks code and small configuration; DVC tracks the heavy artifacts through pointer files in git and content in object storage:

dvc add data/train.parquet            # .dvc pointer committed, data to S3 remote
dvc stage add train -p params.yaml \
  -d data/train.parquet -d src/train.py -o models/model.pkl python src/train.py
dvc repro train                       # reruns only when inputs actually changed
dvc push                              # artifacts to the shared remote
dvc metrics show --json               # what the gate script reads

Two checkouts of the same commit reconstruct the exact training run: same data snapshot, same parameters, same code. That matters the day a gate result is disputed — "the candidate saw different data" becomes a command, not an argument. It also powers dvc metrics diff across runs, so the candidate-versus-champion comparison is a first-class, reviewable object rather than numbers pasted into a pull request comment.

A subtle failure the gate must absorb: nondeterminism. GPU kernels and shuffled loaders mean two identical runs differ in the last decimal place, so write every threshold as a tolerance (delta >= +0.3%, score within 0.002) — never exact equality — and seed the training run so the variance stays inside the tolerance you wrote. A gate that fails on rounding teaches people to bypass it, and a bypassed gate is worse than no gate.

Code, data, and models are versioned — and then the environment that runs them is managed by hand, which is where releases quietly rot. Infrastructure as code (Terraform, Pulumi, CloudFormation) versions the training cluster and the serving stack alongside application code: instance types, autoscaling limits, resource requests, ingress rules, and the model-server image digest. A capacity change becomes a reviewed pull request, so "who moved us to a smaller GPU?" is a git log instead of a meeting.

resource "aws_ecs_service" "inference" {
  name            = "fraud-inference"
  task_definition = aws_ecs_task_definition.inference.arn
  desired_count   = 2
  # the task definition pins the image by digest, not :latest —
  # the digest is what actually ran through canary
}

Pair that with a pinned model version from the registry and every deploy is reproducible: same task definition, same image digest, same model artifact. Five failure modes account for most real-world ML CI breakage, and each has a specific fix:

1. Data not versioned. Training silently pulls a different snapshot than the one the evaluation gate scored, so the gate's verdict describes a model nobody will run. Fix: DVC pointer files in git, the snapshot ID recorded in the registry entry, and a pipeline that fails when the pointer and the training job disagree.
2. Nondeterminism masquerading as regression. Shuffled loaders, unordered reductions, and GPU kernels make exact reproduction impossible on some hardware. Fix: seed every RNG, set deterministic flags where available, and write gates as tolerances — Δ >= +0.3%, never score == 0.8712. A gate that trips on rounding gets disabled by the first frustrated team, and a disabled gate is theater.
3. Full training on every pull request. The cost math turns hostile fast: 45 minutes on an A10G at 20 PRs per day is about 15 GPU-hours daily. Fix: PR runs are limited to lint, unit tests, data validation, and the smoke stage; full training fires on merge or nightly; DVC caching means unchanged data is never re-downloaded; the run-full-train label exists for the exceptions.
4. Floating references everywhere. :latest image tags, unpinned base images, and unpinned inference dependencies mean the artifact that passed canary is not the artifact running next week. Fix: pin image digests, commit a dependency lockfile, and make the lockfile part of what the evaluation gate validated — an untested dependency bump is an unreviewed model change.
5. No rehearsed rollback. The canary alarm fires at 02:00 and nobody knows the command. Fix: treat (image digest, model version, task definition) as one deployable unit, keep the previous unit one command away, automate the trigger — canary breaches SLO → previous task definition plus a registry stage flip — and rehearse it in a game day. Rollback you have never run is a hypothesis.

Add two operating rules. First, pin desired_count and autoscaling bounds in the same Terraform module so capacity intent is reviewable; a hand-scaled console change disappears from history. Second, deploy-stage health checks must include a prediction probe — send a synthetic request and assert a plausible score — because containers can be healthy while the model fails to deserialize, and only a real inference catches that.

The pipeline's last act is handing the new version to the systems that watch it: post-deploy checks on latency, error rate, and input distribution. That is where the lifecycle continues in Production Monitoring and Drift Detection.

In an interview, compress the whole story to one sentence: four things are versioned (code, data, model, infrastructure), gates run in a fixed order, and promotion means an executable gate plus a canary plus a rehearsed rollback. That sentence is the difference between "we use GitHub Actions" and a system you can defend.