Model Cards and Documentation

The launch review is Tuesday. The approver has your 40-page evaluation report, two minutes of attention, and one question: can we ship this, and for what? What they actually open is the model card - the one-page label that lets a consumer judge fitness for purpose without reading your code. The format comes from Mitchell and colleagues at Google (2019) and has become the default expectation in regulated and platform teams.

Two sections carry the decision. Section 2 defines where the model is allowed to operate, and section 7 proves you went looking for the failure modes you claim not to have. The rest is context a reviewer scans in thirty seconds.

Keep the card derived from the run record rather than pasted from a slide: cite the MLflow run id or the Experiment Tracking Tools report next to each number so a reader can pull the live source. Pasted numbers rot the first time somebody re-runs an evaluation, and a card that disagrees with the run of record is worse than no card at all - reviewers will trust one of the two, and it will not be yours.

A skeleton that survives review:

# Model Card - fraud-risk-v3
Owner: risk-ml@corp | Version 3.2.1 | Released 2026-04-18 | Supersedes v2.8
Run of record: mlflow run 3f9a1c2 | Eval data: eval-2026-Q1 (n=120,411)

## 2. Intended use
Primary: rank transactions into the human review queue; analyst decides.
Out of scope: automated declines, thin-file applicants, non-en-US traffic.

## 7. Quantitative analyses
Aggregate: AUC 0.914 [0.901, 0.926], precision@100 0.35
By slice: see table; ages 18-21 recall 0.68 (n=214) - flagged limitation

Three working rules. Store the card in the repo as docs/model_card.md and change it by pull request, reviewed like code - a stale card is worse than no card because reviewers act on it. Cap it at one to two pages; length is not diligence, and every section beyond the eighth is a place for errors to hide. Add a supersession line so a consumer knows which card governs which model version, and close with monitoring: the metric watched, the cadence, and the rollback trigger.

Section 4 deserves one specific warning: the metric list should match the decision the model makes, and each metric needs a one-line plain-language gloss. A fraud-ranking model reporting AUC, precision@100, and recall at the operating threshold is telling a coherent story; the same model reporting AUC, BLEU, and perplexity is telling you the metrics were copied from another project. Reviewers do check, and a metric that cannot be tied to a decision is treated - correctly - as decoration.

Section 3, factors, is the map that makes section 7 computable: it lists the axes along which behavior might differ before any results are seen. Choosing factors is an act of judgment about where the model can be stressed - traffic source, device class, language variety, merchant category - and stating them in advance is what separates a pre-registered analysis from a post-hoc search for slices that happen to look fine.

Start with arithmetic, because it is the argument that convinces skeptics. Overall recall is 0.93 on a population that is 80% group A and 20% group B, so 0.93 = 0.8 x r_A + 0.2 x r_B. If group A scores a perfect 1.00, group B can sit as low as (0.93 - 0.80) / 0.20 = 0.65 and the headline never flinches. The aggregate is mathematically compatible with a disastrous subgroup. You cannot infer slice health from a mean; you have to measure it, and the measurement is section 7 of the card.

Practices that keep the table honest:

• Pre-register the slices - choose them before running the evaluation. Picking slices after seeing results is how cherry-picking gets a clean signature.
• Report n and an interval with every point. 0.68 on n=214 and 0.91 on n=4,180 are not equally solid, and intervals say so. Tiny slices get an insufficient data label instead of a point estimate.
• Intersections explode combinatorially (gender x age band x region), so pre-register a handful and mark the rest unmeasured rather than mining them for favorable stories.
• Check label quality per slice. If one dialect's gold labels came from out-of-group annotators, part of the measured gap is a label gap; disagreement rates by slice expose it before a reviewer does.

When a gap appears, the card owes three more things: a hypothesis about the mechanism (missing training data for that band, feature unavailability, label noise), the mitigation in flight (targeted data collection, a per-slice threshold, removal of a proxy feature), and the owner. A bare number with no mechanism reads as an unowned defect; the same number with a hypothesis and a plan reads as a managed limitation. Note the ordering discipline too - mitigation decisions belong to the model owner and domain experts, not to whoever ran the notebook; the card documents the decision and its rationale.

The metrics choice matters per slice as well. Accuracy is close to useless for a slice with a 1% base rate - a model predicting not fraud for everyone scores 99%. Report the metric that matches the decision (recall at the operating threshold for fraud, precision@k for ranking, F1 where classes balance), and if a slice's metric cannot be computed at the operating threshold because the slice has no positive cases in the eval window, say exactly that rather than substituting a friendlier number.

Interval arithmetic you can do inline:

import numpy as np

def bootstrap_ci(y, pred, metric, n_boot=1000, seed=0):
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, len(y), (n_boot, len(y)))
    stats = np.array([metric(y[i], pred[i]) for i in idx])
    return stats.mean(), np.percentile(stats, [2.5, 97.5])

Why reviewers weigh this section so heavily: a card reporting only the aggregate reads as either unawareness or concealment. A card that reports a weak slice with its n, its interval, and a mitigation plan reads as competence - the gap is now governed, documented, and monitored, instead of waiting to be discovered by a customer. That distinction is often the entire distance between approval and a follow-up meeting.

The reviewer asks: what happens if someone uses this to decline applicants automatically? Your answer must already be in the card, in a form that can be checked. Intended use has three parts - the primary use, the explicit prohibitions, and the operating assumptions the model depends on.

The prohibitions are where cards live or die. The weak version: this model should not be used for high-stakes decisions. It is unfalsifiable, protects no one, and signals that nobody did the analysis. The strong version names the use, the population, and the measured gap that justifies the line:

### Out of scope
- Automated decline of credit applications - the evaluation set contains no
  thin-file applicants (n=0), and recall for ages 18-21 is 0.68 [run 3f9a1c].
- Use outside en-US text - training corpus is 96% en-US; de-DE measured once,
  recall 0.72 (n=302).
- Ranking job applicants - trained on transaction behavior; no validity evidence
  exists for employment contexts.
- Input assumption: account features at most 24h old; scores on staler inputs
  are undefined.

Every line is checkable: a compliance reviewer can verify n=0, read the slice number, and accept or reject the boundary on evidence. The pattern is worth copying exactly: every prohibition cites a measured gap or a missing population - nothing is there for tone. Note what a prohibition is not: it is not a disclaimer of liability, and it does not forbid a use because it is dangerous in the abstract; it forbids a use because this model was never shown to work there. That distinction is what makes the section survive legal review instead of being rewritten into the vague version it replaced.

Round out the section with the envelope the model runs in: feature freshness requirements, validated input ranges, latency budget, the default decision threshold and who may change it, and the known failure modes carried over from your error analysis buckets - negation spans, missing vocabulary entries, whatever your taxonomy found. Close with monitoring: which metric is watched, at what cadence, over what window, and what triggers a rollback. Reviewers approve what they can watch.

The question map a reviewer works through:

The primary-use half carries equal weight and is where human oversight gets specified. Rank transactions into the review queue; an analyst makes the final call and Automatically decline transactions above a score differ by one sentence and by an entire risk class. State who sees the output, what they are assumed to do with it, what feedback loop returns their decisions to the model, and the latency and throughput envelope - a model that serves 40 scores per second inside checkout cannot be documented as batch scoring service without lying about the system it lives in.

Ship discipline follows from the map: the card version travels with the model version, and a model without a current card does not ship - not because a policy says so, but because nobody can state its limits. The meeting where you walk a non-ML audience through these sections, the slice table, and the caveats is the same skill exercised at higher tempo in Communicating Results to Stakeholders.