Model Registry and Versioning

A retrain finished overnight and improved AUC by 0.004. Two more candidates from last week sit in a models/ folder on the training box named xgb_v3_final_FINAL2.pkl. Nobody knows which file the serving pods load, and last month's rollback took ninety minutes of archaeology. A model registry fixes this by making promotion a tracked state transition instead of a file copy — and the first thing to design is not the tool, it is the policy.

The vocabulary, using MLflow (the de-facto standard; SageMaker Model Registry and Vertex AI Model Registry express the same states):

Registering a run's model is one call at logging time:

import mlflow
from mlflow.tracking import MlflowClient

mlflow.set_experiment('churn-classifier')
with mlflow.start_run(run_name='xgb-2025-10-14') as run:
    mlflow.log_params({'max_depth': 6, 'lr': 0.05, 'n_estimators': 400})
    mlflow.log_metrics({'auc': 0.9132, 'logloss': 0.318})
    mlflow.log_tag('git_sha', git_sha())
    mlflow.log_artifact('feature_schema.json')
    info = mlflow.sklearn.log_model(model, artifact_path='model',
                                    registered_model_name='churn-classifier')
    run_id = run.info.run_id

client = MlflowClient()
client.transition_model_version_stage(
    name='churn-classifier', version=4, stage='Staging')

Newer MLflow steers toward aliases instead of stages — a named pointer that does not occupy a lifecycle slot:

client.set_registered_model_alias('churn-classifier', 'champion', version=4)
client.set_registered_model_alias('churn-classifier', 'challenger', version=5)
# serving never loads a version number:
model = mlflow.pyfunc.load_model('models:/churn-classifier@champion')

Serving must resolve by alias, never by hardcoded version. The consequence is your rollback story: a bad release is fixed by pointing champion back to the previous version — seconds, no redeploy, no archaeology. An alias is mutable; a version is immutable. That pairing (append-only versions, mutable pointers) is exactly the design Git uses, and it is why models:/churn-classifier@champion is the only model reference that belongs in serving code.

The promotion policy is where interviews separate people who used a registry from people who designed one. Four rules, each with a reason:

1. Only automation may move a model into Production. Humans (and training jobs) may register and stage; the gate that flips champion is the CI job that re-ran the evaluation harness and compared against the incumbent. This makes promotion reproducible and auditable — the same reason only bots push to main.
2. Promotion requires beating the incumbent on a fixed eval suite, not just on a training metric: golden datasets, slice metrics (the fraud class you cannot afford to lose), and fairness/regression checks. A +0.004 average AUC that costs −0.03 on your priority segment is a loss.
3. Never two producers of truth. transition_model_version_stage(..., archive_existing_versions=True) archives whatever was in Production so exactly one version holds the stage (or one alias value per name). Drift between "the Production model" and "the model actually serving" is the failure this prevents.
4. Every transition carries a note and an actor — who, when, why, tied to a ticket or run id. client.set_model_version_tag(v, 'approval', 'PR-482') is cheap, and the audit trail is the deliverable your future self and your auditors share.

What the registry deliberately does not do: hold training code, retrain models, or validate them. It holds versions of artifacts with metadata and a lineage pointer. The version numbers themselves need a semantics — when does a new version merit a new major number, versus a patch — which is the next chapter's question.

The API team asks whether they can deploy model version 7. Without a versioning contract, the answer is a shrug — and a shrug means every consumer must read your release notes to know whether their client code still works. Models are not source code, but their version numbers carry the same question: what did this change break for whoever depends on the artifact?

Borrow semantic versioning and re-define it for models, in terms of the inference contract (feature schema: names, types, order, ranges) and behavior:

Concrete cases that decide real debates:

• Retrained nightly on the same pipeline, AUC 0.9132 → 0.9151: MINOR. The contract is identical; consumers upgrade by flipping the alias.
• Added a device_type feature so the model can score new rows: MAJOR. Old clients do not send it; loading this model behind their requests produces silent wrongness, which is exactly what a major bump exists to prevent.
• Re-exported the same weights from joblib to ONNX with verified parity (the golden-batch check from Model Serialization and Packaging): PATCH — same behavior, different bytes.
• Changed the positive class from churned_30d to churned_7d — output means something else now: MAJOR, even though every input feature is unchanged.

Two alternative schemes deserve knowing because you will meet them. Calendar versioning (2025-10-14) suits scheduled retrains: the number states provenance rather than compatibility, and it works only while retrains never break contracts. Content-addressed versions (sha256 of the artifact, or a runs:/ URI) give absolute immutability — no ambiguity about which bytes a name refers to — and are ideal as the underlying identity over which human-readable aliases float.

Whatever the top-level scheme, two invariants hold underneath it:

1. Registry versions are immutable and append-only. If the bytes at version 4 change, every historical claim ("version 4 served in Q2 with AUC 0.9132") becomes a lie. Fix forward with a new version; never overwrite an artifact — MLflow does not let you, and that constraint is a feature of the design, not a limitation.
2. The version number is not the deployment handle. Consumers pin an alias (champion) or a stage; versions record history. Code that hardcodes models:/churn-classifier/4 couples a deployable service to a run of numbers and bypasses the alias layer that made rollback cheap.

The mapping between schemes and systems, since teams routinely mix them:

git commit sha        -> code that produced the model     (via run tags)
data snapshot hash    -> the training data                (DVC / lakehouse version)
run id                -> hyperparameters, metrics, logs   (experiment tracker)
registry version      -> the artifact + stage/alias       (model registry)
semantic tag          -> what changed for consumers       (your convention)

That five-line ledger is the whole answer to "how do you version ML models?" — each layer names a different dependency, and a registry entry that links all five is what makes the next chapter's lineage questions answerable.

Eight months from now someone asks the only questions that matter: the Production model is degrading — which training run produced it, on what data snapshot, with which hyperparameters, and what were its acceptance metrics when it shipped? If those answers require searching Slack, the registry is decoration. A registry entry is only as good as the lineage it carries, and lineage is something you attach at training time, not something you reconstruct during an incident.

The minimum viable lineage per registered version:

with mlflow.start_run() as run:
    mlflow.log_param('max_depth', 6)
    mlflow.log_metric('auc', 0.9132)
    mlflow.log_tag('git_sha', subprocess.check_output(
        ['git', 'rev-parse', 'HEAD']).decode().strip())
    mlflow.log_tag('data_snapshot', 'lake://churn_features@2025-10-13')
    mlflow.log_artifact('feature_schema.json')     # the inference contract
    mlflow.sklearn.log_model(pipe, artifact_path='model',
                             registered_model_name='churn-classifier',
                             signature=mlflow.models.infer_signature(X_demo, preds),
                             input_example=X_demo.head(3))

# later, during an incident — one call from the running model's metadata:
version = client.get_model_version('churn-classifier', '4')
run = mlflow.get_run(version.run_id)
print(run.data.params, run.data.metrics, run.data.tags['git_sha'])
print(version.source)            # runs:/<run_id>/model — the artifact's exact bytes

Read that round trip as the acceptance test for your registry setup: from a version number alone you can name the run, the code revision, the data snapshot, the schema, and the bytes (runs:/<run_id>/model — immutable, referenced by URI rather than copied). signature and input_example matter more than they look: they make the registry self-describing, and MLflow serving validates incoming requests against the logged signature, turning a schema mismatch into an explicit error instead of a silent bad prediction.

The lifecycle of an entry, and the hygiene rules that keep the system trustworthy:

Three failure modes the rules counter directly. Manual artifact uploads ("I'll register the file from my laptop") sever the run-to-version link — the artifact is now unattributable, and the next person cannot reproduce it. A latest-style alias that floats to whatever was registered most recently means a retrain by one engineer silently becomes everyone's Production candidate; aliases must be explicit, deliberate pointers. And evaluating at Staging but not re-evaluating at promotion lets a model that failed a new data slice slip through, because the world moved between staging and promotion — the gate must run against current data.

For the interview, compress the whole topic into the sentence a senior engineer wants to hear: every model version is an immutable artifact linked to the run that produced it, the data snapshot it saw, the code SHA that trained it, and the metrics that justified its stage — and serving resolves models by alias so promotion and rollback are pointer moves, audited by the same CI that gates the release. With versions tracked and lineage attached, the remaining question is operational: how the chosen version gets in front of requests with a latency budget — the subject of Online and Batch Serving.