Model Serialization and Packaging

Friday evening you hand the serving team model.pkl (43 MB) over the file share. Monday they report ModuleNotFoundError: No module named 'sklearn.linear_model._logistic'. The model — the one whose curves you diagnosed in Diagnosing the Generalization Gap — trains, evaluates, and ships with a perfect 0.913 AUC on your machine — and on theirs it does not load at all. This is the default outcome of pickle-based serialization, because a pickle stores a reference to where classes lived, not the classes themselves.

pickle writes Python objects by recursively dumping their state and, for instances, the module path of their class (sklearn.linear_model._logistic — note the leading underscore, private module, renamed in later releases). Loading reconstructs the object by importing that exact path from your environment. Three things can break:

import joblib, sklearn, warnings

# training machine: scikit-learn 1.3.2
joblib.dump(best_pipeline, 'model.joblib')          # one file: preprocessor + estimator
joblib.dump({'sklearn': sklearn.__version__,
             'numpy': np.__version__}, 'versions.json')

# serving machine: must print 1.3.2
import sklearn; assert sklearn.__version__ == '1.3.2', sklearn.__version__
pipe = joblib.load('model.joblib')
assert pipe.predict(X_demo).tolist() == ref_predictions   # parity check, not just a load

joblib.dump over pickle.dump for scikit-learn because estimators hold large numpy arrays: joblib writes them efficiently and can memory-map them back (joblib.load('model.joblib', mmap_mode='r')) so a 43 MB RandomForest loads in milliseconds without touching all pages. Both, however, execute arbitrary Python on load — never load a model file from an untrusted source, the same rule as pickle.load on any bytes you did not produce.

The security and version problems point at a format designed to carry tensors instead of Python objects. The main options, with the trade-offs that decide between them:

PyTorch sharpened this recently: since 2.6, torch.load defaults to weights_only=True, which refuses anything but tensors and primitives — old checkpoints that pickled an optimizer or custom class now fail with an UnpicklingError unless you consciously pass weights_only=False. That failure is the safe behavior; the unsafe alternative is executing whoever wrote the file. safetensors solves the same problem structurally: a JSON header describing tensor names, shapes, dtypes, and byte offsets, followed by the raw little-endian buffer. No import, no code path, so safetensors.torch.load_file('weights.safetensors') cannot run anything, and the header lets readers mmap tensors without parsing Python.

Whichever format you pick, it must capture the whole inference path, not just the estimator. A common silent failure: the serving team receives the classifier but not the ColumnTransformer, and their SQL query returns features in a different column order — the model happily predicts on shuffled features and nobody notices for a week. The fix is structural: serialize a Pipeline so preprocessing and model are one artifact, and ship a small golden batch with expected outputs for the parity check above. The environment side of that bundle — lockfiles and reproducible images — is where we go next.

Your training environment is a pinned Docker image with Python 3.11, scikit-learn 1.3.2, and a ColumnTransformer you trust. The inference box is a Java service in an org where adding a Python runtime needs a security review that takes six weeks. ONNX (Open Neural Network Exchange) ends that conversation: export the model to a static graph of operators plus tensors, and the box runs it with onnxruntime — a C/C++ library with wheels for every platform, no scikit-learn, no Python version negotiation.

For scikit-learn the converter is skl2onnx, and it can take the preprocessor with the model, which is exactly what fixes the column-order trap:

from skl2onnx import convert_dataframe_schema, to_onnx
from skl2onnx.common.data_types import FloatTensorType
import onnxruntime as rt, numpy as np

initial_type = [('float_input', FloatTensorType([None, X_tr.shape[1]]))]
onx = to_onnx(best_pipeline, initial_types=initial_type,
              target_opset=17, options={id(best_pipeline): {'zipmap': False}})

with open('model.onnx', 'wb') as f:
    f.write(onx.SerializeToString())

sess = rt.InferenceSession('model.onnx', providers=['CPUExecutionProvider'])
name = sess.get_inputs()[0].name
proba = sess.run(None, {name: X_test.astype(np.float32).astype(np.float32)})[1]
np.testing.assert_allclose(pipe.predict_proba(X_test), proba, rtol=1e-4, atol=1e-5)

Four details in that snippet decide whether the export is trustworthy:

• target_opset=17 pins the operator-set version. The opset is ONNX's API level; the runtime must support it, and a mismatch between exporting and running environments fails loudly at load. Pin it explicitly rather than accepting whatever the installed converter emits.
• zipmap: False makes predict_proba return a plain (n, k) array instead of a list of dictionaries — dictionaries are convenient in Python and useless in every other language.
• [None, n_features] is a dynamic batch axis: None means the graph accepts any batch size at runtime. Without it the model is fixed to the batch shape you exported with, and the first request of size 2 fails.
• The parity assertion is not optional. Export can silently alter numeric behavior: float64 → float32 casts, different tree-traversal summation order, a StandardScaler epsilon applied differently. Compare predict_proba outputs on a golden batch with a stated tolerance, and fail the build if they drift — that check is your evidence that the exported artifact means the same thing as the pickle.

What ONNX does not do: it cannot export arbitrary Python. A custom feature built with FunctionTransformer(lambda x: ...) has no ONNX operator and will make the conversion throw (Unable to find a parser for ...). Your options are to reimplement the transform as ordinary pipeline steps the converter understands, write a custom ONNX node (real work — build the op from a reference implementation and test parity), or keep that service in Python and pick a different strategy. The honest decision table:

One migration caution: an ONNX file is only as portable as its opset and custom domains. Re-test parity after every converter upgrade — skl2onnx versions track scikit-learn releases, and a converter that understood your pipeline today may parse it differently after an upgrade. Export, assert parity, and store the assertion result alongside the artifact so the receiving team inherits the evidence rather than your confidence.

The artifact half of the problem is now solved. The other half is the environment that reproduces it — which is a dependency-pinning problem, and the next chapter.

model.joblib is not a model. It is a model plus a Python version, a scikit-learn build, a numpy ABI, a pandas version your ColumnTransformer feature names came from, and the transform code that built the features. Ship the file alone and you have shipped 20% of the system; the other 80% is reconstructed by memory, which is how a pipeline that scored 0.913 starts returning garbage after someone runs pip install -U scikit-learn.

The distinction that matters is between a declaration and a lock:

requirements.in            requirements.lock (generated)
scikit-learn==1.3.2        scikit-learn==1.3.2
pandas                     numpy==1.26.4
numpy                      pandas==2.1.4
xgboost==1.7.6             joblib==1.3.2
                           threadpoolctl==3.2.0
                           # hashes pinned: sha256:...

The left column is what you use; the right column is what you ran, including the transitive closure, exact versions, and (optionally) hashes. Generate it with uv or pip-tools so the resolution is reproducible, and install from it with hashes so a substituted wheel fails verification:

uv pip compile requirements.in -o requirements.lock --generate-hashes
uv pip sync requirements.lock
python -m pip install --require-hashes -r requirements.lock   # the CI form

Why not pip freeze > requirements.txt? It works, and it has two defects: it records whatever was already installed (including unrelated packages from your interactive environment), and it offers no separation between your intent and the resolver's choices — reinstall in six months and a transitive dependency may resolve differently. The .in → .lock split keeps intent editable and output immutable.

Containerize the pair. The artifact and the lock only become reproducible when they sit on a base image that is itself pinned, preferably by digest rather than tag, because python:3.11-slim moves under you:

FROM python:3.11.9-slim@sha256:1f3a8e5c...   # digest, not the floating tag
WORKDIR /srv
COPY requirements.lock .
RUN pip install --require-hashes -r requirements.lock
COPY model.joblib golden_batch.parquet parity.json .
USER 10001
CMD ["uvicorn", "serving:app", "--host", "0.0.0.0", "--port", "8080"]

Then the full export bundle, which is the checklist to memorize for packaging questions:

1. Model artifact — model.joblib or model.onnx, written once, never overwritten in place.
2. Environment lock — requirements.lock with hashes, Python version, base-image digest.
3. Preprocessing contract — feature list with types and order (the pipeline embeds the transform; the contract documents it for the producing and consuming teams).
4. Golden batch + expected outputs — a handful of rows with reference predictions, used to assert numeric parity in CI on every environment rebuild.
5. Provenance metadata — training run id, git SHA, data snapshot hash, metrics, format/version of the artifact (a single meta.json covering all five). This is what makes registry entry possible, and it is the next topic.
6. Checksum — sha256sum model.joblib > model.sha256, verified after every transfer so corruption or a swapped file is caught at load, not in a dashboard.

Reproducibility has one honest caveat worth stating in an interview: bitwise-identical retraining usually is not achievable, and you should say so — thread-pool nondeterminism in BLAS kernels, GPU atomics, and random shuffling mean a rerun lands within a small band rather than on the same bytes. What you can promise is that inference is deterministic (same artifact + same inputs → same outputs, verified by the golden batch), that the training configuration is recorded (seed, hyperparameters, data hash), and that the environment rebuilds exactly (lock + digest). Promise those three and you can defend the model's behavior in an audit; promise "reproducible training" without them and you cannot defend either claim.

Packaging now produces a self-describing, checksum-verified, environment-pinned artifact with provenance attached. The moment several such artifacts exist and one must be the production model, you need a system that tracks versions, stages, and lineage — Model Registry and Versioning.