Portfolio and Reproducibility

A reviewer opens forty repositories in an afternoon. Yours gets the first screen — the README above the fold — and roughly ten minutes of attention. Everything in that screen exists to answer one question: can I trust this person's numbers, and can I run them myself? Structure the page for that question and nothing else.

The quickstart is the load-bearing section. Lay the repository out so the commands map to the pipeline stages:

amz-churn/
├── configs/v1.yaml          # full experiment config, committed
├── data/                    # empty; fetch_data.py fills it and verifies hashes
├── src/{ingest,train,evaluate,serve}.py
├── reports/expected_metrics.json   # what a rerun should print
├── requirements.lock              # pinned, not floating
└── README.md

make reproduce
# equivalent to:
python scripts/fetch_data.py --verify-checksum
python -m src.train --config configs/v1.yaml --seed 42
python -m src.evaluate --expect reports/expected_metrics.json --tolerance 0.005

The first command a reviewer runs must not require a decision — no version choices, no manual downloads, no editing paths in a notebook. Every decision you push to the reader is a reader who leaves. Data instructions deserve the same rigor: a fetch script that validates a checksum, a note on licensing and size, and a fallback sample so someone on a slow connection can still run the smoke config.

Test the screen the way a reviewer experiences it: open the repository in a fresh session — or hand it to a friend who owes you nothing — start a timer, and note where the ten minutes go. Every pause spent hunting for the run command, guessing at data provenance, or waiting on an unpinned install is a design defect, not a reader problem. Two cheap fixes cover most of them: a make help target listing the four commands that matter, and a committed reports/ directory holding your outputs (result table, slice report, one training curve) so a reader can inspect the evidence before running anything. Where the data cannot be redistributed, say so plainly and offer the alternative — a public mirror, a synthetic generator committed as a script, or the exact query against an open dataset — because "available on request" converts an interested reviewer into someone who closes the tab.

Two paragraphs then do more credibility work than any badge wall. First, the result: model versus baselines, per-slice, on a time-based split. Second, the trade-offs: what you sacrificed and what it bought you. A reviewer who reaches "I chose precision over recall because the review queue holds 4,000 items a day, so recall 0.80 at precision 0.60 beats recall 0.95 at precision 0.20" already understands how you think — that is the whole screening purpose of the page, and it is precisely the reasoning you will be asked to defend live in ML Interview Question Patterns.

You reported 0.871. The reviewer reruns and gets 0.864. Is that a contradiction? Without a declared tolerance, you have no idea — and neither do they. Reproducibility is not bitwise equality; it is a published expectation with a stated variance, plus the controls that keep the variance small.

Work through the sources of surprise before someone else does:

Seed everything in one place, and let the framework fail loudly rather than silently fall back to fast nondeterministic kernels:

import os, random, numpy as np, torch

def seed_everything(seed: int = 42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    torch.use_deterministic_algorithms(True)  # raises instead of quietly varying

Pin the environment with a generated lockfile rather than a hand-edited requirements file, and record the command that produced it so pins change deliberately instead of drifting:

pip-compile pyproject.toml -o requirements.lock   # or: uv lock && uv export --frozen > requirements.lock
pip install -r requirements.lock                  # installs the exact resolved graph

Configuration is part of the contract as much as code: commit configs/v1.yaml holding every hyperparameter, the seed, and the data hash it was trained against, so a rerun that differs can be diffed field by field — code, config, or data, the same three-way localization an interviewer will ask you to perform on a production regression.

Then make the rerun an automated check, not a favor you perform manually. A CI job on every push proves the claim continuously:

name: reproduce
on: [push]
jobs:
  rerun:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.lock
      - run: python scripts/fetch_data.py --verify-checksum
      - run: python -m src.train --config configs/smoke.yaml --seed 42
      - run: python -m src.evaluate --expect reports/expected_metrics.json --tolerance 0.005

Two details decide whether that job is useful or theater. First, the smoke config: two epochs on a 5% data slice that finishes in minutes, which proves the pipeline executes end to end — while the full run stays a documented local command. State the distinction in the README, or a reviewer will judge your project by a run it never claimed to reproduce. Second, the failure message is a user interface: print expected and actual metrics side by side with the tolerance, so whoever opens the log knows immediately whether the pipeline broke or the tolerance is too tight.

Finally, report honestly. A single number is a claim; a mean over three seeds with a declared tolerance is a result. Write 0.871 ± 0.004 across 3 seeds in the README, commit the expected-metrics file the CI compares against, and a reviewer whose rerun prints 0.868 concludes the pipeline is sound instead of concluding you got lucky. That sentence — mean, spread, seed count, tolerance — is the cheapest credibility you will ever buy, and it is the same contract the capstone's rerun test enforces at ship time in End-to-End Capstone Project.

Anyone can list tools. What separates a portfolio that survives technical review from one that does not is evidence of comparison: against baselines, across slices, and against the alternatives you rejected. Rigor is a table of comparisons, not a list of tools.

Start with the baseline table — same split, same code path, same evaluation script:

Each row answers a question a reviewer is silently asking. Skipping the logistic regression row is the most common gap: without it, nobody can tell whether your 0.63 came from gradient boosting or from the features, and your effort is unattributable. Include the domain heuristic even when it embarrasses you — beating a rule your grandmother could apply is the most legible kind of evidence.

Then show where the model fails, because error analysis is the interview answer your repo proves you can do. Two moves suffice:

# slice-wise metrics: same metric, segmented — not a new metric per segment
slices = df.groupby(["device_tier", "tenure_band"]).apply(
    lambda g: pd.Series({"pr_auc": pr_auc(g.y, g.p), "n": len(g)})
)
# then read the 100 worst errors by hand and cluster them by cause

Publish the resulting table — it is shorter than it looks:

A reviewer who sees 0.71 versus 0.44 next to a paragraph noting timeout-related label noise on low-memory devices learns more about your engineering judgment than 500 more notebook cells would. Hand-clustering a hundred errors usually reveals the cause is label noise or a missing feature rather than model capacity — a finding worth committing to reports/error_analysis.md so it survives the interview.

Close with the trade-off paragraph — three or four sentences, explicit about what you gave up:

I optimized precision@4,000 over recall because the review team cannot staff more queue, accepting ~600 missed fraud cases a day as the cost of keeping analyst idle time near zero. Inference runs at 38 ms p99 on CPU, which rules out the larger ensemble that gained +0.02 PR-AUC; the latency budget mattered more than the leaderboard.

That paragraph is the payload. It demonstrates that you know metrics are decision rules, that constraints drive architecture, and that every choice has a price — the exact reasoning you rehearsed in ML Interview Question Patterns. What to delete while you edit: badge walls, twenty-framework stack lists, unedited notebooks with dead kernel outputs, and screenshots of loss curves with no comparison drawn. A page of tooling logos next to a single unexplained number reads as compensation. A short, specific trade-off section reads as depth — and depth is what the reviewer came to find.

One last discipline: replace each deletion with a sentence of reasoning. The reviewer's question never changes — compared to what, and can I run it? — so every element on the page either answers that question or competes with the ones that do. Read your repository once with that sentence as the only lens, and cut whatever fails to answer it.