Portfolio Case Studies

"Built a RAG system" proves nothing. A reader — hiring manager, interviewer, customer — cannot evaluate a claim they cannot measure, so every claim in a case study needs a before-and-after number.

The strong column shares three properties: a baseline, a number, and an outcome a reader can check. "Improved answer quality" fails all three — no baseline, no number, no verifiable outcome. When you do not have a metric, instrument before you build, not after: the cheapest eval is the one you can rerun. For metric design, see Online Metrics for AI Features and Building Eval Datasets.

A note on honesty: a metric you cannot measure yet is a reason to instrument, not a reason to skip the case study. "We did not baseline resolution rate, so the 71% is unverifiable" is a postmortem line that costs you credibility. The fix is cheap — two weeks of logging before you ship — and the discipline of measuring before building is itself a story worth telling.

A case study is not a project description — it is an argument that you made good decisions under constraints. Four sections carry that argument.

Situation. What problem existed, what constraints bounded it, and why it mattered. "Support answered 40% of tickets with a knowledge base of 800 articles" is a situation; "we built a chatbot" is not.

Approach. What you built, with a diagram of the pipeline. This is the only section where architecture matters, and it matters because the reader needs to see the shape of the system before they can judge the decisions.

Outcome. The metrics from the previous chapter, tied to the situation. Resolution rate 54% → 71% closes the loop the situation opened.

Postmortem. What went wrong during the build and what you would change. This is the section candidates skip and readers trust most — a case study with no postmortem reads as marketing, not engineering. For the interview version of this structure, see AI System Design Interviews.

The diagram does more work than any paragraph of architecture prose. A reader who can see the pipeline's shape — where documents enter, where the model is called, where a human sits in the loop — can judge your decisions in seconds. Keep it to one level of abstraction: boxes for stages, arrows for data, and nothing that requires a legend to decode.

Describing only the option you chose hides the judgment that made it a good choice. A rejected alternative, with the reason it lost, proves you evaluated the space.

Running example: the case study's vector store. Chose pgvector because the team already ran Postgres and the dataset was under 1M chunks — one less operational system. Rejected Pinecone because at a projected 10M chunks the managed pricing was 4x the Postgres cost, and the team had no managed-service budget. Rejected a hosted reranker service because the latency budget was tight; a local cross-encoder on the top 50 candidates fit the budget and was good enough on the eval set.

Each rejection names a constraint — cost, latency, operations — and a number. "We considered X but it was slower" is a preference; "X added 180ms against a 500ms budget" is engineering. For the full comparison, see Choosing a Vector Store and Reranking with Cross-Encoders.

How to present the rejections: one sentence per alternative, in the form "we considered X, it lost on Y." The constraint is always a number or a hard fact — a budget, a latency ceiling, a headcount. If you cannot name the constraint that killed an option, you did not reject it, you merely did not choose it, and the reader can tell the difference.

A claim a reader cannot verify is a claim they will discount. Publishing the eval set — or a representative sample — and the scoring script turns your case study from a story into a result.

The minimum viable publication: the golden questions (or 20 of them), the scoring script, and the scores your system achieved. A reader can then rerun the scoring and confirm your faithfulness number is real. Pair it with the pipeline diagram from the approach section, and the case study becomes an artifact others can learn from and build on — which is the actual goal. The postmortem closes the loop: what broke during the build, what you would do differently, and what you would never skip again. For eval design, see LLM-as-Judge Techniques; for what happens when evals are skipped, see Production Incident Drills.

The postmortem is also where you convert the build into reusable knowledge. "Our first chunking split tables in half and retrieval recall dropped 18 points; we fixed it with structural chunking and added a regression test" teaches the reader something they can apply to their own system. A case study that only reports outcomes is a press release; one that reports failures and fixes is engineering literature.