Foundation Model Landscape

Two models tie on the eval your VP pasted into chat. One answers your support feature in 900 ms; the other takes four seconds and bills six times as much per conversation. The benchmark did not lie — it scored a question your product never asked.

Four axes, one score

A model card compresses to four axes, and they fail independently:

Price hides a second variable. Providers bill the two streams separately, and completion rates run higher, so a model that writes a 600-token preamble to answer a yes/no question costs several times a terse one at the same list price. Estimating Token Costs works the arithmetic on a real feature; Context Window Budgeting covers what can be trimmed before it is billed again.

Families, not single models

Frontier providers — OpenAI, Anthropic, Google, and their peers — ship tiers behind one chat API: a flagship for hard reasoning, a small fast model for routine work, and usually a mid tier between them. The request shape survives swapping the model field, so moving traffic between tiers is a routing decision in your code, not a migration:

                       +--> small tier     routine, high volume
request --> router ----|
                       +--> flagship tier  ambiguous, multi-step, high stakes

Providers rename and retire models on their own schedule, so treat the model string as configuration: pin the exact version in one place. When a model you depend on is deprecated, that should be a config change plus a rerun of your eval set, not a rewrite. Which tier dominates also depends on the shape of your feature — Common LLM Application Archetypes maps those shapes, and an extraction pipeline cares about a different axis than a chat assistant does.

Models in the Llama, Mistral, Qwen, and DeepSeek families ship as downloadable weights. Running them on infrastructure you control changes the contract in three ways.

Data path. Requests never leave your network. For medical, legal, or financial text — or a customer contract that forbids third-party processing — that decides the architecture before model quality is even discussed.

Bill. There is no per-token price. Cost becomes GPU hours: roughly fixed whether traffic spikes at noon or sits idle at 3 a.m. That inverts the API economics once volume is steady rather than spiky.

Lifecycle. Nobody retires your model name. You choose when to move to a new checkpoint, and fine-tuning on private data means uploading nothing to anyone.

What you take on

API path:    your code --------------------------------> provider
                                                serving, GPUs, updates, filters

Hosted path: your code --> queue --> GPU serving --> your weights
                   |            |
              retry policy  batch size + quantization
              rate limits   capacity sized for peak
              health checks checkpoint upgrades

• Capacity sized for peak and paid for around the clock, whether or not the feature uses it.
• Serving choices with quality consequences: continuous batching for throughput, quantization to fit more weights in memory (cheaper, sometimes measurably worse at structured output), prefix caching for long stable prompts.
• Safety filters and refusal behavior you configure rather than inherit.
• Feature parity lag: a hosted family may deliver tool calling or structured outputs a release behind the API frontier.

The rule of thumb. Host when data control is non-negotiable or daily volume is high and steady enough to keep GPUs busy. Call APIs when traffic is spiky or nobody on the team wants to own inference operations — buying capacity for a feature that might be cut next quarter is an expensive hedge. Retention terms, training-on-your-data defaults, and subprocessor lists on the API path are diligence you finish before the first prototype, which is where Responsible AI Foundations picks up.

The setup

A support assistant answers 40,000 questions a day. Every request carries 1,200 prompt tokens (policy excerpts plus the question) and reads back 250 completion tokens. Three plans at round illustrative rates shaped like a real price list:

Volume first: 40,000 × 1,200 = 48M prompt tokens a day, and 40,000 × 250 = 10M completion tokens. Substitute your provider's current prices; the shape of the answer rarely changes.

The mixed plan cuts spend roughly 80% against the flagship. It is also wrong if the small tier mangles the 15% it keeps, so it ships with evidence attached:

1. Label 200 real questions with the correct, policy-grounded answer.
2. Run both tiers over the set and record pass rate per tier.
3. Escalate when the policy text is ambiguous, a tool call is required, or the small tier's answer fails a groundedness check.
4. Fall back to the flagship on a small-tier 5xx, and to a stored refusal when both fail (Model Fallback and Degradation).

As months of traffic accumulate, questions the small tier has answered correctly stop being escalated. The boundary is data, and it moves.

Nothing here required picking a winner on a leaderboard. Quality evidence, a latency budget, a monthly ceiling, and whatever compliance constraints apply decided it — that comparison is the selection rule, and it sits next to the scoring you did in Identifying Valuable AI Use Cases.