Identifying Valuable AI Use Cases

The demo is easy: paste a document, get a plausible paragraph, watch the room nod. The roadmap review is where AI ideas actually die. Six features were proposed last quarter, and the two worth building were the ones where someone in the room could name the number that moves — already recorded in a system nobody had to invent for the occasion.

Strong use cases read like this:

• Support reply drafter — share of tickets resolved with no human edit to the draft. Baseline today: 0%, everything is manual.
• Contract intake — minutes from email arrival to fields sitting in the ERP. Baseline: 14 minutes.
• Search reranking on the store — conversion rate on sessions that used search. Baseline: 3.1%.

Weak ones read like "AI copilot for the whole team" with no metric attached. Copilots can be fine products; the problem is that without a baseline, the launch meeting cannot separate improvement from vibes.

Two candidates on one page

The first two have logs that already answer "did it work." The summarizer's survey will be argued about for a month. Prefer candidates whose success or failure is written down by systems that exist.

The metric myopia trap

Optimizing one number invites damage to the next. Acceptance rate climbs if the drafter quietly handles only easy tickets — selection bias wearing a good dashboard. Extraction accuracy climbs if the model abstains on everything hard and abstentions are counted as safe. Pair every primary metric with one counter-metric — escalation rate, abstain rate, or a sampled quality review — before launch, and the pairing becomes the metric definition you can hand to Online Metrics for AI Features later. Which shape the feature takes in the first place is decided in Common LLM Application Archetypes.

A demo answers "can a model do this." Feasibility answers whether your feature can do it under your constraints, and the constraints are always the same three.

1. Data availability

What will the feature read? If the policies exist, are current, and can be retrieved, the project is buildable this quarter. If the knowledge lives in three people's heads and a 2019 Confluence page, the project is a content project first and an AI project second. If you need labeled examples to measure success and none exist, the eval set is part of the scope, not homework — building without it means the launch meeting argues instead of measures.

2. Latency tolerance

Ask what the user is doing while waiting:

A four-second model is fine for a nightly digest and fatal for a keystroke-driven suggestion. Set the budget first — it eliminates half the candidate models before anyone argues about quality.

3. Acceptable error rate

Price the wrong answer:

• Wrong paragraph tone → cheap; regenerate.
• Wrong invoice amount posted to an ERP → dollars plus an audit finding.
• Wrong medical, legal, or financial statement → the reason those domains carry heavier evaluation and human oversight (Responsible AI Foundations).

Worked check. Invoice intake at 3,000 invoices a day. Manual keying costs about $0.35 per invoice to verify, and a wrong field costs roughly $40 to unwind downstream. At those prices a 5% field-error rate is survivable if flagged rows go to a human; a 20% rate is not, no matter how cheap the API call is. The constraint that fails first — data, latency, or error cost — decides what you build, and it is usually cheaper than the constraint everyone expected.

Before a pipeline gets built, run the workflow by hand. People play the model's part for a week or two: users fill in the form the "AI" will eventually read, a human drafts the answers behind the scenes, and delivery looks identical to the shipped feature.

This measures the two things a demo cannot: do users actually submit inputs, and are human-drafted outputs accepted? Teams that skip it frequently automate a feature nobody uses, then discover during the postmortem that the inputs were never being supplied in the first place. The manual week costs a product manager's time; the abandoned pipeline costs a quarter.

proposal
   |
   +--> metric named + baseline known? -- no --> rewrite the idea
   |
   +--> data / latency / error cost OK? -- no --> lighter feature, or data work first
   |
   +--> risk tier
   |      low     --> automate, sample quality in production
   |      medium  --> human approves before the output leaves the building
   |      high    --> human decides; model drafts with citations only
   |
   +--> manual prototype, 1-2 weeks
          +--> usage and acceptance hold --> automate
          +--> nobody submits --> stop, cheaply

The risk tier sets how much oversight the feature owes its users, and impact is the only thing that sets it. A tone-degrading summary needs spot checks. A refund denial or an account lock needs a human approving before the customer sees anything. Credit, medical, and legal outcomes need a human decision, with the model confined to drafting and citing. What disclosure and appeal path ride along with those tiers is covered in Responsible AI Foundations, and the scoring you did in this topic is the input to it.