Tool Selection Under Many Options

The model picks a tool the way it picks a token: by relevance against everything on offer. With three to five tools, the choice is easy. With thirty tools and overlapping descriptions, the model grabs a near-duplicate, and the task fails in a way that looks like a reasoning error but is really a menu problem.

The costs compound:

• Wrong tool, wrong result: the model calls orders_search when it needed get_order, and the answer is plausible but wrong.
• Wasted recovery: the model must notice the mistake and retry, which is the failure mode covered in agent error recovery.
• Token burn: every tool schema in the prompt costs tokens on every request, before the model decides anything.

The fix is not a better prompt. It is a smaller menu. Everything in this chapter is a technique for shrinking the decision space before the model ever sees it.

This is also a context window problem, not just an accuracy problem. Every tool schema in the prompt consumes tokens that could hold conversation history or retrieved documents. See context window budgeting for the full budget picture — tool schemas are one of the largest fixed costs in it.

The cheapest selection aid is not showing the model tools it cannot use. If the task is a refund review, the weather tools should not be in the prompt.

Prune in two stages. First, filter by domain tags — the same tags a tool registry assigns:

def prune_tools(all_tools: list[dict], task_domains: list[str]) -> list[dict]:
    return [t for t in all_tools if set(t["domains"]) & set(task_domains)]

Input: 40 registered tools, task_domains=["refunds"]. Output: the 6 tools tagged refunds. The model now chooses from 6 candidates instead of 40.

When domains are too coarse, add a keyword prefilter over descriptions:

def keyword_prefilter(tools: list[dict], instruction: str) -> list[dict]:
    terms = set(instruction.lower().split())
    return [t for t in tools if terms & set(t["description"].lower().split())]

Input: instruction "look up refund policy for order ORD-42", 6 refund-domain tools. Output: the 2 tools whose descriptions contain refund or order.

Pruning has a failure mode: filter too aggressively and the right tool disappears, so the model improvises with a worse one. Keep the prefilter permissive — recall matters more than precision at this stage.

Keyword matching misses paraphrases. The instruction "what's the status of my purchase" shares no terms with a tool described as "fetch order by id", yet they are the same intent. Embeddings close that gap.

Embed the instruction and every candidate description, then keep the top-k by cosine similarity:

import numpy as np

def shortlist(instruction: str, tools: list[dict], k: int = 5) -> list[dict]:
    instruction_vec = embed([instruction])[0]
    tool_vecs = np.stack([embed([t["description"]])[0] for t in tools])
    scores = tool_vecs @ instruction_vec
    top_k = np.argsort(-scores)[:k]
    return [tools[i] for i in top_k]

The dot product equals cosine similarity when vectors are normalized, which most embedding models output by default. See embedding models for how the vectors are produced and similarity metrics for why the angle, not the magnitude, carries the meaning.

Input: instruction "what's the status of my purchase", 40 tools. Output: the 5 closest descriptions — get_order, list_orders, track_shipment, lookup_return, order_history — ranked by semantic distance.

Shortlisting costs one embedding call per tool description, but descriptions can be embedded once and cached; only the instruction needs a fresh embedding per request. The result feeds straight into the pruning pipeline: embed, shortlist, then hand the survivors to the model.

Embeddings are not a silver bullet. They can rank a near-duplicate above the right tool when descriptions are poorly written, and they add latency from the embedding call itself. The robust design is a cascade: domain prune first (free, deterministic), keyword prefilter second (cheap), embedding shortlist third (semantic). Each stage shrinks the set before the next, most expensive stage runs.

Registries accumulate near-duplicates: search_orders, orders_search, and get_orders are three tools with one intent. Every duplicate is a chance for the model to pick the wrong one. Consolidate them under one parameterized tool:

search_tool = {
    "name": "search",
    "description": "Search a named index for records",
    "parameters": {
        "type": "object",
        "properties": {
            "index": {"type": "string", "enum": ["orders", "products", "customers"]},
            "query": {"type": "string"},
        },
        "required": ["index", "query"],
    },
}

One tool, three jobs, zero ambiguity about which to call. The index enum makes the choice explicit instead of implicit in the tool name.

Then log every selection decision — chosen tool and rejected alternatives:

def log_selection(instruction: str, chosen: str, rejected: list[str]) -> None:
    record = {
        "instruction": instruction,
        "chosen": chosen,
        "rejected": rejected,
        "timestamp": datetime.now(timezone.utc).isoformat(),
    }
    selection_log.append(record)

Rejected alternatives are the signal. If the log shows get_order losing to orders_search on 40% of lookups, the descriptions need rewriting — a systematic mis-selection pattern you can fix before it reaches users.

Failure modes to watch:

• Logging only chosen tools: you cannot see the near-misses that predict systematic mis-selection.
• Over-consolidation: one parameterized tool that does two jobs badly is worse than two clear tools.
• Over-pruning: the right tool is filtered out, and the model improvises with whatever survived.