Design an LLM gateway with routing, caching and budgets

production, gateway, cost, caching, routing

Prompt

Your company's combined LLM API bill is $75,000/month, split between a latency-sensitive customer chatbot and a large, latency-tolerant batch document-classification pipeline. Finance wants a 40% cost reduction with no quality regression. Design an LLM gateway that gets you there.

Interviewer's answers to clarifying questionsopen after you have asked your own

Traffic split? Chatbot: ~2M requests/month, p99 latency budget 2 seconds. Batch classification: ~10M documents/month, no real-time latency requirement.
Current setup? Every request goes directly to the same frontier model, no caching, no routing by difficulty.
System prompt stability? The chatbot's system prompt and tool definitions are mostly static, with a small amount of per-user dynamic content inserted at the end of each request.
Quality bar? Any cost-saving change must be measured against an accuracy/quality baseline before and after; a regression is not acceptable even if it saves money.
Governance ask? Finance wants a per-team cost ceiling, not just an overall bill, enforceable without code changes by each team.

Follow-upsafter your high-level design

How do you detect that your prompt cache's hit rate silently dropped to zero after someone changed the prompt template to include a timestamp?
How would you validate that your cheap/expensive model router isn't systematically misrouting a specific category of hard queries to the weak model?
A team blows past its monthly budget ceiling on day 20. What does your gateway do: hard stop, degrade, or alert and let them keep spending?
Six months later, a new model generation is 30% cheaper at the same quality. How does your gateway let you adopt it without a redeploy of every team's integration?

Answer first: out loud, on paper, or with the interview prompt above. Then compare with the rubric.