Safely roll out a model swap for a production agent

evals, rollout, reliability

Prompt

Your team wants to swap the model behind a production customer-service agent, either to a newer version of the same frontier model or to a cheaper model, to cut cost. Design how you decide whether it's safe to ship, and how you roll it out.

Interviewer's answers to clarifying questionsopen after you have asked your own

Current state? The agent has a 90% task-success rate on your offline eval set; real production traffic is far more varied than that set.
Candidate change? Either a frontier model version bump, or a smaller/open-weight model that's roughly 95% cheaper but scored 8 percentage points lower on your tool-calling benchmark in initial testing.
Risk surface? The agent handles policy-sensitive actions (refunds, account changes); a regression here is customer-visible immediately.
Known reliability issue? Even strong models can go from roughly 60% success on a single try to well under 25% success across several repeated-but-varied real tasks of the same type.
Rollout tooling available? Feature-flaggable traffic split, with the ability to route a percentage of sessions to the candidate model.
Timeline pressure? Finance wants the cost change live this quarter; engineering wants at least one full rollback-tested cycle before 100% rollout.

Follow-upsafter your high-level design

A single offline benchmark score moved in the right direction. What additional evidence would you require before approving this for even 5% of production traffic?
Your rollback trigger fires three days into a 10% rollout. Walk through exactly what happens next, end to end.
How would you measure "does it work consistently across many similar real tickets," not just "does it work once," without replaying real customers' tickets multiple times?
The candidate model is 8% worse on your benchmark but 95% cheaper. How do you frame that trade-off for leadership in a way that isn't just "benchmark score vs. price"?

Answer first: out loud, on paper, or with the interview prompt above. Then compare with the rubric.