Design multi-provider failover for an LLM-dependent product

production, reliability, failover

Prompt

Your customer-facing product relies entirely on a single LLM API provider. Design a failover architecture so a multi-hour, fleet-wide outage at that provider does not take your product down with it.

Interviewer's answers to clarifying questionsopen after you have asked your own

Current state? One provider, one model family, direct API calls from the application, retries with exponential backoff on failure only.
Business impact of downtime? The feature is revenue-critical; every hour of full unavailability is measured in real customer-facing impact, not just inconvenience.
Outage shape to plan for? A provider-wide outage lasting 4+ hours with no advance warning, not a single failed request.
What's acceptable during an outage? Degraded service, cached answers, read-only mode, queuing, is acceptable; a hard failure with no response is not.
Constraint? Prompts and behavior must stay reasonably consistent whether running on the primary or the fallback provider.

Follow-upsafter your high-level design

What breaks, specifically, when you fail over to a different model family mid-conversation, and how do you handle it?
How do you test this failover path for real, without waiting for an actual multi-hour outage to happen?
The fallback provider is also showing elevated error rates during the primary's outage, because other companies are failing over to it too. What's your plan?
What's the added baseline cost of keeping a second provider warm and ready, and how do you justify it against the probability and cost of an outage?

Answer first: out loud, on paper, or with the interview prompt above. Then compare with the rubric.