Incident file · Production
OpenAI global outage, December 11, 2024
~4h22m fleet-wide outage, ChatGPT, API, and Sora all down
The system
ChatGPT, the OpenAI API, and Sora, served across OpenAI's full production fleet, running on a Kubernetes-based control plane.
What was observed
Between 3:16pm and 7:38pm PST on December 11, 2024, ChatGPT, the API, and Sora were significantly degraded or unavailable fleet-wide, OpenAI's longest outage to that point.
Stop and think
- If you build a product on a single LLM provider's API and that provider has a multi-hour fleet-wide outage with no advance warning, what is your system's actual behavior right now?
- A new observability tool is meant to increase visibility into a system. What would have to go wrong for it to instead obscure the blast radius of an incident?
- Engineers' primary tool for fixing the outage (
kubectl) itself depends on the thing that broke. What does that imply about how remediation plans should be designed?
Answer the questions above before you read on: what failed, why, and what would have stopped it.