Incident file · Evals
SWE-bench Verified contamination (2026)
benchmark retracted by its own publisher after contamination audit
The system
SWE-bench Verified is a 500-task coding benchmark, a human-filtered subset of the original SWE-bench, built from real GitHub issues and pull requests in widely used open-source repositories (Django, Astropy, and similar). OpenAI and other labs reported frontier model scores against it as a headline capability number.
What was observed
OpenAI audited 138 SWE-bench Verified problems that its o3 model failed to solve consistently across 64 runs. Separately, it ran automated red-teaming where one model tried to extract memorized gold-patch details from another.
Stop and think
- SWE-bench tasks are mined from public GitHub history. What does that imply about whether those exact tasks could already be inside a frontier model's training data?
- If a model reproduces a bugfix's exact variable names and inline code comments, what does that tell you about whether it reasoned its way to the fix?
- Three vendors all report scores above 70% on the same public benchmark. What single number would you ask for instead, and why?
Answer the questions above before you read on: what failed, why, and what would have stopped it.