Incident file · Knowledge
Stanford study of legal AI research tools (2024)
17-34% hallucination rate in commercial tools marketed as "hallucination-free"
The system
LexisNexis's Lexis+ AI and Thomson Reuters's Westlaw AI-Assisted Research / Ask Practical Law AI are commercial legal research products, both marketed as RAG-based and both marketed as reducing or eliminating hallucination compared to a plain chatbot. Researchers at Stanford RegLab/HAI and Yale built a preregistered benchmark of 200+ legal queries to test that claim.
What was observed
All three RAG-based tools produced incorrect or unsupported answers on a meaningful share of queries, despite the marketing claims.
Stop and think
- A legal RAG tool cites a real, retrievable case for its answer. Is that sufficient evidence the answer is correct? What else would you need to check?
- "Hallucination-free" is a specific, falsifiable claim. What would a vendor need to publish for an outside party to actually verify it?
- If RAG tools still hallucinate 17-34% of the time, what does that imply about where RAG sits on the spectrum between "solves hallucination" and "does nothing"?
Answer the questions above before you read on: what failed, why, and what would have stopped it.