Incident file · Knowledge
NYC MyCity chatbot (2024)
systematically wrong legal advice, kept online after being shown the errors
The system
NYC's Office of Technology and Innovation launched a Microsoft Azure-based chatbot, the MyCity Business Assistant, in October 2023 to help business owners navigate city regulations: licensing, labor law, housing rules, and similar. It was grounded in city informational web pages.
What was observed
The Markup's March 2024 investigation tested the chatbot against known city regulations and found it gave confidently wrong answers on multiple high-stakes legal topics.
Stop and think
- A government chatbot answering regulatory questions gets a question wrong once. Is that a tolerable error rate, and what would change your answer?
- The city added a disclaimer after being shown the errors, instead of fixing or pulling the bot. What does a disclaimer actually change about the risk to the end user?
- If you had to decide, before launch, which question categories are too high-stakes for open generation, how would you draw that line?
Answer the questions above before you read on: what failed, why, and what would have stopped it.