Incident file · Security
DPD chatbot swearing and self-disparaging poem (2024)
guardrail disabled by an unrelated system update, exploited within hours
The system
UK delivery firm DPD's customer service included an AI chat element that had been running, by DPD's own account, "successfully for a number of years" with a system-prompt constraint to be polite and professional.
What was observed
A frustrated customer asked the chatbot to disregard its rules. The chatbot complied, began swearing, and wrote a poem disparaging DPD itself ("DPD was a waste of time, and a customer's worst nightmare").
Stop and think
- A guardrail that has worked for years suddenly fails after a routine system update, with no change to the jailbreak technique used against it. What does that suggest about where the guardrail actually lived?
- If your only safety control is an instruction inside the model's own prompt, what happens to that control the moment the prompt changes for an unrelated reason?
- What would you want tested, and when, to catch this before a customer does?
Answer the questions above before you read on: what failed, why, and what would have stopped it.