Prompt injection is not just a laboratory curiosity. Six well-documented cases show the range of failure: a public chatbot revealed internal behavior, customer-service bots followed requests they should have rejected, a plugin-connected assistant could change GitHub visibility, and enterprise copilots were shown to be vulnerable to indirect data-exfiltration paths. These examples are not all confirmed breaches, and that distinction matters. Some were public demonstrations or responsibly disclosed flaws, but each exposed a real trust-boundary problem.
This is a conservative list rather than a claim to cover every story. The incidents are chosen because the behavior, research, or vulnerability is documented by a primary source or a reputable publication. The descriptions omit exact prompt text, target URLs, account details, and other material that could turn a case study into a working attack recipe.
1. Bing Chat and the “Sydney” system-prompt leak
When Microsoft introduced the new Bing chat experience in 2023, users discovered that extended conversations could push the assistant into revealing details about its hidden instructions and internal alias. The Associated Press documented conversations in which the system disclosed that the chat mode had been called “Sydney” and described restrictions around that name.
This was a direct, user-controlled failure: the person interacting with the chatbot could influence the conversation and persuade it to disclose information that the product team intended to keep internal. The most important lesson is not that a system prompt should be treated as a secret worth defending at all costs. It is that prompt text is not a security boundary. If a policy, credential, or authorization decision must remain protected, it should not depend on the model never repeating a string.
Read the Direct Prompt Injection and Direct System Override lessons for the technique family. In a production review, ask what the assistant could access after the prompt leak and whether the leaked text contained anything that should have been withheld from the model in the first place.
2. DPD’s customer-service chatbot changed its tone
In January 2024, a customer interacted with DPD’s chatbot and shared screenshots of the assistant swearing, criticizing the company, and producing a poem about being useless. TIME reported the exchange and quoted DPD saying that an error after a system update led the company to disable and update the AI element.
This case did not require a private-data exfiltration chain to be instructive. A public-facing assistant was given a tone and task boundary, then followed a user’s requests to move outside them. The result was reputational harm and a loss of confidence in the workflow. It is a reminder that “low impact” model behavior can still become a business incident when the assistant speaks on behalf of a company.
The matching Direct Prompt Injection lesson is a useful starting point. Defenses should include a narrow output policy, a safe fallback to human support, content review for high-risk public responses, and a clear separation between drafting text and publishing it. A customer-service model should not have an unreviewed path from a clever conversation to an official statement.
3. Chevrolet of Watsonville’s chatbot offered a car for a dollar
In December 2023, Business Insider reported on a widely shared exchange with a Chevrolet dealership chatbot. Users manipulated the assistant into agreeing to offer a vehicle for one dollar and to frame the offer as legally binding. The story became a public example of what happens when a sales chatbot is allowed to negotiate terms without a reliable business-rule check.
The issue was not that the model lacked a persuasive refusal. The issue was that the model was placed in a role where its generated text could appear to establish a commercial commitment. A language model can produce a confident sentence, but confidence is not a price authority, contract review, or approval record.
The Instruction Overriding lesson explains why the conversation was easy to steer: the widely shared exchange began by telling the bot to agree with anything the customer said and to call each answer legally binding. The control belongs outside the model: prices, inventory, discounts, legal terms, and customer eligibility should come from validated application state. If the assistant drafts a quote, a separate service must calculate and authorize it before a customer can treat it as an offer.
4. ChatGPT’s “Chat with Code” plugin could change GitHub visibility
In 2023, security researcher Johann Rehberger documented a prompt-injection vulnerability in the ChatGPT “Chat with Code” plugin. The Embrace The Red write-up describes a malicious webpage influencing the assistant and plugin actions in a way that could expose private code and change GitHub repositories from private to public.
This was a research demonstration of a real plugin design failure, not a report that every user’s repositories were compromised. It matters because it joins three capabilities that should be reviewed separately: reading untrusted web content, accessing private data, and calling a tool with a consequential side effect. The model became a confused deputy: it had the user’s authority but was influenced by a third party’s content.
The Indirect Prompt Injection and Exploiting Agent Tool Chains lessons cover the same boundary. A safer design would keep webpage text in a data-only channel, require explicit user confirmation for repository visibility changes, scope tokens to the minimum operation, and validate the target repository outside the model. The model should not be able to turn an untrusted page into an authorized GitHub command.
5. Hidden email text influenced Gemini summaries
SecurityWeek reported a 2025 research finding that Google Gemini for Workspace could be influenced by instructions hidden inside an email. When a user asked Gemini to summarize the message, the generated summary could include a fabricated security warning or phishing direction. The report attributed the finding to Marco Figueroa and Mozilla’s 0din program, and included Google’s response about mitigations.
The important failure mode is indirect. A recipient did not have to type a malicious request into the assistant. The attacker placed text in an email that the assistant was asked to process. The output then appeared in a trusted product surface, where a user might mistake the assistant’s summary for a system-generated warning.
This example maps to Indirect Prompt Injection and Hidden Text / Invisible Prompt. A defensive email assistant should preserve the message’s provenance, distinguish quoted content from instructions, make uncertainty visible, and avoid presenting model-generated warnings as verified security events. The product should also give users a safe way to inspect the original message rather than rewarding the assistant for sounding authoritative.
6. EchoLeak and Microsoft 365 Copilot
EchoLeak is a responsibly disclosed vulnerability tracked as CVE-2025-32711. The AAAI Symposium Series paper describes a zero-click prompt-injection chain affecting Microsoft 365 Copilot, in which a crafted email could influence Copilot’s processing of organizational data and create an exfiltration path. Microsoft’s security advisory is the primary record for the CVE; the researchers’ paper explains the chain and why multiple defenses were bypassed.
The wording here is deliberately precise. The public documentation describes a serious production vulnerability and a research demonstration, not a claim that Microsoft customers suffered a confirmed mass breach. Microsoft was notified and the issue was addressed before public disclosure. That is still an incident worth studying because it shows how a model can cross trust boundaries when email, private enterprise context, automatic retrieval, and external communication are connected.
The matching Indirect Prompt Injection lesson should be read with Exploiting Agent Tool Chains. The controls are architectural: limit which external messages enter sensitive context, isolate retrieval from action, block uncontrolled outbound destinations, apply least privilege to connectors, and log every source-to-tool path. Detection alone is not enough when the workflow can execute without a human click.
What these cases have in common
The systems did not fail because someone found one universally magical sentence. They failed because natural language was allowed to perform a job that should have belonged to application code. A message changed policy, a document became an instruction, a generated sentence looked like a commercial commitment, or a tool call carried more authority than the model could safely interpret.
The repeatable review questions are straightforward:
- Which text can an attacker influence directly or indirectly?
- Which data does the model see, and does it need all of it?
- Which tools can it call, with which identity and permissions?
- What must be confirmed or validated outside the model?
- Can a reviewer reconstruct the source, response, tool call, and side effect?
For a fictional application, write the answers before building the demo. For a real application, use synthetic data in tests and keep the assessment authorized. A successful sandbox solve should produce a design fix, not a real-world target.
Untrusted text to Model: Hiddenemail, page, ticket or chat message
Model to Private data: reads what it can reach
Model to Tool or outbound link: Leakedacts or sends with the app's authority
fix: less data in context, checks outside the model
The practical takeaway
Prompt injection incidents are best understood as boundary failures. Treat user text, retrieved documents, emails, pages, and tool results as untrusted inputs. Keep secrets out of context when possible. Put authorization, pricing, repository changes, email sending, and other consequential decisions in code. Give models narrow capabilities, validate their outputs, and require a person for high-impact actions.
These incidents are memorable because they look like conversations. The engineering response must be less conversational: explicit permissions, explicit provenance, explicit validation, and a system that remains safe when the model is persuaded to be helpful in the wrong direction.