October is the season for scary stories, but some of the most unsettling ones in technology have nothing to do with haunted houses or horror movies. They involve real organizations whose AI initiatives went wrong, sometimes with serious consequences for their customers, employees, and businesses.
From chatbots providing incorrect information to AI systems exposing sensitive data or reinforcing bias, these incidents offer a reminder of what can happen when organizations move too quickly or overlook important safeguards.
And while it's easy to blame the technology, the underlying issues often go deeper. Poor data quality, inadequate testing, unclear accountability, and gaps in governance can all turn a promising AI initiative into a costly mistake.
As businesses increasingly move AI beyond experimentation and into everyday operations, it's worth looking at what went wrong in some of these high-profile cases and what enterprise leaders can learn from them.
Imagine contacting an airline to ask about its bereavement fare policy, following the instructions you're given, and later discovering that the information was wrong.
That's what happened to an Air Canada customer after the airline's website chatbot provided inaccurate guidance about how to obtain a bereavement discount. The customer relied on that information when purchasing a ticket, but Air Canada later refused to honor the policy as the chatbot had described it.
The airline argued that the chatbot was effectively a separate entity responsible for its own responses. A Canadian tribunal disagreed, holding Air Canada responsible for the information provided through its website.
For organizations deploying customer-facing AI, the case illustrates a significant risk: a chatbot can deliver an answer with complete confidence even when that answer is incorrect.
Large language models are capable of generating convincing responses without necessarily having the information needed to answer accurately. If a system isn't properly grounded in reliable company information, it may invent a policy, misinterpret a pricing rule, or provide guidance that is no longer accurate.
Customers, meanwhile, aren't likely to distinguish between information written by an employee and information generated by an AI assistant. When an answer comes from a company's official website, they have every reason to associate it with the business.
Organizations using customer-facing AI should start by ensuring these systems have access to accurate, approved information. That means identifying authoritative knowledge sources, establishing processes to keep them current, and determining when a question should be escalated to a human.
Retrieval-augmented generation (RAG) can help connect AI responses to enterprise knowledge, but it isn't a substitute for testing, source validation, or ongoing monitoring.
Even with the right safeguards, mistakes can happen. Businesses need a clear process for identifying and correcting inaccurate responses before they create larger problems for customers.
Not every AI incident begins with a system malfunction. Sometimes, all it takes is an employee trying to finish a task a little faster.
In 2023, Samsung reportedly banned employee use of generative AI tools after workers entered sensitive internal information, including source code, into ChatGPT. The employees were using AI to assist with legitimate work, but sharing proprietary information with an external service raised concerns about how that data could be stored, processed, or accessed.
The incident reflects a challenge many enterprises continue to face as employees adopt AI tools faster than their organizations can establish policies for using them.
Consider how easily sensitive information can find its way into an AI application. An employee might paste internal documentation into a chatbot to get a quick summary, upload customer information to help troubleshoot an issue, or ask an external tool to review proprietary code. These activities may seem harmless in the moment, but without appropriate controls, they can introduce significant privacy and security risks.
Businesses need clear policies governing which AI tools employees can use, what information they can share, and which environments are approved for handling sensitive data.
Those policies should be supported by technical safeguards, including access controls, data loss prevention measures, and restrictions on sharing confidential information. Just as importantly, employees need secure alternatives that allow them to take advantage of AI without putting company data at risk.
AI governance also needs to extend beyond applications officially deployed by IT. Organizations should account for the third-party tools employees may already be using and provide practical guidance on how to use them responsibly.
AI systems learn patterns from the information they're trained on, which becomes a problem when that information reflects years of unequal outcomes.
Amazon encountered this issue while developing an experimental AI recruiting tool designed to evaluate job applicants and identify promising candidates. Because the historical resumes used to train the system largely reflected male applicants for technical positions, the model learned patterns that disadvantaged women.
The company ultimately abandoned the project, but the case remains an important example of how bias can make its way into automated decision-making.
The system hadn't been explicitly instructed to favor male candidates. Instead, it learned relationships from historical data that led to biased results. This distinction matters because organizations may unintentionally reproduce existing inequalities even when they're trying to make their processes more efficient or objective.
The concern extends well beyond recruiting. AI applications used in lending, insurance, employment, and other high-impact areas can all be influenced by biases or inconsistencies in the data they rely on.
Before deploying AI to support important decisions, organizations need to understand where their training data comes from, how representative it is, and whether it contains patterns that could produce unfair outcomes.
Evaluation should go beyond overall accuracy to examine how a model performs across relevant groups and whether certain populations are disproportionately affected.
For higher-impact applications, businesses should also establish documented evaluation criteria, maintain meaningful human oversight, and continue monitoring outcomes after deployment. A model can perform well against traditional accuracy metrics while still creating legal, ethical, or business risks.
On paper, taking a fast-food order seems like a relatively straightforward task to automate. A customer chooses what they want, the system records the order, and the restaurant prepares it.
Of course, anyone familiar with customer-facing technology knows that real interactions rarely follow such a predictable sequence.
McDonald's tested AI-powered drive-thru ordering with IBM across around 100 restaurants before ending the pilot in 2024. Although the company continued exploring automated ordering technology, the pilot demonstrated how difficult it can be to apply AI in environments where conversations are constantly changing.
Drive-thru orders can involve background noise, different accents, interruptions, menu substitutions, special requests, and customers changing their minds halfway through. A system that performs well in a controlled environment may have difficulty navigating several of these situations at once.
Enterprise AI faces many of the same challenges. A customer service assistant might handle common questions effectively but struggle with an unusual account issue. An automated workflow could process standard transactions without difficulty, only to encounter problems when information is missing or business rules conflict.
These exceptions are often where the limitations of an AI system become most apparent.
AI testing should reflect the conditions a system will encounter once it's deployed, rather than focusing only on scenarios where everything works as expected.
Organizations should evaluate how applications respond to incomplete information, ambiguous requests, unexpected inputs, and situations where the appropriate response is to ask for clarification.
It's equally important to establish escalation procedures so customers and employees can reach a person when the system cannot reliably complete a task.
By identifying those boundaries early, businesses can make more informed decisions about where automation makes sense and where human involvement remains necessary.
Incorrect information becomes particularly concerning when someone relies on it to make a decision with legal or financial consequences.
In 2024, testing of New York City's MyCity business chatbot revealed that the tool sometimes provided inaccurate guidance about regulations, housing rules, and worker protections. Although the chatbot was intended to help business owners navigate city requirements, some of its responses contradicted existing laws.
For a business owner seeking guidance on compliance, that kind of error could have serious implications. Following incorrect regulatory advice may lead someone to violate legal requirements, even when the information comes from a government-provided resource.
The incident demonstrates why AI reliability needs to be evaluated in the context of how a system will actually be used. An assistant generating marketing ideas presents a very different level of risk than one providing guidance on employment law, financial regulations, or contractual obligations.
Organizations need to consider those consequences when determining how much responsibility to give an AI application.
Businesses should evaluate AI use cases according to the potential impact of an incorrect response and apply safeguards that reflect the level of risk.
For applications involving legal, regulatory, or other high-stakes information, this may involve restricting the topics a system can address, grounding its answers in verified sources, providing citations, and requiring human review before certain actions are taken.
Regular testing against current policies and regulations is also essential, particularly when the underlying information changes frequently.
An AI system's ability to generate an answer doesn't necessarily mean it's equipped to provide reliable guidance on every subject.
While these five incidents span different industries and applications, they point to several recurring challenges that enterprise leaders should be thinking about as AI becomes more deeply embedded in business operations.
Problems with data quality, security, bias, testing, and accountability can all become more consequential when an AI system moves from a controlled experiment into production. Addressing these risks early can help organizations avoid repeating the mistakes seen in these examples.
The quality of an AI system's output depends heavily on the information it can access and how that information is managed.
Organizations should identify authoritative data sources, establish processes for keeping information accurate and current, and make sure applications can access only the data they're authorized to use.
For applications using RAG, this also means evaluating whether the system retrieves relevant information and whether its generated responses accurately represent those sources.
AI governance should be part of the planning process, with clear expectations around data privacy, approved tools, access permissions, acceptable use, and accountability.
Teams need to understand who owns an AI application, what decisions it's authorized to make, and how issues will be handled when something goes wrong.
Establishing those responsibilities before deployment makes it easier to manage risk as adoption expands and applications become more integrated into business processes.
An AI application that performs well during a demonstration won't necessarily deliver the same results in production, where inputs are often incomplete, data can conflict, and users don't always behave as expected.
Testing should account for those conditions, including ambiguous requests, missing information, unusual scenarios, and situations where the correct action is to stop or escalate rather than proceed.
Organizations should also evaluate how AI interacts with connected applications and existing workflows, since failures can occur anywhere in the process.
There are plenty of opportunities to automate business processes with AI, but not every decision should be made without human involvement.
Applications influencing customer outcomes, financial transactions, employment decisions, or regulatory compliance often require additional oversight.
Defining when human review is necessary helps organizations maintain accountability and gives employees an opportunity to identify issues that automated systems may miss.
Deploying an AI application is only the beginning. As data, user behavior, business rules, and connected systems evolve, the system's performance can change as well.
Organizations should monitor outputs, track errors, review escalations, and regularly evaluate whether an application continues to perform as intended.
Ongoing monitoring helps teams catch emerging issues, make improvements, and maintain confidence in the technology over time.
These stories shouldn't discourage businesses from pursuing AI. Instead, they highlight the importance of approaching adoption with a clear understanding of the risks and responsibilities involved.
Choosing a powerful model is only one part of building a successful AI solution. Organizations also need reliable data, secure integrations, effective governance, and processes that account for the complexity of real-world operations.
At Concord, we help organizations establish the data, application, and integration foundations needed to bring AI initiatives into production. Our teams support businesses across AI strategy, engineering, enterprise integration, and operationalization, helping them build solutions that are designed to perform reliably as their needs evolve.
Want to avoid your own AI horror story? Connect with us to discuss how we can help you build and deploy AI with confidence.
Enterprise AI projects can run into problems when organizations move into production without sufficient attention to data quality, integration, testing, governance, or human oversight. Real-world conditions often expose limitations that weren't apparent during initial development.
Examples include Air Canada's chatbot providing incorrect fare information, Samsung employees sharing sensitive information with a public AI tool, Amazon's experimental recruiting algorithm exhibiting gender bias, McDonald's ending its AI drive-thru pilot, and New York City's chatbot providing inaccurate regulatory guidance.
Organizations can reduce risk by using trusted data sources, establishing governance policies, protecting sensitive information, testing realistic scenarios, defining human escalation procedures, and monitoring AI systems after deployment.
Not sure on your next step? We'd love to hear about your business challenges. No pitch. No strings attached.
