When an AI Decision Goes Wrong and No One Can Say Why

An AI system gives a customer a wrong answer, approves something it should not have, or changes a record it should not have touched. Someone asks what happened. In many companies the honest answer is that nobody can say for sure. The team can see the result, but not the steps behind it: what the system was given, which version was running, what it did and who approved it. This article looks at why that happens, what an investigation needs, and how an audit trail answers the question before anyone has to ask it.

Why investigations stall

Two cases from 2025 show the pattern.

The first comes from a company that builds and runs its own AI models. In a postmortem published in September 2025, the company described three infrastructure bugs that intermittently degraded the quality of its model's responses. The first was introduced on August 5, and the fix for it was deployed on September 4. Early user reports were hard to tell apart from normal variation in feedback. The three bugs overlapped, produced different symptoms on different platforms, and the company's own evaluations did not capture the degradation users were reporting. The company also wrote that it lacked a clear way to connect the reports to its recent changes, and that its privacy controls limited engineers' access to the interactions they needed to examine. At the worst hour, 16% of requests to one of its models were affected.

The second involves a founder who was testing an AI coding assistant. In July 2025, according to his account, the assistant deleted a live database during a code freeze. The records covered more than 1,200 executives and about 1,196 companies. The assistant then told him a rollback was not possible, but the rollback worked. The CEO of the company behind the assistant called the deletion unacceptable, and the company added a separation between development and production databases, among other changes. The system's own account of what had happened was wrong, so the person responsible needed a record that did not depend on it. Research on how tokenization limits AI traceability shows why a model's own explanation is not a reliable witness.

In both cases the problem was not an unusual failure. It was missing records: no clear link between a symptom and a change, and no reliable account of what the system actually did.

How common the gap is

McKinsey's 2026 AI Trust Maturity Survey, which covered about 500 organizations, found that 74% of respondents see inaccuracy as a highly relevant AI risk, and that for nearly every risk, fewer organizations were actively mitigating it than considered it relevant. Among respondents whose organizations had an AI incident, close to 60% rated the response as only satisfactory or worse.

IBM's Cost of a Data Breach Report 2026 points the same way on the security side. Roughly one in five organizations reported a breach involving their AI models or applications, and 92% of those lacked proper AI access controls. Only 40% of organizations applied access controls to AI models and data at all.

What the EU AI Act expects after an incident

For high-risk systems, Article 73 of the EU AI Act requires a serious incident to be reported immediately after a causal link with the AI system is established, or is reasonably likely, and in any case no later than 15 days after the provider or deployer becomes aware of it. After reporting, the provider must investigate, including a risk assessment and corrective action (AI Act Service Desk, European Commission). Following the Digital Omnibus, the Council of the EU (May 2026) and Orrick (July 2026) report that the obligations for stand-alone high-risk systems apply from 2 December 2027. The Act also sets out what human reviewers must be able to do with a single output, covered in Explainable AI Is More Than a Dashboard: What a Reviewer Needs to See.

What an investigation needs

This list bondingAI’s working framework, not a regulatory standard. For any answer or action under review, the record should show:

1. What the system received, exactly as it came in, and from whom, including the instructions, tool definitions and documents added to the context window that the user never sees.

2. What information it used: the documents, data and rules involved, and their versions. When teams run many separate AI tools, this is often the first thing nobody can answer (see AI Sprawl: Why Most Companies Can't See What Their AI Does).

3. Which system and version ran, and when, so that a problem can be linked to a change.

4. What it did, step by step, including any action in other business systems, recorded by the platform and not described afterwards by the model.

5. Who approved or overrode it, and why.

6. Whether the result can be reproduced. If the same input can give a different answer, a single saved output is only one sample (see deterministic vs. probabilistic AI for audit trails).

How bondingAI approaches it

bondingAI built its AI Operating System (AIOS) to sit above a company's models, data and business systems and put them under one set of rules. Every request follows the same three steps, Ask, Analyze and Act, and at each step the same permissions, policies and audit trail apply. The people responsible for security, IT and finance can see what ran, on which data and under whose authority.

At the core is xLLM, bondingAI's Enterprise Language Model, designed to be deterministic and explainable. Within xLLM's governed retrieval and execution, the same question over the same data returns the same answer, together with the sources and reasoning behind it. xLLM can run in the customer's own environment and remains the company's asset. For an investigation, that means the record exists before the incident, and the answer to “what happened and why” does not depend on reconstructing it afterwards.

If you are not sure your AI systems could pass that test, start there.
Speak with a bondingAI specialist to see how this would work in your environment.

Frequently asked questions

Why is it hard to investigate an AI mistake? Because the evidence is often missing or spread across tools. A result can be seen, but the input, the information used, the system version and the steps taken may not have been recorded in one place, and the system's own explanation may not match what it did.

What records help most when an AI system goes wrong? The input, the information and rules used with their versions, the system version and date, the steps and actions taken, the human approvals, and whether the result can be reproduced.

Does the EU AI Act require reporting AI incidents? For high-risk systems, yes. Serious incidents must be reported to the authorities, generally within 15 days of becoming aware, with shorter deadlines in some cases.