An AI assistant recommends a supplier, summarizes a contract, or flags a transaction. The answer is clear. The path to that answer may be much harder to inspect.
That gap is the AI black box problem: we can observe what a system receives and produces without having a reliable, understandable account of how it reached a particular result. It matters because using an answer often requires more than knowing that it sounds reasonable.
A convincing explanation is a starting point for review. It is not proof that the answer is right.
What makes an AI system a black box?
In a simple rule, the connection between input and output is visible: if an invoice exceeds a set amount, route it for approval. You can read the rule, check the amount, and explain the result.
A deep learning model works differently. Training adjusts many numerical parameters so that the model can recognize patterns and produce useful outputs. Those patterns can be distributed across many calculations. Examining the code or the numerical weights does not necessarily reveal a short, human-readable reason for one answer.
There are also different barriers to understanding. A vendor may withhold information about its system. A model may be technically accessible but difficult to interpret. An organization may fail to record which documents or tool results an assistant actually received. Opening the code, studying the model, and keeping better records address different parts of the problem.
The important distinction is between knowing how a type of model works and explaining why this particular result occurred. Understanding the general technology does not automatically resolve the second question.
A practical example: the cheaper proposal
Imagine asking an assistant to compare two software proposals for the first year. This is a hypothetical example; assume the products cover the same needs and these are the only charges.
| First-year cost | Offer A | Offer B |
|---|---|---|
| Annual subscription | $12,000 | $18,000 |
| Setup fee | $8,000 | $0 |
| Total | $20,000 | $18,000 |
The assistant recommends Offer A. Its explanation says the annual subscription is $6,000 cheaper. That statement is correct, but the recommendation overlooks the implementation fee in an attachment. Under these assumptions, Offer B costs $2,000 less in the first year.
Asking “Why did you choose A?” might produce a polished paragraph about subscription savings. A more useful review asks the assistant to identify every charge, point to the source for each one, and show the arithmetic. Then a person checks the original proposals and attachment.
That review can uncover the error without revealing the model’s internal computations. But identifying the cause still takes investigation. Was the attachment missing from the upload? Did the system fail to extract the fee? Did the model receive the fee and fail to account for it? Those are different failures with different fixes.
For a lawyer reviewing a document summary or a business owner evaluating a recommendation, this distinction is practical: trace the evidence, check the result, and investigate the process separately. A readable explanation alone cannot do all three jobs.
Explanation, evidence, and accountability
These ideas are closely related, but each answers a different question.
- Explanation: What can we understand about how the system produced this result? The answer might describe a rule, highlight influential inputs, or approximate the model’s behavior around one prediction.
- Evidence: What supports using the result? That might include an original document, a checked calculation, or performance measured on examples that resemble the intended work. A source must actually support the claim attributed to it.
- Accountability: Who owns the decision? Someone needs the authority, information, and time to review the result, correct a mistake, and stop inappropriate use.
NIST’s principles for explainable AI distinguish an explanation that people can understand from one that accurately represents the system. They also emphasize recognizing the system’s limits. A tidy story can be understandable and still give a misleading picture of what happened.
An accurate explanation would not, by itself, establish that a decision is fair or suitable. A transparent rule can use inappropriate criteria. An opaque model can score well on a test and still fail on the documents or people encountered in practice. Each concern needs its own evaluation.
What explainability tools can tell you
Explainable AI, often called XAI, includes several approaches. Choosing one starts with the question you need to answer.
Interpretable models
A small decision tree, a scoring rule, or an appropriately designed statistical model can make the relationship between inputs and predictions easier to follow. That can be valuable when people must inspect and challenge individual decisions.
Simpler models do not inevitably perform worse. Cynthia Rudin’s research perspective on interpretable machine learning challenges the assumption that accuracy always requires opacity. Compare suitable models on the actual task; neither simplicity nor complexity guarantees a better result.
Local explanations and feature attribution
A local explanation focuses on a particular prediction. LIME, for example, approximates a model’s behavior near the input being examined. That can help someone inspect an individual result, but a local approximation is not a complete description of the model.
Feature attribution methods, including SHAP, assign contributions to inputs under the method’s assumptions and comparison setup. They can help investigate which features influence predictions. However, SHAP’s documentation warns against treating predictive relationships as causal evidence. A factor associated with a higher score is not necessarily something you can change to improve the real-world outcome.
Testing changes to the input
You can also examine how a system behaves when relevant details change. In the proposal example, explicitly including the implementation fee should affect a comparison of first-year costs. Changing only a supplier’s name should not determine which of two otherwise identical offers is cheaper.
Such checks can expose behavior worth investigating. A handful of successful examples does not establish reliability across future cases, and a changed answer does not fully explain the model’s internal process.
Each approach provides a different kind of visibility. Ask what the explanation covers, how it was produced, and where it may be misleading before treating it as evidence you can rely on.
Why a chatbot’s explanation needs checking
A chatbot can answer a question and then explain its answer in fluent prose. That makes it easy to read the explanation as a faithful report of the process that produced the answer.
It may not be. In experiments using chain-of-thought prompting—requests for step-by-step reasoning—Turpin and colleagues demonstrated cases in which language-model explanations omitted influences that changed the answer and supplied plausible rationalizations instead. This does not mean every explanation is false. It means fluency and detail are insufficient evidence of faithfulness.
Use an explanation to make the work easier to inspect. Request the source passages, assumptions, calculations, and unresolved questions. A citation must be opened and checked; its presence alone does not establish support. A list of reasoning steps should be evaluated on its merits rather than assumed to expose the model’s internal workings.
For document-based work, a useful request is:
For each conclusion, identify the supporting passage and its location in the source. Separate stated facts from assumptions. Show calculations, flag missing information, and identify anything that needs human review.
This prompt can help organize verification. It cannot guarantee that the model will comply accurately, so the review still has to happen.
Six questions before relying on an AI result
The following checklist applies the emphasis on evaluation, oversight, and responsibility in the NIST AI Risk Management Framework to everyday professional work.
- What exactly is the system being asked to do? Define the task and consequences of error. Drafting, recommending, and acting need different controls.
- What evidence supports the answer? Check sources and calculations. For repeated use, test representative examples, including difficult cases, and record significant errors.
- What does the explanation actually establish? Identify whether it is a rule, local approximation, tool record, or generated prose. Recognize its limits.
- Who reviews consequential results? Give a named person the information, time, and authority to check and reject outputs.
- Can someone challenge and correct a mistake? Provide a route to a reviewer who can correct the outcome and address downstream effects. Keep appropriate records with suitable access and retention limits.
- What would cause you to pause use? Define which errors or missing information require stopping. Reassess performance when the model, sources, or workflow changes.
You do not need a complete map of a model’s internal workings to verify an arithmetic calculation or check a quotation against a document. As the consequences grow and independent verification becomes harder, you need stronger evidence and tighter limits on what the system can do.
The goal is to be specific about what you understand, what you have checked, and who remains responsible. For a broader view of those responsibilities, continue with the guide to responsible AI.
