XAI (Explainable AI)
Explainable AI is making a system’s output inspectable: what it drew on, what it did, and why — rather than asking people to take the answer on trust.
Explaining a model’s internal reasoning is genuinely hard and largely unsolved. Explaining an AI system is far more tractable, and it is the part that matters at work: which documents an answer cited, which records an agent read, which tool it called with which arguments, and where a person approved. None of that requires interpreting the model’s weights.
The practical test is whether a wrong answer is diagnosable. If someone can look at a bad output and see that it retrieved a superseded document, the system is explainable enough to fix and to trust. If the only available response is to try the prompt again, it is not — however good it is on average.