What Explainability Means in Document Review

Start with the question that settles most of these conversations. If a tool is explainable, what would an unexplainable version of the same tool look like? A vendor who cannot answer that has not made a claim, they have used an adjective.

The most workable answer available comes from NISTIR 8312, which declines to treat explainability as one property and instead separates it into four principles. Taking them one at a time is tedious and it is the only approach that produces questions a purchaser can actually put to a supplier.

Explanation: the system produces one at all

The weakest principle, and the only one most tools clear comfortably. Something has to come out alongside the output. A highlighted passage, a list of contributing terms, a similarity score against a known example, a rule that fired.

It is worth noticing how low this bar is. A system satisfying only this principle has committed to nothing about whether the explanation is useful, whether it is true, or whether the system knew what it was doing. It has committed to emitting something. Almost every product demonstration you will sit through is a demonstration of this principle, and the demonstration is usually very good, because producing plausible explanatory artefacts is a solved problem.

Meaningful: to a specific reader, for a specific decision

An explanation is meaningful when its recipient can use it. That makes meaningfulness a property of a relationship rather than of a system, which is why it resists being demonstrated in a sales meeting and why it is the principle most often quietly dropped.

In document review the recipients differ sharply. A reviewer deciding whether to escalate a document needs to know which passage drove the classification, and needs it in about two seconds. A supervising lawyer defending the review protocol needs to know whether the classifier's behaviour was consistent across the corpus, which is a completely different artefact. A judge assessing whether a process was reasonable needs neither of those and needs an account of the method.

One interface cannot serve all three, and a tool that offers a single explanation view has implicitly chosen one of them. Asking which one is a fair question and the answer is informative.

Explanation accuracy: the story has to match the mechanism

This is the principle that does the most work and gets the least attention, partly because its name invites the wrong reading. Explanation accuracy is not about whether the system's answer was correct. It is about whether the explanation correctly describes how the system reached that answer.

The two come apart completely. A classifier can be right for reasons its explanation misstates, and it can be wrong while its explanation faithfully reports the flawed reasoning that got it there. The second case is a good system behaving well. The first is a serious problem wearing a reassuring interface.

The gap widens with a category of technique NISTIR 8312 treats separately: post-hoc explanation, where the account is generated after the fact by a second process examining the first. A post-hoc explanation is a model of a model. It is fitted to be plausible. Nothing in its construction guarantees fidelity, and the more complex the underlying system, the more room there is for a compelling explanation of something that did not happen.

The report also has a section on adversarial attacks against explainability, which deserves more attention than it gets in legal contexts specifically. If an explanation can be manipulated separately from the behaviour it describes, then in a setting where someone benefits from the explanation looking a particular way, the explanation is itself a contested claim. Litigation is precisely that setting.

Knowledge limits: where the system should decline

The fourth principle asks whether the system operates only within the conditions it was designed for, and says so when it does not.

This is the one that matters most in document review and the one almost nothing does well. Give a classifier trained on one organisation's correspondence a batch of engineering drawings, or a language it never saw, or a document type that did not exist when the training set was assembled, and it will not fall silent. It will return a label with a confidence score, because returning a label is the only behaviour it has.

Confidence scores are frequently mistaken for this principle and are not it. A score expresses how the model ranks this input relative to its training distribution. It does not express whether the input belongs to that distribution at all. Those are different questions and only the second is a knowledge limit.

The practical test is blunt. Ask what the system does with an input it was not built for, and ask to see it happen rather than to be told. A system that has no answer has no fourth principle, whatever its documentation says.

Where this leaves an evaluation

Four principles produce four separable questions, and the value of separating them is that a tool can pass some and fail others, which a single yes-or-no on explainability conceals.

• What comes out alongside each decision.
• Who that artefact was designed for, and what it is useless for.
• Whether the explanation is generated from the mechanism or fitted to the output afterwards.
• What happens at the edge of competence, demonstrated rather than described.

None of these require statistical training to ask, and the pattern of answers is more informative than any of them individually. A supplier who has thought about the fourth question has usually thought about the other three. One who has only prepared for the first will produce a very polished demonstration of it.

The companion piece takes the single most common artefact in these evaluations, the accuracy percentage, and works through what has to sit behind it before it means anything. For the other half of the subject, which is the quality of the records these systems are aimed at, the rest of this publication is the way across.