Explainable AI in Legal Work

Almost every tool sold into legal work now describes itself as explainable. Very few of those descriptions say what would count as failing the test, which is the only thing that makes a claim worth anything.

The word is doing four jobs at once

The most useful decomposition available is the one the National Institute of Standards and Technology published in NISTIR 8312, which separates the idea into four principles rather than treating it as a single property. A system should produce an explanation at all. That explanation should be meaningful to the person receiving it. It should be accurate as an account of what the system actually did, which the report calls explanation accuracy and which is emphatically not the same as the system being right. And the system should operate within its knowledge limits, declining to answer where it has no business answering.

Splitting them apart immediately clarifies most vendor conversations. A tool that highlights the words which most influenced a classification is addressing the first principle and possibly the third. It is silent on the second, because whether a highlight means anything depends entirely on who is reading it and what they are deciding. It is completely silent on the fourth.

The fourth is the one that matters most in practice and gets discussed least. A classifier asked to sort documents it has never seen anything like will still return a confident label, because returning a label is what it does. Knowing where a system stops being competent is a property the system has to be built to expose. It does not emerge from a heat map.

Two families of technique, with different failure modes

The same report divides the available methods into self-interpretable models and post-hoc explanations, and the distinction survives contact with practice better than most taxonomies do.

A self-interpretable model is one whose structure can be read directly. A short decision tree or a sparse linear model is its own explanation, and there is no gap between the account and the mechanism because they are the same object. The cost is capacity: the models simple enough to read are frequently not good enough at the task.

Post-hoc explanation covers everything applied to a model after the fact to make its behaviour legible, and the report separates these further into local explanations, which account for one prediction, and global ones, which characterise the model overall. This is where the gap opens. A post-hoc explanation is a second model of the first model, and a good story about a decision is not proof that the story is what drove the decision.

NISTIR 8312 also devotes a section to adversarial attacks on explainability, which is worth sitting with. If an explanation can be manipulated independently of the behaviour it purports to describe, then in an adversarial setting the explanation is a claim like any other and needs its own scrutiny. Legal work is an adversarial setting.

Why any of this bites in a legal context

In most commercial deployments a bad explanation costs a bad decision. In litigation support it can cost the ability to defend a process. A party that used a model to decide which of two million documents to review may eventually have to describe that process to an opponent who would prefer it to have been unreasonable, and to a court that will want to know what was done rather than what the marketing said.

The same pressure shows up wherever model output approaches testimony. The Federal Rules of Evidence set out the conditions under which expert opinion is admissible, and the questions they raise about reliable principles and methods reliably applied are not questions a system answers by producing a confidence score. The gap between a model that performs well and a method that can be explained, tested, and defended is exactly the gap this section is about.

What is here

Two articles so far. The first works through what the four principles actually demand of a document review tool, one at a time, with the questions each one generates. The second takes the specific case of an accuracy figure and shows why a percentage without its denominator, its test set, and its error costs is not yet a fact. Both are written for someone evaluating a purchase rather than building a model.

The neighbouring sections take the other half of the problem, which is the data these systems get pointed at. If that is the more pressing question, the full index of research notes lays out how the three fit together.