Using the NIST AI RMF to Question a Vendor

Frameworks aimed at organisations tend to be read by nobody in the room where a purchase is decided. This one is worth an exception, not because a buyer should implement it, but because its structure converts almost mechanically into questions that a supplier either can or cannot answer.

What it is, and what it is not

NIST published the AI Risk Management Framework on 26 January 2023, developed with the private and public sectors through a process that included a request for information, public comment drafts, and workshops. It is intended for voluntary use. Nobody is obliged to follow it and nobody certifies against it.

One thing to know before leaning on it: the agency's own page states that version 1.0 is being revised as part of the White House AI Action Plan, and that a profile for critical infrastructure was proposed in April 2026. Citing it as a fixed standard would be a mistake. Using its structure as a way to organise questions does not depend on the text staying still.

The four functions, turned into questions

The framework's core is organised into four functions: Govern, Map, Measure, and Manage. Each has a natural interrogative form.

Govern asks who is accountable. Put to a supplier: who inside your organisation owns the decision that this system is fit for legal document work, and what would cause them to withdraw that view? A supplier with no answer has no governance, only a product.

Map asks what context the system is meant for. Put to a supplier: what corpus, what document types, what languages, and what matter profile was this built and validated against? This is the question that surfaces the mismatch between a demonstration corpus and the collection actually in front of you.

Measure asks how performance and trustworthiness are assessed. Put to a supplier: what is measured, how often, against what held-out data, and who sees the result? A vendor measuring only at release is measuring the version they shipped, not the one running.

Manage asks what happens when something goes wrong. Put to a supplier: what is the detection path, the escalation path, and the rollback path? On a review, the meaningful version is whether a misclassification discovered in month three can be traced back through everything the system touched.

The seven characteristics, used as a scorecard

The framework also sets out characteristics of trustworthy AI: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.

Read that list as seven separate axes rather than as a definition of a good system, and its usefulness appears immediately. A tool can be strong on several and weak on others, and the pattern of strength is what tells you whether it fits your use.

For legal document work, three of the seven usually dominate. Privacy-enhanced, because the corpus is somebody's confidential material and the question of where it is processed is not incidental. Accountable and transparent, because a process may have to be defended later. Explainable and interpretable, which the companion article in this section unpacks using the NIST report devoted specifically to it.

Valid and reliable matters too, obviously. It is listed first in the framework and it is the one every supplier already volunteers, so it needs the least prompting and the most scepticism.

The appendix worth reading

The document carries an appendix on how AI risks differ from traditional software risks, and it is the part most likely to change how a buyer thinks.

Traditional software fails in ways that are, in principle, reproducible: the same input gives the same wrong answer, and a test catches it. The failure modes the framework is concerned with are statistical, dependent on data that shifts under the system, and often invisible until someone aggregates outcomes. That difference is why a procurement process designed for software tends to ask the wrong questions of a model, and why the acceptance test that satisfied your IT function may not tell you anything.

Using it without overclaiming

Two honest limits. This is a voluntary framework, so a supplier saying they align with it has made a statement about themselves that nobody has audited. And it is being revised, so any contractual reference to it should name a version and a date.

What it gives you is a vocabulary that is neutral, public, and not written by anyone selling you anything. Asking the four functions in order takes about ten minutes and separates suppliers who have thought about deployment from suppliers who have thought about demos. That is worth more than most of what happens in a procurement cycle.

For the data these systems get pointed at, and how it behaves when counted, the rest of this publication covers the other half.