← Notes

Human review belongs in the design, not at the end of it

NIST's AI risk management framework treats trustworthiness as a design consideration across the whole lifecycle — which includes sizing whoever reviews what the system holds.

When an automated control holds material, someone has to decide what happens to what was held. That step is usually treated as operations rather than as part of the system.

NIST's AI Risk Management Framework starts from a different premise. It "is intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems" — trustworthiness considerations enter at design, development, use and evaluation. They are not a finishing touch.

Applied to holding, that means who reviews, with what capacity, and in how long belongs to the same conversation where the control's threshold is set. A stricter control is not only safer; it is more expensive in review, and that share of the cost has to show up in the decision that tunes it.

For a small operation the implication is both harsh and useful. Harsh because there is no second reviewer to push the queue to. Useful because it makes a design limit explicit: the control can be as strict as review capacity will sustain. Above that, what is gained in safety is lost in material that never circulates again — and the loss is silent, because nothing breaks when a queue grows.