Human review is part of the architecture

A fraud platform we built for a financial institution scores transactions as they happen. Most clear. A small number are blocked outright. Between those sits a band where the model is not confident enough to decide on its own, and those cases go to a person. When the programme was first scoped, that band was one box on the architecture diagram, labelled manual review.

That box turned out to be roughly a third of the build.

The queue has a service level whether anyone sets one or not

A referral queue is a system with an arrival rate and a service rate, and the same arithmetic applies to it as to any other queue. If reviewers clear four hundred cases a day and the model refers six hundred, the backlog grows by two hundred a day until someone intervenes. Long before the queue becomes unmanageable it becomes useless: a held transaction that gets a decision four days later has already cost the customer relationship the decision was meant to protect.

This makes the referral threshold a capacity decision as much as a modelling one. We size the queue before we tune the model. Expected volume multiplied by referral rate, set against reviewer headcount and a handling time measured by sitting with people while they work rather than by asking them how long a case takes.

The reviewer's screen is a production interface

Reviewers read fast under time pressure. Show them a raw feature vector and they will invent a heuristic within a week, usually one that keys off whichever field happens to sit at the top of the screen. The decisions will still arrive, and they will look like review, and the quality will be poor in a way that no dashboard reports.

What a reviewer needs is a compact account of the case: what triggered the referral, what the comparable history looks like, what this customer did in the previous hour. Anything requiring a second system to be opened will not be opened. We treat the review console as a product surface, with design work, usability testing against the actual reviewers, and a latency budget. A console that takes six seconds to load a case has set the ceiling on throughput before a single reviewer is hired.

The escalation path is where hard cases go to die

Every review system needs a second tier and a written rule for reaching it. Without one, ambiguous cases circulate until they are resolved by whoever is least likely to be challenged about it. Define the tiers, put a clock on each and decide in advance what happens when a clock expires. Defaulting to deny is defensible. Defaulting to allow is defensible in some contexts. Defaulting to silence is the one option that fails audit.

Reviewer decisions are the most valuable data in the system

Each decision is a label, produced by a qualified person, on precisely the cases the model finds difficult. Most programmes discard it. Capture the outcome and the reason together, with a short structured field alongside free text. Free text alone cannot be analysed at volume. A dropdown alone will be filled in with whatever option is least trouble. Both, and the reasons start to tell you which failure modes the model actually has.

We got the order wrong on an early engagement of this kind. The model went first and the console was built six weeks before go-live. Reviewers were slower than the plan assumed, the queue grew, and the fix on offer was to raise the referral threshold. Raising it meant referring fewer cases, which meant the model was quietly deciding more than anyone had signed off. Nobody made that decision. It emerged from the schedule.

The human in the loop is a component. It has a throughput, a failure mode, a cost to operate and a person who owns it. Draw it in the diagram at the same time as the model, and give it the same scrutiny in design review, because on a bad day it is the part carrying the system.