Who Is in the Loop?
A steering committee asks what happens to the forty people who review this work today. Someone says there will still be a human in the loop. The room relaxes. Nobody asks what those people are there for.
Definition
"Human in the loop" says where a person sits relative to an AI system. It doesn't say what outcome the person protects, why they are there, or whether they are effective.
One reviewer catches errors before they reach customers. Another signs off so the plan can say a person looked. The label calls both of them the human in the loop, so it can't tell you which one to keep.
A plan has to settle three things, in this order: the outcome the work must achieve, including the harms it must avoid; what the person is there for; and which job they are doing.
The recognition pattern
- Human oversight is one line in the plan, with nothing said about what the person does.
- Nobody can say what a given review is there to catch, or what would make it unnecessary.
- Reviewers' corrections go nowhere, or into a vendor's product.
Start with the outcome
A widely used health algorithm predicted cost as a substitute for need. Less is spent on Black patients with the same need, so at the same score they were sicker than White patients. The score prompted doctors to consider patients for extra care. By the authors' simulations, the doctors' choices narrowed the gap only slightly, far less than a change in the algorithm's target would have (Obermeyer and colleagues, 2019).
A reviewer checks work against a target. If the target is wrong, careful review can pass the flaw along. So start with what the work must achieve, and for whom. Sometimes the right call is not to run it this way at all.
What the person is there for
Catching an error and showing that a review happened are different reasons to be there. Crootof, Kaminski and Price list nine roles. We picked six purposes that plans can easily confuse:
- Correcting: catching what the AI got wrong.
- Fitting: adapting a general output to the case at hand.
- Being accountable: taking responsibility for the result. That's fair only with the power, information, judgment and time to decide.
- Justifying: giving reasons the people affected can accept or contest.
- Standing in: showing that oversight happened, whether or not it did. The authors call this a stand-in and suspect it's common.
- Keeping a job: being there so that someone stays employed. The authors call this a warm body, and the aim can be legitimate if it's stated openly, as they urge.
One person can serve several purposes. For each, the person needs the authority it requires, and the purpose needs its own evidence. For justifying, that evidence is whether the people affected got reasons they could contest.
Acceptance is not protection
People rate algorithmic HR decisions as somewhat less fair than identical human ones (Newman, Fast and Harmon, 2020). Among staff of one university who read about a process that was never adopted, the gap was small. In a smaller online study, HR's power to adjust an algorithm's bonus decision by up to 10% didn't close the gap. An HR team free to consult an algorithm was seen about as fair as a team alone. Acceptance tracked who appeared to decide. These are reactions to descriptions, not evidence of better decisions.
A person who signs off on nearly everything has even less say than one who can adjust a decision by up to 10%. So we infer that such sign-off buys little acceptance.
Don't count "human reviewed" as evidence that anyone was protected. Protection can depend on the right to challenge a decision, and few decisions get challenged. From 2014 to 2016, 1% of Medicare Advantage denials went to the first level of appeal. Insurers fully or partly overturned 75% of those appeals (HHS Office of Inspector General, 2018). An overturn can reflect new information rather than a wrong denial, and most overturned denials concerned provider payments. Reviewers err too. In a 2019 sample, 18% of payment denials refused requests that met Medicare's and the insurer's rules, most through manual-review mistakes or system errors (the same office, 2022).
Re-review can reach people who never appealed: Michigan's unemployment agency re-reviewed 62,784 unappealed fraud penalties. It reversed 85% of the findings a computer program had resolved, and 44% of those referred to an investigator (Michigan Talent Investment Agency, 2017). The groups weren't randomized, and the agency reports on itself. Most reversals found an overpayment without intent to defraud. None of these sources isolates generative AI.
Ask what the person changed, and what happens to the people who never complain.
The four jobs
One line of "human oversight" in a plan can hide at least four jobs. A job here is a kind of work, not a position: a position bundles several, and a job that persists can need fewer people.
- Answering: the person a court, regulator or board asks about the result. An English court says lawyers must check AI research, and that those with leadership responsibilities must take practical and effective measures (Ayinde, 2025).
- Designing: the person who decides what runs alone, what waits for a person and how failure escalates.
- Relating: the person the customer or other party deals with.
- Checking: the person who reviews each output.
Foundation Capital's Joanne Chen and Leo Lu name four roles like these in agent-heavy firms, from conversations with 25 companies. Their map is an investor's thesis, and partial: it leaves out verifying outcomes and maintaining encoded criteria and evaluations.
This mapping of what better AI does to each job is ours:
| Job | What better AI does to it | Where it goes wrong |
|---|---|---|
| Answering | Leaves responsibility for the result in place. Fewer people may carry it. | Answers without the power to decide, and absorbs blame for the system: Elish's "moral crumple zone," discussed by Crootof et al. |
| Designing | More work runs alone, so placing limits and escalations matters more | Nobody does it, so the arrangement stays frozen |
| Relating | Shrinks where customers want speed, and stays where they want a person | Assumed unaffected by better AI, without asking the customer |
| Checking | Its scope can narrow, once encoding is shown to work | Keeps reassuring planners when nobody knows whether the check would still catch an error |
Why checking can shrink
Every fix a checker makes shows what the AI got wrong. Where a fix is right and applies beyond the one case, that knowledge can be encoded, in the system or in automated checks around it. Choudary argues that validation is self-liquidating: each correction teaches the system to need less correction.
But the check exists because the AI can't yet be trusted alone. So its scope can narrow only once the encoding is shown to work: tested against cases that people have reviewed, it misses no more than a limit stated in advance. Applied Enterprise AI adds a condition of its own, for narrowing and closing alike: failure can be verified and its effects undone. It recommends closing a check only when both hold, and calls that earning a closure. If a closure is earned, the uncomfortable part is that the checkers' own corrections earned it.
The evidence that checks are narrowing this way is thin. By its founder's account, Garfield Law, a small-debt law firm in England and Wales, samples its rule-driven documents but still checks every document a language model produces. It is building an AI for first-level review and would consult its regulator before going further. The founder stays accountable but reports no volumes or error rates. That's more plan than result. We also found no measured instance of reviewer positions declining.
What a closed check still needs
When a check closes, the organization still needs people who can tell when the AI is wrong. They judge escalations, set the limit the encoding must meet and notice drift.
That judgment comes from doing the work. People who only check, and stop doing the work, can lose it (Bainbridge, 1983). Cutting the check removes even the practice that checking gives. And the next generation doesn't get the work that would make them experts, because agents now do it: Chen and Lu's one-generation problem.
So before closing a check, ask who will keep that judgment, and how. Later parts of this series take up both risks, and how a closure is earned.
Two questions every closure raises
- Who owns what the review teaches? Ownership here means control: if you changed vendors, could you still inspect, change and reuse what the review taught? Yes, if it's in your own criteria and evaluations. Maybe not, if it went into a vendor's model, and one correction can go to both places.
- What do the people do next? Verifying outcomes and maintaining encoded criteria and evaluations are real work, but we doubt they would absorb most of the people who did the checking. We have no tested answer.
Boundaries and distinctions
- Not "review is finished." A check stays until it meets the conditions for closing.
- Not a claim that review doesn't work. Two later parts of this series take up when it does.
Example
Take a vendor-contract team, constructed for this page. AI produces the redlines, and twelve people review them. Asked what the review is for, the team says "catch errors." What it most needs is to protect negotiating positions nobody has written down.
Sort the work into the four jobs. Most of the time goes to checking routine clauses against those unwritten positions. One lawyer answers for the outcome. Two people handle the hard negotiations. But the work the four jobs leave out has nobody doing it: no one maintains the playbook that decides what the AI may change alone.
So the plan changes: name a playbook owner, record corrections in the playbook, and decide what the checkers do next. Only then, run a trial. For a set period, the reviewers keep reading every routine clause and record which wrong clauses a sample would have missed. The team cuts back to sampling only if those misses stay under a limit set in advance.
Applied test
- Pick one workflow. State the outcome it must achieve and the harms that are unacceptable.
- List every point where a person is involved, including answering for the work and dealing with the customer.
- For each point, say what the person is there for, including standing in or keeping someone employed where that applies.
- Name which of the four jobs they are doing: answering, designing, relating or checking. Note any work that fits none.
- For each check, ask what it catches and whether that knowledge could be encoded.
- Decide what to do at each point. Record each decision, and who is responsible for it, on a one-page map:
- Keep when the purpose is one you want and evidence shows it's served.
- Change when the purpose is right but the method or the person's authority isn't.
- Close only when three things hold: tested against cases that people have reviewed, the encoding misses no more than a limit stated in advance; failure can be verified and its effects undone; and no law requires the check.
- Staff when needed work has nobody doing it: name someone.
- Leave undecided when evidence is missing, and say what evidence would settle the decision.
- For each closure, work through who owns what the review teaches and what the people do next.
Stopping the workflow is allowed too. Take any decision you can't make to whoever can.
Sources and lineage
Who Is in the Loop? is part of Applied Enterprise AI, a body of work developed by Curtis Hibbs and Joshua Barnes. It draws on:
- Obermeyer, Powers, Vogeli and Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations," Science 366, 2019.
- Crootof, Kaminski and Price, "Humans in the Loop," Vanderbilt Law Review 76, 2023.
- Newman, Fast and Harmon, "When eliminating bias isn't fair: Algorithmic reductionism and procedural justice in human resource decisions," Organizational Behavior and Human Decision Processes 160, 2020.
- HHS Office of Inspector General, OEI-09-16-00410, 2018, and OEI-09-18-00260, 2022.
- Michigan Talent Investment Agency, news release, August 11, 2017.
- Joanne Chen and Leo Lu, "The great reorg: A human's guide," Foundation Capital, March 2026.
- Sangeet Paul Choudary, "The 'Human-in-the-Loop' Fallacy," Harvard Business Review, November 2026.
- Ayinde v Haringey and Al-Haroun v Qatar National Bank, [2025] EWHC 1383 (Admin).
- Philip Young, interview with Alex Baker, Re-Leveraged, 2026; Solicitors Regulation Authority, Garfield authorization, May 2025.
- Lisanne Bainbridge, "Ironies of Automation," Automatica, 1983.