Automation bias: when a human “in the loop” stops actually deciding

"A human is always in the loop" is offered as reassurance more often than it is actually tested. Automation bias, the well-documented tendency for a person to defer to an automated recommendation even when their own judgement would say otherwise, means a human in the loop can become a human rubber-stamping the loop, without anyone deciding that on purpose.

Why deference happens, even to people trying not to

Overriding a system takes more effort than accepting its output, and if the override is later shown to be wrong, the reviewer who overrode it often bears more visible blame than one who simply followed the recommendation, an asymmetry that quietly rewards deference over independent judgement, especially under volume and time pressure that make careful, case-by-case scrutiny genuinely hard to sustain.

The gap between "can override" and "actually does"

A system being technically overridable is a necessary condition for meaningful human oversight, not a sufficient one.

The real question is the override rate in practice: if a human reviewer disagrees with a model's output on a genuinely single-digit percentage of cases, that is either evidence the model is very good, or evidence the human has stopped meaningfully reviewing, and those two explanations look identical from the outside without deliberately checking which one is true.

What meaningful human oversight actually requires

Enough time and information to form an independent view before seeing the system's recommendation, not after, since seeing the recommendation first is itself a well-documented way to anchor a reviewer's judgement toward it. A genuine cost to deferring incorrectly, not just to overriding incorrectly. And periodic testing of the override rate itself, treating a rate near zero as a signal worth investigating rather than a sign the system is simply excellent.

Why this matters more as systems get more confident-sounding

A system that states its output with high apparent confidence tends to increase automation bias further, regardless of whether that confidence is actually warranted. The more persuasive a system sounds, the more deliberate the design of the oversight around it needs to be, precisely because the natural human tendency runs the other way, toward trusting the confident-sounding answer rather than scrutinising it harder.