“Humans in the loop” is not a safety plan

Every AI product in a high-stakes domain reaches for the same phrase: a human stays in the loop. I’ve started treating it as a yellow flag — including in my own work — because it’s usually doing reassurance, not protection.

A human “in the loop” protects no one when that human can’t actually intervene. When the system makes checking its answer harder than doing the work yourself. When the interface quietly nudges toward “approve.” When the person on the hook for a bad outcome is the operator at the end of the line, and not the people who built and shipped the model.

Oversight you can’t realistically exercise isn’t a safeguard. It’s a place to put the blame.

The version I believe in is mechanical, not rhetorical. It names the decisions the person actually owns — the ones the system is not allowed to make alone. It makes verifying the machine cheaper than redoing its work, so review is real rather than ceremonial. And it keeps accountability with the builder: if the system is wrong, that’s on the people who shipped it, not on the clinician who trusted it under time pressure.

I learned the shape of this years before generative AI, designing a consent handshake for Family Link. Supervision could only begin with a real, legible, reversible “yes” from the person being supervised — on their own device, never silently, never by someone else acting for them. The technology didn’t grant the right to oversee another person. It asked for it. That’s the bar I’m holding the healthcare work to: the loop only counts if the human in it can actually say no.