A human in the loop is not enough

Imagine an AI system preparing 100 customer responses. A member of staff checks every response before it is sent. Ninety-five are approved unchanged. Five receive closer attention. The organisation can reasonably claim that a human remains in the loop. But what does that actually tell us? Who interpreted the customer’s question? Who selected the relevant information? Who framed the answer? How carefully did the employee independently assess each response before clicking approve? There can be a significant difference between declared delegation and effective delegation. Declared delegation is what the organisation believes it has given the AI.

Effective delegation is what the AI is actually doing within the work. That distinction becomes more important as AI gets better. When the system is unreliable, people naturally check its work carefully. When it proves reliable, something entirely rational happens: they begin trusting it. Review becomes quicker. The employee encounters fewer of the underlying cases directly. Eventually they may no longer be performing the work at all. They are supervising work performed elsewhere. That can create enormous productivity. It can also weaken the very human capability upon which our safeguard depends.

If people spend less time doing the work, will they still recognise an unusual case? Will they retain enough familiarity with the underlying evidence to challenge a plausible but incorrect answer? This is why I think the phrase “human in the loop” can provide false reassurance. The important issue is not simply whether a human checkpoint exists. It is whether the person at that checkpoint remains capable of exercising independent judgement. Human oversight is a capability, not a checkpoint. If organisations intend human judgement to remain their safeguard around artificial agents, maintaining that judgement becomes part of the design. Otherwise, we can preserve formal human oversight while gradually losing meaningful human oversight.