“Human in the Loop” Isn’t an AI Governance Model

By Valerie Chan

Decision rights might be the next big AI governance fight — not model accuracy, not hallucinations, but who actually gets to decide what.

“Human in the loop” is the phrase everyone reaches for. It’s in board decks, vendor pitches, highway billboards, and policy language. It sounds like an answer, but it isn’t.

It ducks the real question: Which human, in which loop, and with what actual authority?

A recent paper, “AI Agents Push Humans Out of the Loop,” makes an argument worth paying attention to. The authors argue that as AI systems become more autonomous, effective human oversight can actually get harder. The skills required to supervise AI well — judgment, attention, and deep domain knowledge — weaken when people rely too heavily on automated systems. That matters, because simply putting a human somewhere in the workflow does not mean that person is exercising meaningful control.

When you give an AI agent more autonomy,  the human “supervising” it often becomes worse at their job over time, failing to exercise appropriate judgment, paying less careful attention, and demonstrating less sufficient domain knowledge. The authors put it bluntly: Oversight typically degrades the overseer.

Someone can technically be “in the loop” and still lack the time, information, expertise or authority to challenge what the AI is doing.

Organizations need to stop asking only whether there is a human in the loop and start asking something much more specific: What decision rights have we actually assigned?

For every consequential AI-enabled decision, I would start with five questions.

1. Who owns the decision? There should be an identifiable person, role or authorized body with final authority. Not simply “the business.” Not “the AI team.” And not an assumption that everyone somehow owns it collectively. Who can approve the decision? Who can reject it? Who can stop it? Who is ultimately accountable for the outcome?

2. What authority has been delegated to AI? This is where governance needs to get specific. Can the AI research? Draft? Recommend? Initiate an action? Communicate externally? Commit money? Change pricing? Access customer information? Execute a transaction? Simply saying “the AI can assist” is not enough. Organizations need to define where delegated authority begins and where it ends.

3. What still requires human approval before action? There will be plenty of situations where AI can do most of the analytical work but should not have the authority to execute the final decision.

Whether the use case is changing a customer’s pricing. accepting contractual risk, making a regulatory representation, taking an employment action, or approving a significant financial commitment, it’s essential to recognize the boundary will differ by organization, industry, and risk level. But the boundary should be clearly defined before something goes wrong.

4. What automatically triggers escalation? AI should not just keep operating until someone gets uncomfortable. Organizations need predefined escalation conditions. That could be a dollar threshold, regulatory exposure, conflicting data, low confidence, unusual customer impact, an exception to policy, or a situation the system has not seen before. When one of those conditions is met, the AI’s authority should stop and the decision should move to a designated person. That is the point of the rule: not simply to flag risk, but to transfer authority.

5. Who owns the outcome after AI acts? This may be the most important question of all. Imagine an AI agent operating exactly within the authority the organization gave it. It follows the rules. It stays within its spending limit. It does not trigger an escalation threshold. And the outcome is still bad.

Who owns that result? Accountability cannot disappear into the technology. Someone still owns the authority that was delegated, the limits that were established and the governance architecture that allowed the action to occur. Once that human authority is established, the operating model becomes much clearer.

If “human in the loop” isn’t doing the work people think it’s doing, what is?

I’d argue it’s a decision-rights map. Something like this:

AI drafts, human decides. AI produces the first pass — for example a memo, an analysis, or a recommendation — and a specific person owns the final call. Not a committee. One person.

AI recommends, human can override. The recommendation usually gets followed, but someone named has standing authority to push back on it. The questions that actually matters are: Who has that authority? Do they have the bandwidth to review the agentic AI work? And will they use that authority when it counts, or just wave things through?

AI acts within defined limits. The agent moves without asking permission every time, but only inside boundaries set by a human such as a spending cap, a pricing floor, a routine workflow step. That’s all fine until the agent stays inside those limits and the outcome is still bad. Who owns that?

AI stops, humans take over. Something trips a threshold: risk, uncertainty, an exception and the system hands it back. What actually trips that wire, and who’s on the hook the second it does?

Human only. Some decisions don’t belong to AI. Period. The real test isn’t whether people agree with that in theory. It’s whether anyone wrote the list of human-only decisions down before an incident forced the question.

PwC has come to a similar conclusion in its own work on AI agents: More autonomy should mean a tighter definition of permissions and limits, not a looser one.

Once you’ve actually answered who decides, who can override, when the use of AI stops, and who owns the result, the rest of the governance work gets a lot less abstract. Policy language stops being generic. Training stops being a slideshow nobody remembers. Guidance for employees becomes concrete. People know, specifically, what they’re accountable for.

Communications plays a critical role here. It’s a job that doesn’t end once the map is created. Technology changes, risk shifts, and someone has to keep telling people what the current rules actually are, not what they were when the policy was first written.

If your team has already started sketching this out, or is still leaning entirely on “human in the loop,” I’d genuinely like to hear how it’s going.