When the human in the loop becomes a rubber stamp: what UK regulators found about AI checks

Somewhere in most organisations using AI, a person is working through a queue of machine-made recommendations, approving each one before it goes out. On the risk register, that person is the control. The uncomfortable question is whether they are still reading.

What the regulators found

On Monday 26 October the Bank of England and the Financial Conduct Authority published their fourth survey of artificial intelligence in UK financial services. It drew 196 responses, with 105 firms completing the full survey.

Adoption is now close to universal among respondents: 86% use AI, and 80% have AI agents deployed. Autonomy is limited. Most agents provide decision support, where the agent recommends and a person decides, and only 2% of respondents have agents with broad autonomy. So in most of these firms, a human really is in the loop.

The more revealing findings are about what happens around that human. More than 90% of full-survey respondents reported human approval, pre-deployment review and vendor due diligence as controls. Ongoing monitoring, leakage testing and periodic review were each reported by only around half. Just 39% monitor customer impact, and fewer than half have incident management and response playbooks. For high-materiality systems supplied by third parties, 81% rely on vendor assurance and contracts, while only 24% say they have deep knowledge of how those systems work.

In other words, the checks cluster at the front door. Once a system is live, firms know far less about how it behaves or what it does to customers.

The human check that stops checking

The FCA's good and poor practice notes on AI, published a week earlier, show what this can look like in practice. They warn against "excessive reliance on human-in-the-loop controls" and describe one firm where the human reviewer role was unclear and "became a tick-box exercise over time". Reviewers increasingly accepted AI recommendations "without sufficient scrutiny" and became less likely to challenge, even where there were signs of errors.

Nobody in that story did anything dramatic. The work arrived faster, the recommendations were usually right, and agreeing became the path of least resistance. The FCA calls the underlying tendency automation bias: placing too much trust in an automated output, even where it may be wrong or incomplete.

The FCA also gives examples of good practice. One firm had reviewers assess affordability cases without seeing the AI's proposed outcome, so it could compare independent human and machine judgements. Others set clear thresholds for when a person must intervene, trained reviewers on a system's purpose and limits, and kept a central inventory of AI agents with named owners.

Why this matters beyond financial services

Banks and insurers are regulated, so they are asked these questions. Most organisations are not, yet nearly every AI programme leans on the same reassurance: a person will check it. If that check has drifted into a signature, the organisation is carrying risk it believes it has removed.

None of this is new. John Sansbury, writing about service management transformation in The Journey to Service Excellence, put it plainly:

"However, do be aware that governance is sometimes little more than a box-ticking exercise."

Applied to AI, Esm reads that as a warning that a control can exist on paper while doing nothing in practice. His book also gives the counterweight, addressed to the people doing the work:

"So as an employee working collaboratively in a team, please don’t be afraid to say, ‘Excuse me, this doesn’t look right.’"

A reviewer who never says that is not reviewing.

Where the AI Enablement Toolkit helps

The AI Enablement Toolkit for Leaders is a leadership method, not software. Several of its tools speak directly to this problem.

- Tool 10, human and AI role design, separates modes such as draft, recommend and act with approval, so the human's job is stated rather than assumed. Its accountability rule is that "The AI did it" describes a mechanism, not an owner. - Tool 11, the ownership and boundaries canvas, records the named owner, approval points, quality checks, escalation route and fallback before a workflow goes live, and when the design will be reviewed. - Tool 16, the minimum governance check, asks whether known failure modes have checks, whether users know how to challenge the output, and whether there is a review trigger after incidents. - Tool 12, the trust-by-design check, includes the pre-mortem prompt "The AI sounded confident while being wrong." - Tool 22, the leader AI enablement dashboard, puts errors, rework and challenge alongside customer outcome, so the measure of a review step is what it catches, not how fast it clears the queue.

Where training fits

The FCA's own description of what reviewers need is close to a training brief: understand the system's purpose and limits, know how to interpret and challenge outputs, manage automation bias, and know when to escalate or override. Senior managers need a different layer, enough understanding of AI risk to challenge it without being technical specialists. That is leadership and reviewer capability, and it is where Esm training and enablement is designed to help.

Honest limits

The toolkit is a management control. It will not monitor a model, detect drift or manage agent credentials; those need technical tools and specialists. It has no agent inventory tool, which the FCA highlights as good practice. It is not legal or regulatory advice, and firms in regulated sectors should check their current obligations rather than recall them. The survey also reflects participating firms, not the whole sector, and the FCA stresses its examples do not create new requirements.

A question to take away

If the people checking your AI stopped reading tomorrow, what in your organisation would tell you, and how long would it take?

Tags: ai-governance, human-in-the-loop, financial-services, leadership, ai-adoption

Sources

- https://www.bankofengland.co.uk/report/2026/artificial-intelligence-in-uk-financial-services-2026

- https://www.fca.org.uk/publications/good-poor-practice/ai-financial-services