Pro Global: When AI Starts Acting – The Compliance Challenge for MGAs

By Mike Dalzell, Group Head of Governance, Risk and Compliance, Pro Global

The insurance industry’s conversation about AI is already moving on. For the past couple of years, much of the focus has been on generative AI: what it can write, summarise, analyse or retrieve. The more interesting development now is agentic AI: as I am sure all readers already know, these are AI programmes that do not simply produce an answer, but can autonomously take actions, make a series of decisions and adapt what they do in response to new information.

As you may suspect, that raises some questions in my mind from a compliance perspective. A tool that helps an underwriter summarise a submission is relatively easy to place within an existing control environment. A system that can interpret information, decide what should happen next, interact with other systems and potentially influence a customer outcome is something different.

The risk is not necessarily that the AI gets one decision spectacularly wrong. More difficult is the possibility that a system begins to behave slightly differently from the way the organisation intended, and does so repeatedly, at scale. That is the issue I think firms need to start spending more time on.

Most governance frameworks are built around reasonably stable things. A business defines the proposed use, considers the risks, approves it, implements controls and periodically reviews whether those controls remain effective.

Agentic systems challenge that model because the thing being governed may not remain entirely static. An agent can be given an objective rather than a fixed sequence of instructions. It may choose between different routes to achieve that objective, respond to changing data or interact with other technology in ways that are difficult to anticipate exhaustively before deployment.

That does not make those systems inherently unsafe. It does mean that a compliance assessment carried out six months ago may tell you rather less about today’s risk than firms are accustomed to.

The important question therefore changes from “Did we approve this use of AI?”, to “Do we still understand how it is operating?”

For an MGA – or indeed any insurance business – that could matter in underwriting, pricing, claims, fraud detection, servicing or any other process where an apparently small change in system behaviour could shape an unintentional new pattern of customer outcomes.

A model does not need to be explicitly instructed to treat customers differently for a conduct issue to emerge. It might discover that one route is more efficient, favour one category of data over another or reinforce a pattern contained in previous decisions. If that happens thousands of times before somebody spots it, the compliance issue is very different from a single human error.

This is where I think the role of compliance begins to change. We are used to compliance acting at defined points: reviewing a proposition, challenging a process, signing off a change, monitoring outcomes afterwards.

With agentic AI, compliance has to be more closely connected to the operation of the system itself, and in particular to a firms IT/ IT Security functions. That does not mean compliance teams need to become data scientists. They do, however, need a sufficiently good understanding of how an agent works to challenge the right things.

• What objective has it been given?
• What decisions is it permitted to make?
• What happens when it encounters something outside its expected parameters?
• Which decisions require human intervention?
• How would we know if its behaviour had started to drift?

Those questions are more useful than asking whether a business “uses AI”, which is rapidly becoming too broad a question to tell us very much.
The unit of governance I think needs to be the use case, not the technology label. A low-impact productivity tool and an autonomous claims decision process may both involve AI, but they plainly should not be governed in the same way.

Human oversight only works if humans can actually intervene There is also a danger that “human in the loop” becomes a comforting phrase rather than a meaningful control.

Human oversight is only useful where the individual concerned can understand what the system has done, recognise when something looks wrong and has both the authority and the practical ability to intervene.

If a person is presented with hundreds of automated recommendations a day and routinely accepts them, the fact that a human technically approved each decision tells you very little about the strength of the control.

The same applies to explainability. Firms do not necessarily need to be able to describe every mathematical calculation taking place inside a model. But somebody within the organisation must be able to explain why the system is being used, which information materially influenced an outcome, what limits have been placed around it and how a decision can be challenged.

That is particularly important where an AI-supported process touches customers. The more autonomous the system, the more important it is that the escalation route remains genuinely human. The supplier question becomes harder.

For many MGAs, another complication is that the AI will not be theirs. Capabilities will increasingly arrive embedded within policy administration platforms, underwriting tools, claims systems and other third-party technology.

That creates a difficult governance question: how far into a supplier’s technology does an MGA need to look? There probably is not a single answer. But accepting “our vendor manages the AI” will not be sufficient where that technology materially influences decisions for which the MGA remains responsible.

Traditional supplier due diligence tends to focus on areas such as resilience, security, data handling and contractual performance. Agentic AI adds another layer.

Firms increasingly need to understand what level of autonomy a supplier’s system has, whether its behaviour can change over time, what monitoring the supplier performs, how material changes are communicated and what evidence is available if an outcome needs to be reconstructed.

There is also a concentration issue emerging. A relatively large number of insurance businesses may ultimately depend on a relatively small number of underlying models or AI infrastructure providers. That has implications not simply for technology risk, but for operational resilience and the industry’s ability to understand shared points of failure.

The temptation with a fast-moving subject like AI is to begin by writing a policy. I would start somewhere else. First, find out what is actually happening in the business.

Not simply which formally approved AI projects exist, but where AI capabilities are already embedded in software, where teams are experimenting with tools and where suppliers are introducing functionality into existing services.

From there, firms can build an inventory based on the significance of the use case. The most useful information is likely to be quite practical: what the system does, what it can influence, what data it uses, who owns it, what degree of autonomy it has, which controls apply and how its performance is monitored.

Once that picture exists, the existing governance architecture can do much of the heavy lifting.

Material AI risks can feed into risk registers. Higher-impact uses can be considered through product and conduct governance. Supplier processes can be expanded to address AI. Data protection assessments can incorporate the way models use information. Management information can identify changes in outcomes rather than simply confirming that systems remain available.

The key is to avoid creating an isolated “AI governance” structure that sits apart from the way the rest of the business is managed. AI risk is increasingly going to appear inside underwriting risk, conduct risk, operational risk, data risk, outsourcing risk and resilience risk. Treating it as a completely separate discipline risks missing precisely those connections.

Boards should also resist the temptation to ask for reassurance that an AI policy exists. A much more revealing conversation would be: show me our highest-risk AI use cases, tell me what could go wrong with them, and show me how we would know if it was happening.

That gets much closer to the real governance challenge. Govern the behaviour, not the label There will undoubtedly be more guidance, standards and regulation around AI. Firms should follow those developments closely.

But MGAs do not need to wait for a new rulebook before acting. Most already have the basic governance tools they need. The harder task is adapting them to systems whose behaviour may be dynamic, whose decision-making can be difficult to reconstruct and whose scale means relatively small problems can become significant ones quickly.

That, to me, is the important shift with agentic AI. The compliance challenge is moving away from approving a piece of technology and towards governing its behaviour over time.

Firms that understand that distinction early will be in a much stronger position to use the technology with confidence rather than discovering, after the event, that the system they approved is no longer quite the system they are operating.

Get in touch to continue the conversation: getintouch@pro-global.com