Who manages the AI workers?

Imagine giving an AI agent responsibility for generating new sales opportunities. It identifies potential customers, researches their businesses and decides which are worth approaching. It prepares personalised messages, follows up, answers routine questions and arranges meetings when somebody is interested. For a while, it works well. Then it does something you would rather it hadn’t. Perhaps it makes a claim about your business that isn’t quite true, continues pursuing a prospect after a polite refusal, or offers something that you cannot deliver. Nothing catastrophic needs to happen for an interesting question to arise: who was managing that work?

There are good reasons to think about this as an AI governance problem. The agent needs appropriate permissions, security controls, auditability and clear limits on what it is allowed to do. Current AI risk frameworks already recognise the need to define human responsibilities for operating, monitoring and overseeing AI systems. But governance does not quite answer the question I am interested in, which is whether the work itself was being done properly. That is a management question, and I think it becomes increasingly important as AI changes from something people use to perform their work into something to which we delegate parts of the work itself.

From using AI to delegating work

Most organisations have so far encountered generative AI as a tool. You ask Copilot to draft an email, produce a summary or help analyse a document. It contributes to the work, but a person remains visibly engaged in performing it. Artificial agents create a different possibility. Instead of asking for an individual output, we can give a system an objective, provide access to information and tools, and allow it some freedom to determine the intermediate actions required to pursue that objective.

None of the underlying ideas is new. Autonomous agents, goal-directed systems and multiagent systems have been studied in computing for decades. My own research background is in intelligent and multiagent systems, where questions about goals, roles, coordination, norms and organisations of agents have a substantial history. Organisational approaches to multiagent systems have, for example, used concepts such as roles, goals, norms, authority and organisational structures to coordinate the behaviour of autonomous agents. What has changed is the accessibility of these ideas. Questions that once belonged largely to researchers and specialist developers are becoming practical questions for ordinary managers.

A sales manager, operations manager or business owner can increasingly contemplate giving meaningful work to an artificial agent without commissioning a specialist software engineering project. I use the phrase AI worker cautiously. I am not suggesting that an artificial agent is equivalent to a human employee, and nothing in the argument depends upon treating it as one. I mean something much narrower: we have given it organisational work to do on our behalf. Once we do that, managing the technology is not enough because somebody still has to manage the work.

We may have delegated more than we realise

The boundary between using AI and delegating work is not as obvious as it first appears. Suppose an AI system prepares 100 routine customer responses and a member of staff reviews every one before it is sent. Ninety-five are approved unchanged and five receive closer attention. It would be reasonable for management to say that a human remains responsible because there is a person in the loop and nothing is sent without their approval. But that description tells us surprisingly little about how the work is actually being done.

We need to look beneath the approval step. Who interpreted the customer’s request, decided which information was relevant, selected the appropriate response and framed the answer? How much independent judgement did the person approving it really exercise? There can be a difference between what I think of as declared delegation and effective delegation. Declared delegation is what the organisation believes it has handed to the AI; effective delegation is what the AI is actually doing within the work. A human approval step can give us confidence that responsibility has remained with a person when significant parts of the work, including some of the judgement within it, may already have moved elsewhere.

This is one reason why understanding how work actually happens becomes so important. A process description might tell us that a person reviews and approves an AI-generated response. The work itself may tell a different story: the person reads it quickly, recognises that the agent is usually right and clicks approve. Eventually the activity being performed by the human may be quite different from that represented on the process map. As I argued in The process map is not the work, if we do not understand the checking, judgement, exceptions, informal coordination and compensating activity through which work is really performed, it becomes difficult to know what we are changing. The same applies to delegation. Understanding the work becomes a prerequisite not only for improving it, but for deciding which parts of it we are prepared to give to an artificial agent.

Capability does not confer authority

Once we recognise that work is being delegated, another distinction becomes important. What an artificial agent can do and what it should be authorised to do are different things. Consider an agent dealing with customers. It might technically be capable of reading a customer record, changing that record, contacting the customer, offering a discount and issuing a refund. Those are not simply five technical capabilities; they represent different forms of organisational authority.

The useful management question is therefore not simply what the agent can do, but what we are prepared to allow it to do on our behalf. That includes the information it may access, the actions it may take, the resources it may consume, the commitments it may make and the circumstances in which it must stop and involve somebody else. Governance clearly has a role here because organisations need mechanisms through which authority can be established, constrained and reviewed. The NIST AI Risk Management Framework, for example, calls for human roles and responsibilities to be differentiated, processes for human oversight to be defined, and appropriate proficiency to be maintained among people operating and overseeing AI systems.

Yet an agent can remain entirely within those permissions and still do poor work. Being authorised to perform an activity tells us nothing about whether that activity is being performed well, which is why the management problem remains.

Data access is not organisational understanding

The problem becomes more difficult when work depends upon judgement. There is a tendency to think that an increasingly capable agent can be given the context it needs by connecting it to enough organisational information. Give it access to the CRM, SharePoint, email, procedures, policies and transaction history, and perhaps it has what it needs to act. I think that assumption is too simple because organisations contain a great deal that is never completely represented in their information systems.

People know the history behind a difficult customer relationship. They remember what happened the last time an unusual order was accepted. They know that two colleagues interpret the same policy differently, or that a stated priority is not actually the most important issue this week. They acquire information through meetings, telephone calls, informal conversations and shared experience. The information available in an organisation and the organisation as understood by the people working within it are not the same thing. Data access is not organisational understanding.

This is also why I am cautious about describing artificial agents as exercising discretion. Human discretion is enacted in the circumstances of the moment. We can describe objectives, rules, principles, previous examples and known exceptions, but we cannot necessarily describe in advance everything that an experienced person will notice when confronted with an unusual situation. Yet we can increasingly authorise artificial agents to act in situations in which a human would previously have exercised that discretion. The risk is not simply that the AI might make a mistake; the deeper problem is that we may have delegated action under an incomplete description of how we expected judgement to be exercised.

Seen in this way, escalation becomes more than an error-handling mechanism. It represents the point at which circumstances have moved beyond those in which the organisation is prepared to allow artificial agency to substitute for situated human judgement. Deciding where that boundary lies, and recognising when it has been reached, is part of managing the work.

So who manages the AI workers?

The obvious answer might be IT, the AI team or whoever introduced the system, but I don’t think that is sufficient. The people responsible for the work need to remain involved in managing its delegation, supported by people who understand and can assure the technology through which that work is being performed.

Consider a manufacturing business using AI to help prepare quotations. Drawings and other information might be interpreted by AI to produce an initial proposal, but an experienced estimator may still need to determine whether the proposed solution actually makes sense. After that, somebody with the appropriate commercial responsibility may need to decide whether the resulting quotation is one the business is prepared to make. A technical specialist brings something different again: confidence that the system is operating appropriately, using the right information and respecting its technical constraints.

Simply saying that there is a “human in the loop” tells us very little about this arrangement because the humans are not interchangeable. The estimator brings domain expertise about the work, while the commercial manager holds the authority to make a commitment on behalf of the organisation. Technical expertise provides another form of assurance. Consequential artificial agency may therefore require a supervisory structure, rather than simply a named person nominally responsible for the AI. Responsibility for the work needs to remain connected to people who understand and are accountable for that work, with technical expertise supporting that responsibility rather than replacing it.

A human in the loop is not enough

There is another difficulty with relying upon human oversight: the human needs to remain capable of exercising it. Imagine an experienced employee who initially checks every output from an artificial agent carefully. The agent proves reliable, the employee finds fewer errors and naturally begins to review the work more quickly. Eventually they are no longer really performing the work themselves; they are approving work performed elsewhere. That might be exactly what we intended and one of the ways in which automation creates productivity, but it may also change the capability of the person doing the checking.

As people gain less direct experience of doing the work, they may encounter fewer of its ordinary variations and unusual cases. Their familiarity with the underlying evidence may decline, and as the agent becomes more capable the human may gradually become less capable of challenging it. This is not a new problem. Human-factors research has studied what is sometimes called the out-of-the-loop performance problem for decades. Endsley and Kiris found that moving people from active control towards supervising automation could reduce situation awareness and make it harder for them to take over effectively when automation failed.

This makes the familiar instruction to “keep a human in the loop” less reassuring than it first appears. A human approval step is useful only if the person remains capable of making an independent judgement about what they are approving. Human oversight is therefore a capability, not a checkpoint. If human judgement is the organisation’s safeguard, maintaining the ability to exercise that judgement becomes part of managing the delegation. That may require retaining domain expertise, ensuring people continue to encounter exceptions, examining apparently successful work as well as failures, and preserving access to the evidence required to challenge the agent’s output. Otherwise, an organisation can retain formal human oversight while gradually losing meaningful human oversight.

When one person can manage fifty agents

The issue becomes more interesting as we scale it. Imagine a small business owner using agents for prospecting, research, administration, scheduling, content preparation, customer support and financial preparation, with some of those agents themselves using or supervising more specialised agents. We might still describe this as a one-person business, but the headcount would tell us very little about the amount of organisational activity taking place.

This creates an interesting asymmetry because agentic AI may increase managerial leverage faster than it increases managerial capacity. Being able to delegate work to twenty or fifty artificial agents does not mean that one person suddenly acquires the capacity to understand and supervise twenty or fifty streams of activity. Management has encountered versions of this problem before. Span of control, management information, performance measures, exception reporting and delegated supervision exist precisely because managers cannot directly observe everything taking place within an organisation. Artificial agents do not make those problems disappear; they change their shape.

It would also be a mistake to assume that every artificial agent requires the same degree of human supervision. Where work is highly deterministic and transactional, acceptable outcomes can be specified clearly and problems are readily observable, supervision itself may be delegated substantially to artificial systems. Where the work is ambiguous, variable, consequential or dependent upon situated human judgement, a different arrangement is likely to be required. The objective cannot be to have humans inspect everything agents do because, at any meaningful scale, that becomes impossible and defeats much of the point of delegation. The objective is to create a supervisory arrangement in which the organisation can still determine whether the work is being performed acceptably and intervene when it is not.

Delegation is not a one-time decision

There is one final complication. The conditions under which we decide to delegate work today may not remain the same. The work changes, the organisation changes and customer expectations change. The information available to the agent may change, as may its permissions, tools and underlying technology. Increasingly, agents may also accumulate context, memory or other mechanisms that alter how they perform their work over time.

We do not need dramatic predictions about artificial agents becoming uncontrollable for this to create a management problem. It is enough that the assumptions which made the original delegation reasonable may cease to be true. AI risk management already recognises the need for continuing monitoring and review across the lifecycle of an AI system. The management question is related but slightly different because it focuses on the work being delegated rather than only the system being governed.

This suggests a question that managers should return to periodically: would we still delegate this work, with this authority, to this agent, under these conditions, today? If the answer changes, the delegation should change with it.

Managing the delegation

None of the underlying management ideas is particularly new. Managers have always had to think about delegation, authority, performance, supervision and accountability. Multiagent-systems researchers have spent decades thinking about agents, goals, roles, norms, coordination and artificial organisations, while human-factors researchers have long examined what happens when people supervise automated systems rather than perform the work themselves. What is interesting is that these previously separate concerns are beginning to meet inside ordinary organisations as managers acquire the ability to give artificial agents meaningful organisational work to perform.

That does not make those agents employees, nor does it mean that they should be managed in the same way as people. It does mean that management responsibility does not disappear simply because the actor performing part of the work is artificial. The fundamental management problem is therefore not really how to manage an artificial person, but how to manage the delegation of organisational work to artificial agency. That requires us to understand what has actually been delegated, decide what authority accompanies it, create the conditions within which the work can be performed, maintain appropriate supervision and preserve human judgement where that judgement remains necessary.

As artificial agents become more capable, the distinction between capability and authority will become more important rather than less. An agent becoming capable of doing something does not itself answer whether we should allow it to do that thing on behalf of our organisation. That decision remains ours, as does the responsibility for what follows from it.

Perhaps, then, the most useful question for a manager is not simply what else we could get AI to do. It is who is actually managing the work our AI is already doing?

Further reading and intellectual foundations

The argument developed here draws together several established areas of research and practice. The management questions of delegation, authority, control and accountability are longstanding ones; organisational approaches to multiagent systems have similarly examined roles, norms, goals and structures for coordinating autonomous artificial agents. Human-factors research provides an important warning about what can happen when people move from performing work to supervising automation, while contemporary AI risk-management frameworks are beginning to formalise the human roles and oversight arrangements required around increasingly autonomous systems.

• Endsley, M. R. and Kiris, E. O. (1995), “The Out-of-the-Loop Performance Problem and Level of Control in Automation”, Human Factors, 37(2), 381-394. DOI: 10.1518/001872095779064555.
• Aldewereld, H., Dignum, V., Jonker, C. M. and van Riemsdijk, M. B. (2012), “Agreeing on Role Adoption in Open Organisations”, KI - Kuenstliche Intelligenz, 26, 37-45. DOI: 10.1007/s13218-011-0152-5.
• National Institute of Standards and Technology (2023), Artificial Intelligence Risk Management Framework (AI RMF 1.0) and accompanying AI RMF Playbook.