The Future of AI Agents in Enterprises: Why Human Oversight Still Matters

In this article, I want to explain why the use of AI agents in enterprise environments must always involve some form of human oversight.

We can break down the usage models of AI agents across existing workflows into three main categories:

1. Human + Software + Agent
   In this model, the human manages or directs the agent, involving it directly in operational processes. The agent then uses the necessary software, services, and APIs—just as a human would—to gather data, perform queries, and take the required actions. However, these actions are not completed entirely automatically. At the final stage of critical decisions and operations, human approval or oversight (“human in the loop”) is always required. This approach achieves a high level of automation while still reducing risk, creating a safer and more accurate workflow through human control.

2. Autonomous Agent
   In this model, the entire process is managed and executed end-to-end by the agent, without any human intervention. The agent independently uses services, APIs, and software to achieve its predefined goals: collecting data, analyzing it, taking action, and making decisions on its own. Human oversight is either absent or extremely limited.

3. Human + Software
   This reflects the status quo today. To complete tasks, humans rely on software services, APIs, and interfaces to carry out the work manually.

At present, the vast majority of companies automate their business processes with a combination of humans and software, without the involvement of AI agents. In the near future, however, as AI agents become more widely adopted, we will see that the most efficient workflow is the Human + Software + Agent model.

In this model, agents analyze data, gather information, and propose solutions; software then routes these outputs to the relevant services; and finally, humans review the results and take the necessary final actions. This approach delivers such a significant productivity boost that tasks currently requiring 10 people could be managed by as few as 4. The efficiency gains made possible by AI represent a leap not seen since the Industrial Revolution.

Take the onboarding agent workflow as an example. With one of our clients, we transformed the client onboarding process using the Human + Software + Agent combination. A fully autonomous process was not feasible—within enterprise environments, the risk of error is unacceptable. After transformation, 92% of the onboarding process was automated by agents and could be completed in just 2 minutes. The remaining 8% was handled by a three-person team in about 30 minutes, with the final control left to humans. By contrast, our client’s competitors still run the same process with 13 people over the course of two full days.

Agentic transformation, in this sense, creates an extraordinary competitive edge.

Human + Software + Agent: The Most Reliable Model for Enterprises

The use of autonomous agents is generally suited to non-critical internal processes where mistakes do not lead to serious consequences. For example, startups might deploy autonomous agents for tasks like bulk email distribution, where small errors can be tolerated. However, for large enterprises—and especially for regulated institutions—the margin for error is too high for fully autonomous agents to be viable in critical processes.

Another factor limiting the adoption of autonomous agents is that the current human + software model used in enterprises often operates with fewer errors than a purely autonomous setup. To illustrate with a real-world example: in a correspondence unit, clerks review letters received from courts and municipalities and classify them into categories such as asset seizure, precautionary measures, or declarations of assets. The clerk then forwards the court’s request to the relevant department, which, depending on workload, responds as quickly as possible.

With the agent, we interpret letters sent by the courts and decide which department they should be routed to. The agent extracts the individual’s details from the document and identifies the appropriate action. It then makes an API call to select from a set of predefined responses to be sent back to the court. After review by the responsible clerk, the final response is submitted to the court. In this way, we’ve implemented a comprehensive agentic transformation: incoming court documents are interpreted, classified, and actions such as asset seizures are initiated, with human-in-the-loop oversight ensuring that the ultimate decision remains with people.

Why Full Autonomy Breaks Down in Critical Processes

Placing a fully autonomous agent in such a critical process would be the wrong approach. In our most sensitive workflows, humans already operate at near-zero error rates, and for agents to be viable in these processes, they must come close to the same standard.

The accuracy achieved today by the human + software combination can only be matched when extended into the human + software + agent model. By contrast, attempts to use fully autonomous agents in critical workflows consistently end in failure. This is especially true in highly regulated environments, where relying on local LLM models for autonomous operations is simply not practical.

In summary, human oversight remains essential in the use of AI agents. Looking ahead, we foresee that the vast majority of agent-powered processes in enterprises will be executed through the human + software + agent model, rather than full autonomy.