How to get your agents into production

Executive Summary

We've been working on taking agents to production for about 2 years now. Some of our agent experiments fail while others generate incredible ROIs. You've probably heard that many AI agent experiments fail as well. In this guideline, I'll cover what we've learned in the field from our roughly 2 years of working with agents, the misguided approaches of CTOs, CEOs and CIOs, and how you can successfully position agents within your workflows. I'll explain these concepts by providing examples from agentic transformations currently used in production within enterprise operational processes. I'll try to explain everything in a simple yet comparative and detailed manner, supported by visuals. What I'm about to share applies to agentic transformations in enterprise operations teams. The agent world is truly vast, and some of what I say may not fully reflect the agentic situation at a startup.

The Human Role in an Agentic World

Agents will be embedded within every company's processes, but we'll see agents most prominently within enterprise processes. Startups and SME will save by using agents, but will never save as much as an enterprise. This is due to agents' current level of success. Agents' context management and memory are not sufficient. Their poor performance in context management and memory prevents them from doing many different things simultaneously or managing dynamic tasks. Agents have started transforming most of the repetitive back-office work in enterprises. These are typically repetitive, well-articulated tasks where the instructions for each step are very clear. When you give agents this type of task, they can perform the tasks very successfully. But think about an SME or a startup - no one has a single, fixed job. Everyone has to deal with multiple tasks simultaneously. This requires extensive context and knowledge management:

What's the latest situation in our company?

Have there been changes to our product?

Were we able to collect payment from the customer?

99% of the operations that exist in a large company haven't even started in a small company yet. Because AI agents cannot yet manage context and information as well and successfully as humans, they cannot perform independent, ever-changing, dynamic tasks as successfully as we would like. Even when they do, people are not satisfied with the quality of the output. If you give agents stable, unchanging tasks with well-articulated steps, you can get outputs close to the quality you want. As good as a human? No, but sufficient for task completion.

So will agents perform these stable tasks in a fully autonomous way? No. In enterprise workflows, humans make very few mistakes. But agents don't have the same success rate - they can't deliver outputs as successfully and consistently as humans. Therefore, humans will become the operators of agents. Agents will be positioned in companies not as employees, but as technologies that increase their teams' end-to-end productivity. This means humans, not agents, will be responsible for what agents do. For example, the person operating needs to check whether the price was written correctly and whether critical sections were configured properly when working with an agent to prepare a price quote document for a customer.

There's a lot of noise around agents. In our 2 years of work in enterprises and the agents we've taken to production, we see that expectations toward agents are extremely high. We believe people's expectations are exaggerated due to this noise. Agents are definitely an important revolution - in fact, they're providing a productivity increase unseen since the industrial revolution. There's a breathtaking productivity boost and competitive advantage. But people's expectations are like expecting cars produced after the industrial revolution to suddenly be magnificent - they expect agents to work perfectly like Aladdin's magic lamp.

We see that it's highly unlikely for agents to replace teams in enterprises, for work to be completed without humans, and for fully automatic structures to emerge. We don't think it's right to position agents as fully autonomous, with approaches like having one agent evaluate another agent's output, agents forming teams, and outputs being managed by agents. We know there are agentic experiments that definitely won't succeed, especially when used with local models. We're currently forced to use local models due to privacy concerns when working with Fintechs. Local models are definitely not good at tasks requiring large models, like forming teams from agents.

Human intelligence, approach, creativity, and the ability to bring independent things together still make humans unique.

What is Agent Orientation

In the human resources world, orientation programs are recognized as critical processes that ensure new employees' adaptation to the organization. This process aims for teams to deliver successful outputs in the company's workflows as quickly as possible. Similar processes need to be designed for AI agents with technical teams and domain expertise. Every company's workflow is independent from one another. Even in the SaaS world, enterprises wanted custom solutions for their own workflows. The same thing is happening with vertical agents for enterprise workflows. Enterprises requested customization even in SaaS solutions that didn't fit their workflows. Due to small nuances in their processes, they will need even more customization in agentic solutions. To better understand agent orientation, I find this example very accurate: How quickly a person working at one company can start on tasks given to them when they switch to a competitor is how successfully an agent solution brought to you can deliver results in your company.

What Is the AI Agent Orientation and Onboarding Process?

AI agent orientation and onboarding is the process of transferring to AI agents how work is done, what to pay attention to, and what the measures of successful task completion are, so that AI agents developed for our workflows can work successfully in those workflows.

In our current workflows, when someone new joins teams, they go through an orientation and onboarding program. Through this program:

1. The company's mission, values, culture, policies, procedures are taught

2. How the work done in the department is performed is transferred in detail

3. How tasks are completed, what to pay attention to step by step is shown

4. What should not be done, how to react when faced with a negative situation is explained

If a person doesn't go through this program, the outputs of their tasks won't be of sufficient quality. The language they write won't reflect the company's language and objectives. AI agents also need the same kind of orientation and onboarding process.

When technical teams perform the agent's onboarding and orientation, they should work together with the team lead from the team undergoing agentic transformation to transfer in detail to agents how work is done, what needs attention at each step, and what the agent should do when unexpected situations occur. Otherwise, agents cannot be taken to production. Even after agents are taken to production, some small improvements still need to be made. So the process of taking agents to production is not as simple a process as one might think.

So what should be transferred to agents during the orientation and onboarding process?

Workflow and Standards

Detailed Process Procedures: For every document or record processed within a workflow, the exact step-by-step procedures and critical checkpoints must be clearly conveyed to the agents. Without this level of detail, agents are likely to produce inconsistent outcomes.

Quality Control Criteria: Success metrics and quality standards must be explicitly defined for each type of document.

Edge Cases: Clear guidance should be provided on how agents are expected to handle unusual or exceptional scenarios.

Error Scenarios: Agents should be familiar with common error situations and the standardized responses required in each case.

Like humans, AI agents also need detailed guidance, clear instructions, and comprehensive context to be successful.

Orientation programs designed through collaboration between technical teams and domain experts ensure that agents work reliably, efficiently, and consistently in production environments. Without this systematic approach, agents exhibit variable performance and undermine organizational trust. When developing software, we think in detail about many aspects like the experience it provides, how it will be used, how it will be positioned, etc.

In addition to these processes, the slowness in enterprise decision-making processes also has a negative impact. Developing agents doesn't take time - think about it: how long does it take to develop a software project? But enterprise decision timelines, document delivery, testing, teams being busy, going on vacation, etc. greatly extend the time to take agents to production. If the company or consulting firm chooses the wrong agentic case during this process, if they choose a task that agents shouldn't do, they curse the entire process.

Let's take a look at agents built with wrong approaches

Now let's first successfully implement an agentic transformation of a workflow, then examine how the same agentic workflow can fail. I think this is a multi-billion dollar agentic case!

First, let's try to understand this case in detail, which exists in most financial companies.

Case: Banks and fintechs receive letters and notifications from courts. The subjects of these letters are seizure, divorce cases, obtaining financial information, etc. During legal processes, courts send court letters or petitions to enterprise institutions regarding their customers' data.

There are 3 frequently encountered situations:

1- In these petitions, the court requests that Company X's customer accounts be seized due to their debt, that information about how much money is in their accounts be reported to the court, and that their accounts be seized for a certain period.

2- When persons X and Y are getting divorced, precautionary measures are placed on both parties' bank accounts for the division of assets during the divorce process.

3- Due to a lawsuit filed against person X, information is requested about the person's bank accounts, e-money accounts, or wallets.

In this case, in the country where we are currently implementing agentic transformation, incoming emails come in .zip file format. Therefore, files are downloaded and then opened, and relevant actions are taken.

Now if you understand the agentic case, we'll first make this agentic case successful, then we'll turn it into a failed agentic case.

How can we maximize ROI with agents in this case?

1-) Each type of letter and case type coming from the court needs to be separated.

2-) In some cases, courts only request information while in other cases they want the bank to take action (such as seizure). There are approximately 20 types of letter categories. There are 2 different action situations that can be taken, so we have a total of 40 different situations.

3-) It needs to be determined what kind of action is taken within the bank for which case types.

4-) The main banking application or third-party tools that people currently use to write responses to letters from the court. The APIs of these tools need to be given to agents.

5-) Analysis of the structure of responses sent to the court and detailed coding for agents on which mail to use and edit in which situation. Creating separate prompts for each case.

6-) You need to write an integration to pull the emails sent by the court into your system. We're writing a mail integration.

7-) User experience was considered at every step. Files coming in zip format were automatically downloaded and texts were extracted. The user didn't spend any effort on retrieving files from the court.

8-) Actions to be taken - seizure, if information is to be provided to the bank, preparing the requested information was completed by making requests to the banking application.

9-) Agents presented the results of their operations with an experience where humans could approve, reject, and redo, or where humans could take action themselves.

10-) No agents were used anywhere the software would be used.

Weakening a Successful Case: How to Undermine Agents by Following the AI Hype

Sometimes, simply following the AI hype without carefully designing workflows leads to failure. Let’s look at how this successful case could be undermined, how the user experience could be degraded, and ultimately how agents could be made unusable.

This successful case would fail if the following critical mistakes were made:

• Stopping at classification: If court documents are merely classified but no further action is taken (e.g., seizing assets), then only a tiny portion of the workflow has been transformed.

• Manual uploads instead of automation: If people are asked to manually upload files into a screen instead of automating steps like email integration, the workflow loses efficiency.

• Replacing deterministic APIs with LLM-based uncertainty: Instead of integrating the agent’s output into core banking APIs with guaranteed execution, using MCP (and involving an LLM) sacrifices certainty for probabilistic output.

• Misusing OCR vs. VLLM: For local processing, if OCR is replaced with a VLLM model to read emails, the reliability of document extraction is severely compromised.

• Overloading the agent with all tasks at once: Asking an agent to classify, extract case details, and trigger actions in a single step increases hallucination risk. Breaking tasks into smaller, focused steps is essential for accuracy.

A Critical Note on Tooling: Horizontal vs. Vertical Applications

Horizontal LLM-based tools often overshadow vertical applications. For example, you might adopt a classification tool or a parser like LlamaIndex. These tools can complete specific tasks but don’t provide true customization. Nor do they claim to. At best, they deliver a partial agentic transformation, not an end-to-end one. Without designing the entire workflow holistically, you only automate fragments.

A Concrete Example

In this case, instead of just classifying emails, a properly designed workflow could send requests to the core banking system, retrieve the required information, and execute the appropriate action. But in the failed scenario, all that happens is a glance at the contents of a .zip file containing emails, followed by classification. Nothing else.

No further steps are taken in the banking application, no requested information is retrieved, and no enforcement action (like asset seizure) is triggered. The result? A partial, superficial transformation that falls far short of the promise of agentic operations.

As a result of these wrong moves, this billion-dollar agentic case will fail to work. This great agentic case will be shelved. Because:

1- Agents were used where software could be used.

2- Only 30% of the task could be completed because requests weren't sent to the main banking application's APIs.

3- MCP was used unnecessarily.

4- The experience that humans operating agents would have was never considered.

5- Tools used by humans weren't given to agents.

Ultimately, a very correct agentic case failed because it wasn't designed correctly, and board members and C-levels became convinced that agents are hype and cannot be taken to production.

Other Wrong Approaches We See:

Agentic transformation should be led by a senior developer or senior data scientist who is involved with agents. This person should be both senior and curious - if they're not both senior and curious, your success rate will decrease. Additionally, they should be someone who has previously solved real-life problems with coding. However, in companies, agent-related work and experiments are given to interns or junior developers with 1-2 years of experience. Because senior developers aren't so enthusiastic about learning agents and they also have a fear of failure. They don't want to put themselves at risk. Therefore, they hand the job over to a junior. Rest assured, only 30% of agentic transformation is related to LLMs and agents. The remaining large portion requires architecture knowledge, development knowledge, and knowledge of how to solve a real-life problem with coding.

Senior developers should put their hands under the stone, not leave their teams alone in the AI agent world that we can consider new, and not load their own responsibilities onto others' shoulders.

Agent experience

I won't explain the agent experience topic in detail. When developing agents, the people or companies that develop agents and serve them to humans need to perform agentic transformation without making the existing workflow even more difficult. However, in the field, we see that in agentic work done under the name of agentic transformation, performing tasks with agents becomes even more difficult. Companies add a step that the user wouldn't normally take, a tool that's not in the workflow comes into play and extends the workflow, etc. There are many examples. Software creates a user experience (UX), and the same applies to agents. In the agent user experience, some aspects should be automated, while others should be streamlined. But in the field, the opposite is happening.

Remember the example I gave above about responding to emails from courts. If teams' workflows become even longer; for example, an experience like downloaded zips being sent to a central location and then taking action with the results from there is very bad for the above case. Such processes that can be converted to automation need to be solved with software, then zips should be automatically downloaded with the written mail integration and then given to the LLM as text. People should be able to complete in 3 steps with agents what they normally do in 5 steps.

Organizations think that the design of agents depends only on the success of models, but the reality is: alongside model success, development architecture, Agent UX and UI are also very valuable steps. Companies that develop agents in organizations or sell agents to organizations don't proceed with a proper implementation process due to concerns about process extension and purchase process extension. You can do this implementation process by understanding in detail how a person does that job, then designing separate workflows for each enterprise if necessary. Correct transformations can be made by giving agents the APIs that people use, with UX, establishing the right architecture structure, and selecting the right-sized LLM.

Another mistake I see in the field regarding agent experience is that if agents are being made for a team that's not very important to the company, not enough care is shown. In other words, the same degree of care isn't shown to agents made for operation teams as to AI data analyst agents made for C-levels. It's annoying, but I have to say it.

Digital transformation and its relationship with agents

Organizations have been trying to achieve their digital transformations for years. It's a deep and challenging process. Digital transformation forms the foundation of agentic transformation. When agentic transformation is attempted in sectors or places where a good level of digital transformation hasn't been reached, the success rate decreases. Digital transformation is an indispensable part of agentic transformation. When performing the agentic transformation of a workflow, some time should be allocated for digital transformation at necessary points related to the workflow. Because only 30% of agentic transformation is related to LLMs. The remaining 70% is related to digital transformation, software architecture, user experience, and identifying the right agentic case.

Agents should not be used where software can be used when performing agentic transformations of workflows

There's a frequently seen mistake in companies doing agentic transformation. Regardless of a person's seniority, everyone makes the same mistake. They try to use agents for tasks that should be completed by writing code/software, tasks requiring digital transformation that don't require LLMs. There are two reasons why they do this:

1- They see LLMs as a magic box because the media has been saying this for the last 2 years. 2- Due to the fact that making any transformation in an enterprise is very difficult, agents are expected to complete all necessary steps for the company. Because getting a requested API, firewall access in a company takes days. If someone needs to work on something and create new things, getting permissions and such can take years. Therefore, they want agents to complete the transformation without bothering us. It's actually a very well-intentioned thought.

So what kind of problems do we encounter when we use Agents or MCP instead of software?

Software is deterministic. The action we want to take in the next step is taken with 100% certainty. We don't encounter surprises or unexpected things. But LLMs are not deterministic. They resemble humans, not machines, which is why they are important technology. It's necessary to define very well what should come in the next step and the expected output. We don't have the 100% certainty in LLMs that we have in software. AI agents are more fragile structures compared to software.

When we use AI agents instead of software, we give up reliability. AI agents already make some mistakes when they work. You cause their error rates to increase even more.

Imagine a workflow has 7 steps - let's say 4 of these steps can be solved with software and let's solve 3 with AI agents. Remember the example of emails coming from courts to banks above. I need a mail integration - I can pull emails using MCP. But instead of using MCP, I can write a mail integration. Writing a mail integration is more laborious but guarantees that my established flow won't break. I can build a structure that will work in a stateless manner. When MCP is used, there's a possibility of not being able to respond to requests sometimes due to load. As a result, this will cause some operations to be missed and since the court doesn't get the return it wants regarding the person or institution, it will fine the company it's dealing with. Ultimately, we won't get the output we want from agents.

When I used agents instead of software in 4 of the 7 steps, I experienced a decrease in the case's success probability. But when I use software in the 4 stations where software should be used, I'm sure of the guarantee of outputs. Now I'm left with the task of designing agents in great detail for the 3 steps that require reasoning, meaning that require me to use agents.

If I approach from a different perspective, you don't expect humans to do things that machines should do, robotic tasks. We use Excel or try to make automations. We use software. It's necessary to see what agents can do as what humans can do. Therefore, solving with software is valuable. Agents aren't great structures that provide software + reasoning. They only provide reasoning.

Also, there's a very important point. Local model usage - if you're forced to use local models in your company due to data privacy. You have to solve more things with software when designing agents, you must define inputs and outputs better. Since we work with financial institutions, we're forced to use local models. You can access our insights and experimentation about local models from here.

In general, we see that using agents in our current workflows is not that simple. It's very difficult to produce something successful within the enterprise's sluggishness. It becomes difficult for companies to deal with agents when they're doing things different from their main business.

How much efficiency do correct agentic cases create?

We've always talked about how taking agents to production is not an easy task. So how big an ROI do we achieve when we take agents to production? The size of ROI varies according to the company's size and team size. There's a different ROI situation for each case.

Currently, in our onboarding agents, since the outputs' results can be verified from governmental APIs, in the client onboarding process that 30 people do, all data is verified, intelligence is gathered, and tasks are completed by agents in a way that's ready for action. This way, the work that 30 people do can be completed with 5 people and most importantly, it becomes scalable.

In the operations that banks perform regarding information requests and seizure demands from courts, which I gave as an example above, when agents are used, the workload decreases dramatically.

The productivity increase each case creates varies. But here's a fact: As humanity, we hadn't encountered a technology that increases productivity this much since the tractor (industrial revolution). Agents create incredible ROIs in the right cases.

Beyond the efficiency agents create, the real big impact shows itself in terms of competitive advantage. An enterprise that can position agents correctly can redirect 30% of its workforce to non-robotic processes that require more intelligence and are more critical for the company. Companies currently using us can make client onboarding completely scalable with 2-3 people. A company not using agents does the same job with 20 people.

The work done in less time, with less cost, and higher accuracy with agents is done by our customer's competitor in longer time, with more cost, and more operational complexity. In companies using agents, the gap created over competitors is spine-chilling

how can you compete?

Why agentic workflows should be integrated and what kind of system design should be implemented

When developing agents, there are many interconnected workflows within companies. These workflows are separated for different teams to handle due to the volume of work. In agentic transformations, such workflows should be transformed piece by piece by determining the initial starting point of the flow. When combining these pieces, it's necessary to provide software solutions where needed with the reasoning that agents provide, along with digital automations that we couldn't do before in our software due to the lack of reasoning, even if minimal.

For CIOs

I see CIOs in the ecosystem don't believe in agents. This is due to the excessive noise created by AI hype. Developers are very talented and equipped people who have dedicated their years to technological transformation and solved many business problems - they are the people who built the technological infrastructure of our current world. What I love most about developers is that when they see something that's bullshit, they know it's bullshit. Due to extreme statements about AI and excessively high expectations, CIOs believe that agents will fail badly and fail terribly. But the reality is: agents are not as successful as hyped - they won't leave humanity unemployed to the point where we'll never work again. However, agents are also not as incompetent and unusable as CIOs think. They generate big ROI in enterprise operational processes, in branch work, wherever there are repetitive tasks.

I see this problem with senior developers in the ecosystem too. There are developers who still haven't tried Cursor and don't write code with AI. AI is being taken too lightly. When you find the right agentic cases, incredible ROIs are created. Let me explain with an example we've done, experienced firsthand, and taken to production:

We sold our client onboarding agent to an overseas customer. Merchants and SMEs who want to get POS from our customer need to be onboarded (their documents need to be checked). During this onboarding process, their certifications and tax number checks are performed. Then they gather social intelligence by researching company owners. Money laundering lists are checked, etc. A customer's onboarding takes 30 minutes, but due to workload, a customer could only be put through the KYB process after 2 days. The company was doing the job with 13 people in 2 days. With our agents, the work that was done in 30 minutes can now be done with 3 people in 5 minutes. Additionally, we analyze more intelligence than what humans gather.

CIOs in traditional sectors are turning a deaf ear to developments regarding agents. However, agentic transformations at the right points, the very high ROIs that agents create, and the disruptive competitive advantage require thinking twice about this ear-plugging.

Committees responsible for agentic transformation

As Clausewitz said, "The farther managers are from the action, the more they must see through other people's eyes, and this increases the risk of error."

Since the agent field is very new, neither current managers nor current teams have received training on AI. We don't yet definitively know AI best practices, how we should design agents, what the right approaches are. This know-how is slowly forming - within the next 2 years, AI know-how, approaches, what is agentic and what isn't agentic will all be clearly understood. Then this know-how will slowly begin to become commoditized.

1- Since the field is new, everyone is acting through someone else's mouth. Because there's noise, all middle-level managers are confused. Senior executives say they want such an agent and then disappear. The job is difficult for the middle-level managers who do the work.

Sometimes we encounter this in meetings - basic information isn't known by managers, which isn't their fault due to the field being new, but their biggest mistake is not researching themselves and assigning the process to a junior or middle-level person - the cost of this to the institution is very heavy.

2- The number of developers in created AI committees is too low. Developers will do agentic transformation today just as they did digital transformation yesterday. The vision of a senior developer dealing with agents is very valuable for the company. You must keep the number of developers/technical people in committees at more than half. The developers trying to develop agents need to be inside the committee. Reducing the C-level number is a very correct step. C-level, board members, managers and such people who have left coding - yes, they contribute vision-wise, but without execution, vision is useless. We're going to discuss a new technology, but most of the people who will discuss it don't have command of the subject - this is ridiculous. Those who get their hands dirty should be there.

You should make agents do the work that humans do the way humans do it. In tasks that humans don't do and whose boundaries cannot be well defined, agents' success rate will be low.

When developing agents, due to the noise about agents, all sorts of inappropriate tasks are wanted to be done by agents. Agents are expected to perform tasks that exceed human capabilities or that humans don't want to be assigned due to complexity. What's correct is that when creating an agentic workflow, you need to copy exactly how a human currently performs that task, with every step that humans use to complete that task in each phase. Every detail and step should be transferred to agentic products.

Agents are not better and more successful than humans. You can't do a task with agents that you can't do with humans. However, you can complete a stable, repetitive task that you do with humans at incredibly high efficiency with agents.