An AI agent in production: from demo to product

An AI agent in production has little in common with the demo that got it adopted: the demo shows what it can do once, in front of someone watching, whereas production requires it to do it every day, with no audience. This guide, from our Business AI: private and at the right cost section, describes what takes you from one to the other, with four public cases and what we run ourselves. For the definition, first read what an AI agent is, and what it is not.

What distinguishes an agent in production from a demo?

A demo answers one question: does it work? Production adds two: who deals with it when the agent gets it wrong, and how do you know it does better than before. The answers come down to seven elements, to be written down before launch:

1. a named owner, on the business side, who answers for the results;
2. a written scope: what the agent does alone, what it gets validated, what it never does;
3. a budget per task, with a clean stop once it is reached;
4. a starting measurement of time spent and errors;
5. an escalation path: the user knows how to reach a human, the agent knows when to hand over;
6. versions of the model and the instructions, which can be rolled back;
7. a log of what the agent read, decided and did.

The guide Building effective agents from Anthropic, the maker of the Claude models, describes two of them: an agent can pause “for human feedback at checkpoints or when encountering blockers”, and it needs “stopping conditions (such as a maximum number of iterations)”.

What does the Klarna case show about service quality?

The trade publication CX Dive retraced the case in May 2025. In February 2024, the payments company Klarna estimated that its AI assistant was doing work equivalent to that of 700 representatives. It also presented the results of the assistant’s first month:

In May 2025, Klarna’s chief executive, Sebastian Siemiatkowski, told Bloomberg that cost seemed to have been “a too predominant evaluation factor”, with “lower quality” as the result. Since then, Klarna has been recruiting human representatives again: customers must know, its chief executive says, that “there will be always a human if you want”.

The assistant stayed: according to a spokeswoman, it was still handling two thirds of enquiries, with response times improved by 82% since launch and 25% fewer repeat issues. The lesson is about measurement: an agent judged first on its savings will be tuned to save, and quality is only defended if it is measured too.

Who is answerable for what an AI assistant says? The Air Canada case

In early 2024, a tribunal in British Columbia, Canada, settled a dispute that the law firm McCarthy Tétrault summarises in a note dated 19 February 2024. Air Canada’s chatbot had led a passenger to understand that he could apply for the reduced bereavement fare after buying his ticket, whereas the airline required a request in advance. Air Canada argued that its chatbot was a separate legal entity, responsible for its own actions. The tribunal rejected the argument and held the airline responsible: the company answers for “all the information on its website”, whether it comes from “a static page or a chatbot”, and it had not taken reasonable care to ensure the accuracy of its own.

A chatbot that informs already commits the company; an agent that acts commits it even more. One that talks to the public must rely on sources you control and say when it does not know, like our AI assistant on your documents. What an AI may decide alone about a person also has legal limits: see automated decisions: can AI decide alone?.

What does Project Vend, the shop run by agents, teach us?

In 2025, Anthropic, with the company Andon Labs, entrusted a small automated shop in its offices to an agent named Claudius. After a failed first phase, which we describe in what an AI agent is not, the second, published on 18 December 2025, describes the attempt to turn things around.

The team changed the model, added tools, including a CRM, and asked Claudius to double-check prices and lead times with its research tools on every product request, which “helped a lot”. A CEO agent, Seymour Cash, joined it: after its arrival, the number of discounts fell by about 80% and the number of items given away by half. But Seymour approved the goodwill gestures that Claudius asked for on behalf of customers about eight times as often as it refused them: if the shop started making money, it may have been “in spite of the CEO, rather than because of it”. Over the course of that phase, the weeks with a negative margin largely disappeared.

The passage we keep:

Procedures and checklists, Anthropic adds, exist for a reason: they provide a kind of institutional memory that helps employees avoid common mistakes at work. An agent in production does well to be given a new employee’s procedures; a supervising agent does not replace them. Claudius improved, Anthropic concludes, but “it’s still vulnerable in lots of important ways”, and the agents still needed a great deal of human support.

How did the British government evaluate its tool before opening it up?

In 2025, the British government tested Consult, a generative AI tool that analyses responses to public consultations. According to its press release of 14 May 2025, the tool drew out the themes from more than 2,000 responses to a Scottish consultation; experts checked and refined them, then the tool sorted each response into them. It agreed with the human analysis “the majority of the time”, with an F1 score of 0.76, a common measure of agreement. The government estimates that it could save about 75,000 days of analysis a year, or £20 million in staffing costs: a projection, for a tool “currently in trial”.

Consult is not an agent, but its approach applies to any agent: measure it against human work, on real cases, before opening it up, and tell what was measured from what was estimated.

What do we run ourselves?

Our examples come from our internal tools and from a portal we operate for a client: design choices, not measured gains.

An agent in the team’s messaging app

You send a photo of a receipt on Telegram; our accounting data-entry agent pulls out the supplier, the amount and the VAT, then files the supporting document. Checks look for duplicates, consistency of amounts and a confidence threshold before any entry is made; one human action remains, approving the record with a button. In production since July 2026, it is one of the tools we run ourselves, and lives in the messaging app, like our business bots on Telegram or WhatsApp.

An SEO engine that prepares, a human who publishes

The search optimisation of this site follows the same split.

This review rule comes from three made-up pieces of content that a model had published on a site we operate for a client.

Our first unsupervised run, on 30 August 2026, produced nothing: the agent running it asked a question instead of deciding, and nobody was there to answer. Its instructions have since forbidden any question along the way: the human’s moment is the review.

A translation that changed provider

On Palmora Property, a property portal we operate, translation into six languages is not entrusted to an agent: it is an AI step in a controlled chain, with a glossary, a persistent cache and a parity check. Changing provider there cut the cost by a factor of 60, and 95% of segments come from the cache without calling the AI again. Isolate the AI step in this way: the day a cheaper model is enough, it can be replaced without touching the rest.

None of these tools was given a broad mission: each has a narrow task, housed in a tool the team already uses.

How do you evaluate an agent before opening it up to the whole team?

Evaluation takes four steps, two of them precautions specific to agents:

1. 01Test.

   Try several models on your real cases, difficult ones included: our guide to evaluating an AI model before putting it into production gives the method, and our AI at the right cost offer compares 3 to 5 models on 20 to 30 samples of your content.

2. 02Set the threshold.

   Agree the acceptable error rate with the business side.

3. 03Open up gradually.

   Start with one or two people who know the task, with a review at 30 days.

4. 04Put a figure on it.

   Count the cost per task, repeated model calls included, as our guide to the cost of AI in business explains.

Where to start

First check that you need an agent: Anthropic recommends adding multi-step agentic systems only when simpler solutions fall short, and our article Automation without AI: when a rule is enough helps you sort it out. If the agent is justified, name its owner and write its scope before the first line of code: the AI projects that pay off are first of all organisation projects, as our analysis of the return on AI in business shows. To choose the task, our digital and AI maturity audit helps you know what to automate before you spend, with a written GO or NO-GO per project. For what follows, from the data to the choice of model and its cost, our guide to integrating AI in business takes the decisions in order.

Frequently asked questions

Sources

1. Building effective agents (new tab)anthropic.com
2. CX Dive retraced the case in May 2025 (new tab)customerexperiencedive.com
3. Summarises in a note dated 19 February 2024 (new tab)mccarthy.ca
4. Second, published on 18 December 2025 (new tab)anthropic.com
5. Press release of 14 May 2025 (new tab)gov.uk
6. Require a human’s approval before an agent uses a specific tool (new tab)docs.n8n.io