Contents5
Every AI model that comes out arrives with its scores on public leaderboards. None of those scores tells you whether it will sort your emails well or summarise your reports faithfully. Evaluating an AI model before putting it into production means answering that question on your own tasks, before your customers answer it for you at your expense. This guide, from our Business AI: private and at the right cost section, gives the method in six steps, then its limits. Choosing the model is the third of the five questions in our guide to integrating AI in business, after the task and where the data goes.
In short
Success criteria are written before the first test, quantified, with a cap on cost and response time.
About twenty real examples, difficult cases included, give a first verdict; more of them make it stronger.
Code scores what can be checked, an AI judge scores what takes judgement, and a human reviews a sample.
The model you keep is the cheapest one that meets every criterion, correction time included.
The test is replayed whenever the model or the prompt changes; at Anthropic, for example, older models are retired regularly.
Why a public leaderboard is not enough
A public leaderboard measures a model on someone else’s questions. Your use has its own vocabulary, its own document formats and its own edge cases. The CNIL (France’s data protection authority), in its questions and answers on generative AI, recommends “starting from concrete needs to choose the most suitable system” and, if necessary, “carrying out or commissioning a tailored evaluation for the intended use”. Anthropic’s guide to testing says the same from the vendor’s side: a test must reproduce the real distribution of your tasks, without forgetting edge cases.
The level of requirement then depends on the use. The CNIL distinguishes writing assistance offered to employees, which carries less risk, from a system that helps make decisions about customers or candidates, which calls for “a higher level of requirement”. Sorting internal emails and shortlisting job applications are therefore not tested with the same rigour.
How to evaluate an AI model in six steps
The method applies task by task: the same model can succeed at a summary and fail at extracting amounts.
1. Write the success criteria before the first test
A criterion has to be measurable. “Classify emails well” cannot be checked; “classify at least 95% of test emails correctly”, for example, can. Anthropic’s guide asks for criteria that are specific, measurable, achievable and relevant to the use, and points out that most uses combine several: for example fidelity to the task, consistency of answers, tone, confidentiality, response time and price.
Also distinguish annoying errors from serious ones: in Anthropic’s example, 90% of errors must remain mere inconveniences. A misfiled email can be put right; a wrong answer sent to a customer, much less so. Add the points the CNIL asks you to check: security, relevance to your need, robustness, absence of bias and licence terms.
2. Gather real examples, difficult cases included
The examples come from your own business: real emails and real documents, in the proportions in which they arrive. Our benchmark starts from 20 to 30 real samples: enough for a first verdict, not for certainty. Anthropic’s guide, incidentally, prefers a large number of automatically scored questions to a few scored by hand.
Do not forget the difficult cases. Anthropic cites four families: an empty or off-topic input, an input that is too long, a badly worded or harmful request, and ambiguous cases on which even humans would struggle to agree. Finally, set some of the examples aside and never use them to tune the prompt: the real score is read from them.
Watch out
A test sends your data to the provider exactly as real use does
If your examples contain personal data, anonymise them, or test at the same level of control as production. Our comparison of the four levels of control over a private AI helps you choose that level.
3. Compare three to five models, including a small one
Test several candidates on the same examples, with the same starting prompt: the model you already use, which serves as the reference, a smaller and cheaper model, and, if your data is sensitive, an open-weights model that you could host yourself. Our benchmark compares 3 to 5. Then adjust each model’s prompt on the working examples, never on the ones set aside: a badly guided model can lose a comparison it would have won.
4. Score the answers: code first, then an AI judge, and a human to check
Anthropic advises choosing, for each criterion, the scoring method that is the fastest, the most reliable and the best at scaling. There are three:
- Code, when the right answer is known: a category, an amount, an extracted field. It is the fastest and most reliable method, but it cannot judge nuance.
- An AI judge, that is, a second model that scores the answers against a written rubric: tone, relevance, the fidelity of a summary. Anthropic recommends a detailed rubric, a simple score (“correct” or “incorrect”, or 1 to 5), a judge that reasons before scoring and, in general, a different model from the one that answered. Its reliability should be checked before you rely on it.
- The human, the most flexible and the most discerning, but slow and costly: Anthropic advises avoiding it when possible.
An AI judge has its biases. The study by Zheng et al., presented at NeurIPS in 2023, shows that it is swayed by the order of the answers, by their length and by a preference for its own answers, and that its reasoning remains limited. It also measures that a strong judge (GPT-4 at the time) agrees with human preferences in more than 80% of cases, as much as humans do with each other.
Our benchmark judges quality in two stages: an AI judge, then a human review. The judge can be a top-of-the-range model: it is expensive per call, but it only reads the sample.
5. Work out the cost of a usable answer, then choose
The price per token is not enough to tell two models apart. What counts is the cost of a usable answer.
Definition
The cost of a usable answer
Everything an answer costs before it is of use: input and output tokens, retries when an answer is unusable, and the time a human spends correcting.
A model that is half the price but needs a correction on one answer in three can therefore end up costing more overall. Also measure response time, decisive for a live assistant, secondary for overnight processing. The token calculation is detailed in our guide to the cost of AI in business.
Pick the cheapest model that meets all the criteria from step 1, and enter it in your routing table with the date of the test and the reason for the choice: later on you will know why it was chosen.
6. Replay the test at every change
A model you have chosen does not stay chosen forever. According to Anthropic’s page on the retirement of its models, the vendor regularly retires its older models, gives at least 60 days’ notice for a public model, and requests to a retired model fail. On 30 September 2026, it thus announced that Claude Sonnet 4.5 would be retired on 30 November 2026, with Claude Sonnet 5.5 as the replacement, and it advises testing your applications with the replacement well before the date. These dates apply to the platforms it operates: Amazon Bedrock and Google Cloud set their own calendar.
A replacement cannot be plugged in blindly: according to the same page, on Claude models from version 4.7 onwards, a temperature set to anything other than the default value returns an error.
So replay the test when the model changes, but also whenever the prompt is modified. Keep the test set in-house, with the criteria and the results: it is what lets you compare a replacement, or another provider, without starting from scratch.
What an evaluation does not tell you
An evaluation judges a sample, not everything the model will produce: that is the limit we state for our own benchmark. The CNIL explains why it matters: a generative model is not a knowledge base, it follows a probabilistic logic and can produce inaccurate results “which can, nonetheless, appear plausible”. “Excessive trust” without appropriate verification can lead to erroneous decisions.
From the field
A client learned this the hard way
On their site, a model published three property projects that did not exist, and they had to be taken down. Since then, our generated content goes through a tool-assisted check of facts, sources and named entities, and human review before publication is a non-negotiable rule for us.
Production is therefore an extension of the test. Our own expense-entry agent has been running on Telegram in production since July 2026: it reads the photo of a receipt and extracts the information, then a human validates before it is written to the accounting system. The CNIL recommends the same caution: never reproduce the outputs of an AI system verbatim, warn users about “automation bias”, and regularly collect their views on the system’s usefulness and limits.
Where to start
A first test comes down to four steps:
- Choose.
Start from the use that costs the most or carries the most risk.
- Define.
Write three quantified criteria.
- Gather.
Collect about twenty real examples with their difficult cases.
- Compare.
Test your current model against a smaller one.
This method comes from the choice of model for the translation pipeline we operate for Palmora Property, a property portal we work with, where changing supplier cut the cost by a factor of 60.
To get this verdict on your own content, our AI at the right cost offer includes this benchmark: 20 to 30 samples, 3 to 5 models compared, a quantified verdict per task, and the dated, explained table that says which model for which task.
Frequently asked questions
How many examples do you need to test an AI model?
Enough to cover your real cases, including the difficult ones. Our benchmark starts from 20 to 30 real samples, which gives a first verdict, not a guarantee of an error rate. Anthropic’s guide recommends favouring volume: many automatically scored questions are worth more than a few scored by hand. The more sensitive the use, the larger the sample needs to be, and the more the human review matters.
How do you know whether a cheaper model is good enough?
By putting it through the same test as the current model: same examples, same criteria, same scoring. If it meets every criterion, it is good enough for that task. If it only fails on one category of cases, give it the rest and keep the more expensive model for that category. Compare the cost of a usable answer, correction time included, and not just the price per token.
Can one AI model judge another?
Yes, for what code cannot check, such as tone or the fidelity of a summary. The study by Zheng et al., published in 2023, measures more than 80% agreement between a strong judge and human preferences, as much as between humans. But that judge is swayed by the order of the answers and by their length, and it favours its own answers: give it a written rubric and keep a human review on a sample.
Do you need to rerun the tests when the provider changes its model?
Yes, and before the retirement date. Anthropic gives at least 60 days’ notice before retiring a public model, and requests to a retired model fail: on 30 September 2026, the vendor announced that Claude Sonnet 4.5 would be retired on 30 November 2026. A test set you have kept lets you compare the replacement on your own examples before the switch, instead of discovering its differences in production.
Sources
- Its questions and answers on generative AI (new tab)cnil.fr
- Anthropic’s guide to testing (new tab)platform.claude.com
- Presented at NeurIPS in 2023 (new tab)arxiv.org
- Anthropic’s page on the retirement of its models (new tab)platform.claude.com
Jim · founder of Brain Plus AI
Also in business ai
Practical guide10 min
The cost of AI in business: tokens, routing and forecasting
What you really pay when you use an AI through an API: input and output tokens, the context sent back with every call, an agent's turns. And a method to forecast the bill before you receive it.
Choose and price9 min
Private AI in business: the four levels of control
A "private" AI is not a product, it is a degree of control over your data. We distinguish four levels, from an API used as is to a model installed on your premises, and a simple way to choose task by task.
Understand9 min
Enterprise RAG: an AI that answers from your documents
RAG connects an AI model to a company's documents so that it answers by citing its sources. How it works, when it is useful, why it sometimes misses the point, and what to set up before opening it to the team.