Contents7
In short
Translation quality is measured in errors, sorted by type (meaning, term, style) and severity, not by overall impression.
Automatic checks run on the whole text without reading it: numbers, required terms, missing texts, lengths.
Automatic metrics are for comparing tools, not for approving a page.
A sample reviewed by someone who reads the language remains the arbiter: it decides whether a language can be published.
A website translated into six languages quickly adds up to tens of thousands of sentences. Nobody rereads them all, and nobody should: a reviewer’s time is kept for the pages that commit the company. But publishing without measuring anything means discovering errors through your customers.
Between these two extremes there is a method, the same as for any volume production: check automatically what can be checked, measure the rest on a sample, and track the figures from one release to the next. This article describes it from the point of view of the company that publishes. It extends our articles on website translation.
What you measure: errors, not an impression
“It’s well translated” cannot be compared from one month to the next. A list of errors can. Translation professionals have a common framework for counting them:
Definition
MQM (Multidimensional Quality Metrics)
A framework for analytic evaluation of translation quality, human or machine: each issue found is given an error type, from a hierarchical catalogue (accuracy, fluency, terminology, style and others), and a severity. The annotated errors are then turned into a score.
The definition comes from the MQM website. The ISO 5060 standard, published in 2024, follows the same logic: error types and penalty points, giving an error score and a quality rating, for human translation, post-edited machine translation or raw machine translation, with room for sampling.
For a company, a simple version is often enough: four types (wrong meaning, wrong term, tone or style, form) and three severities (critical, major, minor). A critical error on a price or a condition blocks publication; ten minor style errors in an article do not.
Automatic checks: catching without reading
A machine does not judge a sentence, but it is very good at checking what must be identical from one language to another. These checks run on the whole text, at every update:
- numbers: a price, a dimension, a date or a percentage must appear in the translation;
- required terms: each term in the brand glossary must appear in its approved form;
- missing texts: a key, a field or a block absent in one language;
- untranslated texts: a paragraph left in the source language;
- variables and markup: a
{count}or a link broken by the translation; - lengths: a title too long for its space, a description that exceeds the search results limit.
From the field
On this site, French is the source language, and a check run before every build rejects any offer page whose French and English versions do not have the same blocks, lists and numbers, or that contains an empty text. On Palmora Property, a six-language real-estate listings site we operate, German was opened with 1,092 keys in strict parity with the other languages, verified by a check of the same kind.
These checks say nothing about meaning. They free the reviewer from what a machine sees better than they do, so they can focus on what it does not see.
Automatic metrics: comparing tools
There are also scores computed by a machine, which compare a translation with a reference translation, or assess it without one. Researchers have ranked them. The title of the results of the 2022 WMT annual evaluation sums up the conclusion: “Stop Using BLEU: Neural Metrics Are Better and More Robust”. Metrics that count shared words, such as BLEU, follow human judgement poorly; learned metrics, such as COMET, follow it better.
General-purpose AI models can also act as judges. A study presented in 2023 showed that they reach the best accuracy known at the time at the level of a translation system, that is, to say which tool translates best over a set of texts, not to judge a single sentence.
That is the right use of these scores: choosing a tool, tracking the effect of a model or prompt change, finding the weakest texts to review first. The comparison method is detailed in DeepL, Google or an LLM: which translates your trade best and, more broadly, in evaluating an AI model before putting it into production. To decide whether a page can be published, they are not enough:
Automatic rankings did not always match human judgments, reinforcing that human evaluation should remain the final arbiter of translation quality.
The reviewed sample: the arbiter
The reviewer does not read everything; they read what is needed to reach a conclusion. Five rules make the sample reliable.
- Separate pages by what is at stake.
Pages that commit the company (offers, prices, terms) are reviewed in full; the sample applies to repetitive pages, as detailed in automatic or reviewed translation.
- Draw the sample from each page type.
A few product pages, a few articles, a few emails, rather than the first ten pages of the catalogue.
- Have it reviewed by someone who reads the language
Without telling them which tool produced the text when comparing tools.
- Count errors by type and severity
With the same grid every time.
- Set a threshold before reading.
Zero critical errors, a maximum number of major ones: beyond that, fix the template, the glossary or the instructions, and start again.
For human translation, the ISO 17100 standard already provides for revision by a second person. Machine translation does not remove that second pair of eyes; it focuses it on a sample and on the pages that matter. The reviewer remains essential, and their time is worth all the more when it is well used.
Errors found are not only fixed on the page. A wrong term that keeps coming back goes into the glossary; a corrected sentence goes into the translation memory, so it never goes out wrong again. That is what brings the error count down from one sample to the next.
What no translation measure sees
A faithful translation faithfully reproduces an error in the source text. A wrong figure, an invented fact, a promise the company does not keep: the translation is perfect, and the content is still wrong, in every language.
Watch out
Check the source before translating it
A client learned this the hard way: on Palmora Property, a model published three real-estate projects that did not exist, and the pages had to be taken down. No translation score would have caught the problem. Fact-checking is done on the source text, before it goes out in six languages.
A language can also be well translated and badly adapted: a price in the wrong currency, a form that rejects the country’s addresses, keywords nobody types. That is no longer translation quality, but localization and keyword research abroad.
Where to start
Pick a language already published, draw twenty pages spread across your page types, and have them reviewed by someone who reads the language using the grid of four types and three severities. In parallel, add an automatic check of numbers and glossary terms across the whole site. The first survey gives you your starting point; the next one, after fixes, tells you whether the method works.
Measurement applies to every text in this section: product pages for export, customer support emails, software and apps, blog articles and, with a lawyer as reviewer, contracts and legal documents. It counts twice for languages nobody reads in-house, such as those of China and Southeast Asia, and when opening an English-speaking market or a first European market. Its cost, review included, belongs in the budget described in website translation cost: per word, per page, per market, and the tool chosen for sensitive texts must also answer the questions in translation and GDPR.
To set up automatic checks and a review grid on your website, send us the website address and its languages: we start from a first survey, your reviewers keep the final word.
Frequently asked questions
How do you know if a machine translation is good?
By having a sample reviewed by someone who reads the language, with an error grid set in advance: meaning, term, style, form, and three severity levels. Automatic scores help compare tools, but human evaluation remains the arbiter, as the organisers of the annual WMT evaluation conclude.
What is the BLEU score, and should you trust it?
BLEU counts the words and word sequences shared by a translation and a reference translation. The researchers of the 2022 WMT evaluation recommend no longer using it: it follows human judgement poorly. Neural metrics such as COMET follow it better, but they too are for comparing tools, not for approving a page.
Can one AI review another AI’s translation?
It can triage: flag suspicious sentences, compare two tools over a large volume, find the texts to review first. It does not replace the human reviewer in deciding that a language is ready to publish. Model judges are most reliable at the level of a set of texts, less so at the level of a sentence.
How large a sample should you review?
Large enough to cover each page type, and for the threshold you set to mean something. Twenty or so pages spread across product pages, articles and emails give a first survey; widen it when results are close to the threshold. Pages that commit the company are not part of the sample: they are reviewed in full.
Sources
- MQM, Multidimensional Quality Metrics (new tab)themqm.org
- ISO 5060:2024, evaluation of translation output (new tab)iso.org
- Results of WMT22 Metrics Shared Task, 2022 (new tab)aclanthology.org
- Kocmi and Federmann, Large Language Models Are State-of-the-Art Evaluators of Translation Quality, 2023 (new tab)aclanthology.org
- ISO 17100:2015, translation services (new tab)iso.org
Kocmi and Federmann, Large Language Models Are State-of-the-Art Evaluators of Translation Quality, 2023
Read on aclanthology.org(new tab)Jim · founder of Brain Plus AI
Also in translation and new markets
Choose and price7 min
Translating your website with AI: automatic or reviewed?
Having everything reviewed is expensive, and reviewing nothing lets errors through in languages that nobody at your company reads. The split is decided page by page, depending on what each page commits you to.
Choose and price7 min
DeepL, Google or an LLM: which translates your trade best
Every provider publishes a test it wins, and independent evaluations crown a different winner from one year to the next. For a specialised text, the only reliable answer comes from a trial on your own texts, language by language, judged by someone who reads the language.
Practical guide8 min
Brand glossary: keeping translations consistent
The same product called three different things on three pages, and readers wonder whether there are three products. A brand glossary fixes the terms that are not up for debate, before the first translation, whatever the tool or the translator.