Contents7
The question is often asked as “will AI replace this profession?” The experiments run since 2023 invite another one: what is the role of humans alongside AI, and which part of the job gains value when a machine can produce? Their answer is fairly clear. Production loses value; scoping, checking and accountability gain it. The person who makes the difference is not the one who writes the best prompts. It is the one who knows their job well enough to see when the machine is wrong.
In short
On suitable tasks, AI brings gains in volume, speed and quality; on a neighbouring task beyond its capabilities, it leads people to produce more wrong answers.
It helps beginners most. For the most experienced the gain is small, and a 2025 study even measured lost time.
Delegating the task you need to learn stops you learning it, unless the AI is designed as a tutor.
The law is starting to require that a human answers for consequential decisions, and that companies help their staff become proficient with these tools.
The jagged frontier
The most telling experiment was run with 758 professionals, on consulting tasks designed with Boston Consulting Group, and published in Organization Science in March 2026, after preregistration. It compares participants equipped with AI with those who work without it.
more tasks were finished with AI, 25.1% faster and with markedly better quality, on 18 realistic tasks within its capabilities
less likely to produce a correct answer: that is what AI cost participants on a complex management task, chosen to lie outside its capabilities
The authors speak of a “jagged technological frontier”. The line that separates what AI does well from what it does badly does not follow the apparent difficulty of the tasks. Two tasks that look alike can fall on either side. And from the outside, the result can look the same in both cases: a well-written, confident text.
The skill therefore lies in knowing which side you are on. That knowledge is not in the tool. It lies in knowing the job.
AI mostly helps beginners
A study published in the Quarterly Journal of Economics, using real data on 5,172 customer service agents, measures an average rise of 15% in problems resolved per hour thanks to an AI assistant. The gain is very uneven: the least experienced and least skilled agents improve in speed and in quality, whereas the most experienced gain a little in speed and lose a little in quality.
One way to read it: the tool has captured some of the best agents’ know-how and passes it on to newcomers. That is good news for building skills. It does not make expertise useless, quite the opposite. Someone is still needed to design the right answers, spot the ones that drift and train the next people. The expert spends less time producing and more time designing, checking and passing on knowledge.
Among the most experienced, AI may even have cost time. In July 2025, a randomised trial by the organisation METR, with sixteen developers working on projects they had known for five years on average, measured that they took 19% longer with AI, while they estimated afterwards that they had saved 20%. The sample is small, and METR has since said these results are out of date: its February 2026 update judges the effect probably more favourable in early 2026, without being able to put a reliable figure on it. Depending on the task and the person, AI can help a lot, a little, or not at all.
The risk: handing over your judgement along with the task
If the value lies in judgement, the question becomes: does using AI maintain that judgement or erode it?
A study presented at the CHI 2025 conference by Microsoft Research and Carnegie Mellon, covering 319 knowledge workers, finds that the more people trust AI, the less they exercise critical thinking, and the more they trust themselves, the more they exercise it. Two cautions: it is a self-reported survey, so it shows a correlation and not proof that AI atrophies judgement; and Microsoft sells AI tools, which makes the finding all the more notable.
A field experiment goes further. Published in PNAS in June 2025, it covers nearly 1,000 pupils at a large Turkish high school, in mathematics. Some have access to a general-purpose assistant, others to a version designed as a tutor, which guides without giving the answer. Their marks are compared with those of pupils who never had AI.
on marks during the exercises: that is the lead of pupils who have the general-purpose assistant
on marks during the exercises: that is the lead of pupils who have the tutor version
on marks once access is removed: that is how far behind the pupils who had the general-purpose assistant fall
The tutor version largely limits this effect. (A correction was published in August 2025: it concerns only the affiliation of an author, not the results.)
These results concern pupils who are learning, not professionals at work, and do not transfer as they are to a company. But the idea they illustrate holds beyond school: delegating the task you are supposed to learn stops you from learning it. For a company, the question is concrete: how will junior staff be trained if AI does all the tasks that used to teach the trade? We deal with it in our article on AI and young people’s first jobs.
You also read a lot of studies claiming that “AI makes us dumber”. The ones that circulate most are either correlational or preprints on a few dozen people, not peer-reviewed, which we do not take as proof. We go through them, with their limits, in our article on the question “does AI make us dumber?”.
Answering for the result: the law steps in
Human judgement is no longer just good practice. Legislation is starting to require it, in Europe and in California.
- 2 February 2025
In Europe, Article 4 of the AI Act comes into application. It requires companies that deploy AI to take measures so that their staff are proficient with these tools.
- October 2025
In California, the governor vetoes a first bill, SB 7.
- 27 July 2026
The European Article 4 applies in the version rewritten by Regulation (EU) 2026/1744.
- 30 September 2026
In California, the law SB 947 is signed.
- 1 July 2027
SB 947 will take effect.
In Europe, the rewrite changes the aim: where the 2024 text aimed to “ensure, to their best extent, a sufficient level of AI literacy”, the aim is now to “support the development of AI literacy”, with no obligation to “guarantee any specific level” for each person. Taking measures remains compulsory, but the requirement has been relaxed.
In California, SB 947 prohibits an employer from relying solely on an automated decision system to discipline or dismiss an employee; when it relies primarily on such a system, a human must corroborate the decision. Its author, who was already behind SB 7, presents it under the name of the “No Robo Bosses Act”, in a political press release that should be read as such.
The direction is clear: the machine can propose, the human signs. The human becomes answerable for what the machine produces, and must be able to explain it. For decisions that affect a person, such as a dismissal or a loan, what the GDPR and the AI Act provide for is set out in our article on automated decisions by AI.
What it changes day to day
In our own work, the rule follows from everything above: the machine produces, a human validates anything with consequences.
- Anything with consequences goes through human validation. Our expense-entry agent, in production on Telegram since July 2026, starts from the photo of the receipt, extracts the information, then waits for human validation before writing to the accounting system.
- No generated public content goes out without human review. For us this is a non-negotiable rule. On this blog too, a generated article starts life as a draft, and it is human review that decides whether to publish it.
- Large volumes are checked by sampling. For a site translated in bulk by a model, our method provides for reviewing a sample per language, with a judge model and a human, on the first pass and then by spot checks.
From the field
A client learned this the hard way
On Palmora Property, a model published three real estate projects that did not exist, and they have since been unpublished. That is why we check the facts, sources and entities cited in all generated content.
For a team that is deploying AI, we add two pieces of advice, drawn from the experiments cited above. Train people in the job with the tool, not only in the tool: which tasks it is reliable on, where it goes wrong, how to check it. And keep tasks that serve for learning for beginners, where AI supports without doing the work for them, like the tutor in the Turkish experiment.
This is also, in our view, one of the reasons why so many AI projects do not yet pay off: the tool is bought, and nobody redefines who checks what.
What we think
Our position, signed Jim: AI does not replace the professional, it shifts their work towards scoping, checking and accountability. It is not a consolation, it is a different job, and a more demanding one. It requires knowing your field better than before, because the machine’s mistakes are often better dressed than human ones.
We remain cautious on two points. In October 2026, the studies on critical thinking are still fragile, and nothing proves that supporting use, properly scoped, damages judgement. And the gains by level of experience vary greatly from one task to another. We would change our minds if controlled experiments showed that an unsupervised AI does as well as a professional who checks it, on tasks where errors are costly. That is not what the studies cited here say.
The same question arises for creativity, with a similar conclusion: AI produces the average, humans bring the difference. For a first use, choose a tool that leaves the judgement to your team, for example an AI assistant that answers from your documents and cites its source: every answer can be checked. Choosing the model is also a matter of judgement: our AI at the right cost offer compares models on your own content, with cost measured and quality judged, before settling on one per task. Our guide to integrating AI in business explains how to scope this first use, from the task to the rules.
Frequently asked questions
Will AI replace experts?
The available experiments show a shift rather than a replacement. On real customer service data, AI helps beginners most, and the most experienced agents gain little. On some tasks, it even makes professionals’ work worse, and it is expertise that lets you notice. Experts spend less time producing and more time scoping, checking and passing on their knowledge.
Which skills become important with AI?
Knowing which tasks the tool is reliable on, checking its results, setting out the problem correctly and owning the final decision. In other words, professional judgement more than prompt-writing technique. It is also what the law is starting to ask for: in Europe, companies that deploy AI must take measures so that their staff are proficient with these tools.
Do you have to check everything AI produces?
Not necessarily everything, but everything with consequences. AI can get a task wrong that looks like another one it gets right, and the error often looks just like a good answer. Anything that reaches a customer or an employee, or moves money, deserves human validation; repetitive internal tasks can be checked by sampling, as long as you look regularly.
Can a company dismiss someone on an AI’s decision?
In California, this will be prohibited from 1 July 2027: an employer will no longer be able to rely solely on an automated system to discipline or dismiss. In Europe, Article 22 of the GDPR already gives the right not to be subject to a decision based solely on automated processing that significantly affects you, with exceptions. And the AI Act classes these systems as high-risk, with obligations applying from 2 December 2027.
Sources
- Published in Organization Science (new tab)doi.org
- A study published in the Quarterly Journal of Economics (new tab)doi.org
- A randomised trial by the organisation METR (new tab)arxiv.org
- Has since said these results are out of date (new tab)metr.org
- Its February 2026 update (new tab)metr.org
- A study presented at the CHI 2025 conference (new tab)doi.org
- Published in PNAS in June 2025 (new tab)doi.org
- Article 4 of the AI Act (new tab)eur-lex.europa.eu
- SB 7 (new tab)leginfo.legislature.ca.gov
- Regulation (EU) 2026/1744 (new tab)eur-lex.europa.eu
- SB 947 (new tab)leginfo.legislature.ca.gov
- “No Robo Bosses Act” (new tab)sd05.senate.ca.gov
- Article 22 of the GDPR (new tab)eur-lex.europa.eu
Jim · founder of Brain Plus AI
Also in business ai
Stepping back11 min
AI return on investment: why it is taking so long
The figure of 95% of AI projects with no return went round the world. What it really measures, why the other studies say roughly the same thing, and the practical lessons to draw from them for an SME.
Stepping back8 min
Does AI make us dumber? What the studies say
Not all the studies doing the rounds on this subject are worth the same. We rank four of them by strength, from a field experiment to a preprint on 54 people, and say what emerges.
Stepping back9 min
Is AI creative? What the experiments say
AI beats the average human on creativity tests, but not the best, and it makes outputs more alike. Why value now lies in the gap from the average.