Contents8
- 01How does RAG work?
- 02Do you really need a RAG?
- 03RAG or fine-tuning: which to choose for your documents?
- 04Why does a RAG sometimes miss the point?
- 05What should you set up before opening the assistant to the team?
- 06Where do your documents go when the assistant answers?
- 07Where to start
- 08Frequently asked questions
Your procedures, contracts and files hold answers that nobody can find, and the same questions end up with the same colleagues. Enterprise RAG promises to make that knowledge available with a single question: an assistant that reads your documents and cites its sources. The principle fits in a few lines; putting it into production takes more care. This article in our Business AI: private and at the right cost section explains how RAG works, when it is useful, why it sometimes misses the point, and what to set up before opening it to the team. Connecting AI to your documents is only one step: our guide to integrating AI in your business places it among the decisions that come before it, from the target task to the choice of model.
In short
Below about 500 pages (a rough size given by Anthropic in 2024), your documents can fit whole in the model’s prompt: worth trying before you build a RAG.
Fine-tuning changes the model itself and requires significant computing resources; RAG leaves the model intact and makes its answers more traceable.
When the search brings back the wrong passage, the model answers off target: a passage cut off from its context or a missed exact reference is enough.
Before opening it up, four settings: access rights, index updates, the source displayed, and tested answers.
How does RAG work?
RAG is short for Retrieval-Augmented Generation. The CNIL (France’s data protection authority), in its questions and answers on generative AI, describes it as integrating an “information retrieval mechanism into a vectorised database”, which gives answers “potentially more specific and easier to update than the model itself”. Anthropic sets out the steps in an article from September 2024:
From document to answer
- Splitting
Each document becomes a series of short passages, generally no more than a few hundred tokens.
- Embedding
A model turns each passage into a vector, a numerical fingerprint of its meaning.
- The vector database
It stores these vectors and finds passages that are close in meaning.
- Retrieval
For each question, the most relevant passages are retrieved and added to the model’s prompt, which writes the answer.
The model learns nothing from your documents: for each question, it reads the excerpts it is given. That is what makes it possible to display the source of an answer, and it is also the system’s limit: if the search brings back the wrong passage, the model answers from the wrong passage. The CNIL adds that the method “requires more resources and skills than using simple off-the-shelf systems or models”, but that it “offers better adaptability and traceability of the information contained in the answers”.
Do you really need a RAG?
Not always. In the same article, Anthropic says that a knowledge base of fewer than 200,000 tokens, or about 500 pages, can be placed whole in the model’s prompt, without RAG. “Sometimes the simplest solution is the best,” the company writes, adding that prompt caching then more than halved the response time and cut the cost by up to 90%. These figures date from September 2024 and from the models of the time: they give an order of magnitude, to be rechecked on the models you use.
This simple version has a price: each question sends the whole knowledge base back to the model, and the bill is counted in tokens. For knowledge bases that do not fit in the model’s context window, RAG is the usual solution, Anthropic also notes. So the right order is to try the simple version on your documents, then build a RAG if volume, cost or response time demand it.
RAG or fine-tuning: which to choose for your documents?
Fine-tuning, according to the CNIL, means “changing the parameters (weights) of a model while maintaining its core architecture”. It improves its performance in a given domain or on specific queries, but “demands significant resources, particularly in terms of computing capacity”. The CNIL presents the two methods as options for improving a system, one and/or the other.
| RAG | Fine-tuning | |
|---|---|---|
| What changes | The excerpts the model reads for each question | The model’s parameters |
| Resources | More than a model used as is | Significant, especially computing |
| Your responsibility | Data controller if the knowledge base contains personal data | Responsible for the fine-tuning, including as a provider under the EU AI Act |
The last row comes from the CNIL and matters for an SME: fine-tuning a model with your own data can make you a provider within the meaning of the EU AI Act, as far as that fine-tuning is concerned. To make an AI answer from documents that change, with a verifiable source, RAG is therefore the starting point; fine-tuning aims at something else, better performance in a domain or on specific queries.
Why does a RAG sometimes miss the point?
Anthropic sums up the problem: traditional RAG systems remove context from passages when they encode them, which often stops them finding the right information. Three causes stand out.
A passage cut off from its context. In Anthropic’s example, an excerpt from a financial report says “The company’s revenue grew by 3% over the previous quarter”, without naming either the company or the quarter. On its own, it becomes hard to retrieve and hard to use. In your documents, it might be a contract clause that says “the time limit is extended to thirty days” without saying which contract it belongs to.
An exact reference that search by meaning does not see. Embeddings capture meaning, but they can miss an exact match: for the query “Error code TS-999”, search by meaning may bring back passages about error codes in general without finding that one. A keyword search (the BM25 method) finds it, and the two can be combined. Contract numbers and product references are in the same situation.
Poorly tuned splitting. The size of the passages, where they are cut and how much they overlap all affect the search. So does the number of passages passed on: Anthropic tested 5, 10 and 20, got the best results with 20 among those options, and advises trying it on your own case; too much information can also distract the model.
To reduce these failures, Anthropic adds to each passage, before indexing, a few sentences of context written by a model (50 to 100 tokens). On its own test sets (code, fiction, scientific papers), each added technique reduces failures:
Retrieval failure rate among the top twenty passages
These are the vendor’s measurements on its own data, in 2024: an order of magnitude, not a promise for your documents. The same article reminds you that you must always measure: our method for evaluating an AI model before production applies to a RAG too.
What should you set up before opening the assistant to the team?
A RAG in production also raises organisational questions, which the model does not settle.
- Access rights. For each person, the assistant must retrieve only the documents they are allowed to read. A single index that mixes commercial contracts and personnel files answers everyone with everything: an intern’s question can bring up an excerpt from an HR file. Rights are filtered in the search, before the model sees anything at all. Organising them department by department, while keeping knowledge bases common to the whole company, is the subject of our guide to AI governance in business.
- Index updates. A document that is changed or withdrawn must be changed or withdrawn in the vector database too, otherwise the assistant confidently cites an outdated procedure. Decide who updates it, and how often.
- The source displayed and the “I don’t know”. The CNIL recommends favouring systems “limiting hallucinations, citing their sources, filtering outputs”. Every answer must show the passage it relies on, and the assistant must say it does not know when no passage answers.
- Testing and follow-up. A set of real questions with their expected answers is run before opening, then again at every change of model, prompt or documents. After opening, the CNIL recommends that you “regularly gather the opinion of end-users on the usefulness and limitations” of the system; questions left unanswered can point to a missing document.
Where do your documents go when the assistant answers?
For each question, the retrieved excerpts go wherever the model runs, and the index holds your documents split into passages. The level of control is therefore chosen according to the documents queried: our comparison of the four levels of control of a private AI sets out the options, from the API to a server installed on your premises. For personal data or sensitive documentation, the CNIL writes that “it generally seems more appropriate and secure to favour the deployment of on-premise solutions”, while noting that the cost often makes remote hosting easier, covered by a data processing contract.
From the field
With us, Ollama serves a self-hosted model, in production on our own server, for tasks whose data must not leave it.
Where to start
Four steps to get started:
- A scope
Start from a single, well-defined set of documents, for example a team’s procedures, and from questions that really do come up.
- The simple version
First check whether everything fits in a model’s prompt.
- The RAG
If not, build it on a first batch, and test its answers.
- Opening up
Open it to a small group, before the whole team.
If you want an assistant that finds the answer in your procedures and contracts, cites its source and says when it does not know, our AI assistant on your documents offer builds it in three weeks: access set up by team, answers tested before opening, and a self-hosted option included, so that everything runs on your server or ours, in Europe.
Once the assistant is in service, its bill is managed like everything else: our AI at the right cost offer hands each task to the model that is good enough, with a budget and caps per use.
Frequently asked questions
RAG or fine-tuning: which to choose to make use of company documents?
To make an AI answer from documents that change, with a verifiable source, start with RAG: the CNIL describes it as easier to update than the model itself and more traceable. Fine-tuning changes the model’s parameters to improve it in a given domain or on specific queries, and requires significant computing capacity. The two methods can be combined.
Can a RAG make up an answer?
Yes. The model writes its answer from the retrieved excerpts: if the search brings back a wrong passage, or none, it can produce a plausible but wrong answer. The CNIL points out that a generative model is not a knowledge base. The countermeasures: display the source of every answer, tell it to answer “I don’t know” when there is no relevant passage, test it on real questions, and have a human check before any decision.
Do you need a vector database, and which one?
Not necessarily: if your documents fit in the model’s prompt, about 500 pages according to Anthropic in 2024, you can do without RAG, and so without a vector database. Beyond that, it stores the passages and finds those that are close in meaning; combine it with a keyword search for exact references. To choose one, first look at where it runs: it holds your documents split into passages.
Can you run a RAG without sending your documents to an AI provider?
Yes: the model, the vector database and the embedding computation can all run on a server you control, on your premises or at a host bound by contract. For sensitive documents, the CNIL writes that it “generally seems more appropriate and secure” to favour an on-premise deployment, noting that its cost often makes remote hosting easier. We run a self-hosted model ourselves, in production, for such cases.
Sources
- Its questions and answers on generative AI (new tab)cnil.fr
- Anthropic sets out the steps (new tab)anthropic.com
Jim · founder of Brain Plus AI
Also in business ai
Choose and price9 min
Private AI in business: the four levels of control
A "private" AI is not a product, it is a degree of control over your data. We distinguish four levels, from an API used as is to a model installed on your premises, and a simple way to choose task by task.
Practical guide9 min
Evaluating an AI model before putting it into production
Public leaderboards do not tell you whether a model will do your work well. This guide gives a method for testing it on your own tasks: quantified criteria, real examples, scoring, cost, and a new test whenever the model changes.
Practical guide10 min
The cost of AI in business: tokens, routing and forecasting
What you really pay when you use an AI through an API: input and output tokens, the context sent back with every call, an agent's turns. And a method to forecast the bill before you receive it.