Contents6
Running an AI model on your own servers guarantees that the data it processes goes to no model provider. It is also a hardware purchase or rental, and the first question we are asked is always the same: “what kind of machine do we need?”
This guide from our Infrastructure and cloud section answers without getting technical: the resource that decides everything, the orders of magnitude for hardware, when to rent rather than buy, and how to size by use. It takes no side between models: the open model families (Mistral, Llama, Qwen, Gemma, DeepSeek and others) are tested on your task, not on a reputation.
In short
The graphics card’s memory decides what a server can run, well before computing power.
Count about 2 GB per billion parameters at standard precision; compressing the model divides that need by two to four.
Long conversations and simultaneous users add their own memory needs.
A single card on the market ranges from 32 to 80 GB, and draws 350 to 700 W: electricity is part of the cost.
Renting an equipped server lets you size on real use before buying.
The resource that decides everything: memory
A language model is first of all a very long list of numbers, its parameters. To answer quickly, it has to fit entirely in the graphics card’s memory. The documentation of Hugging Face, the reference platform for open models, gives the rule of thumb: loading a model with X billion parameters takes about 2 × X GB of memory at the precision most models are trained in today.
Definition
Quantisation
Compressing a model by storing each parameter with less precision (8 or 4 bits instead of 16). It divides the memory needed, at the cost of a slight loss of quality that has to be checked on your own task.
Hugging Face measured it on a coding model with just over 15 billion parameters:
Memory measured to load a 15-billion-parameter model
The same document warns that quantisation often slows answers down a little and can change some of them: it is validated by a test on your own cases, like any choice of model. Our method to evaluate an AI model before production applies as is.
What gets forgotten: room for conversations
The model is not alone in memory. While it answers, it keeps a trace of all the text in progress, and that trace grows with the length of the text and with the number of conversations running at the same time. Again according to Hugging Face, for the same model and a text of about 16,000 tokens, that trace takes about 15 GB, roughly half the size of the model itself. An assistant that reads long documents for ten people at once therefore needs far more than the model’s weight.
Key takeaway
An AI server is sized on three numbers: the size of the model, the length of the texts processed and the number of simultaneous users. Spec sheets only give one of them.
Orders of magnitude for hardware
The market reads in three tiers, which the spec sheets of the manufacturer NVIDIA help to place. Other manufacturers exist, but these three cards serve as a benchmark:
of memory for a high-end consumer card, the GeForce RTX 5090, at 575 W
for a server card such as the L40S, at 350 W
for an H100 in server format, up to 700 W
Combining this with the memory rule: a 70-billion-parameter model needs about 140 GB at standard precision, according to Hugging Face’s calculation. It fits on none of these cards alone, but fits across several, or on a single one once heavily quantised. A model with 7 to 15 billion parameters, quantised, fits on a consumer card, with room to spare for conversations.
Without a graphics card, tools such as Ollama, which we use, can also load a model entirely into system memory and run it on the processor, more slowly. That is enough for short, infrequent tasks, not for an assistant used by a whole team.
Watch out
Electricity is part of the bill
A 700 W card running day and night consumes about 6,100 kWh a year on its own (700 W × 8,760 hours), not counting the rest of the server or the air conditioning of the room. In an office, it also means heat and noise. Our article on AI’s energy consumption puts these orders of magnitude in perspective.
Buy, rent, or stay on an API
Hardware is only one of the ways to keep control of your data. Our article on the four levels of private AI compares them, from a market API governed by contract to a server on your premises. For those who have chosen an open model on a dedicated machine, the question becomes: buy or rent.
Buying the server
Cost known in advance, spread over several years
Data never leaves your premises
Electricity, cooling, failures and replacement are on you
Hardware ages fast as new models arrive
Renting an equipped server
Up and running in a few days, with no investment
European host possible, in the company’s name
Change of card when needs evolve
Monthly cost that runs as long as the server does
Renting has a decisive advantage at the start: it lets you measure real use (number of requests, length of texts, simultaneous users) before committing to a purchase. Our comparison of dedicated server, VPS or cloud details the options, and the five items that inflate a cloud bill help read a rental offer.
Size by use, not by spec sheet
- Start from the task.
Sorting requests, summarising documents, a writing assistant: the expected quality is not the same.
- Test the smallest model that passes your real cases
Quantised or not. A bigger model is only useful if it succeeds on cases the small one misses.
- Calculate the memory
The model, plus room for the longest texts, multiplied by the number of simultaneous users.
- Keep a margin
Then rent the configuration while you measure, before buying anything.
From the field
We run Ollama in production on our own server, the one that also hosts our CRM and our automations, for tasks where data must not leave and for low-stakes sorting tasks. Each additional call costs almost nothing there; in return, quality is that of a model this server can run, and updates are our job. We never hard-code a model name: a routing table says which model handles which task, and is revised when a better model comes out.
What hardware does not solve
An AI server is still a server: it gets updated, monitored, backed up and protected like the others. Open models evolve quickly, and a configuration bought for one model can turn out too tight for its successor. Nor does hardware replace the people who define the tasks, review the results and decide what can be automated.
A local model is part of the infrastructure foundation shared by your projects, just like monitoring and access. The rest of the section covers neighbouring decisions: Kubernetes and mid-sized companies, monitoring, the disaster recovery plan, NIS2 in France, backing up your own server, self-hosted n8n in production, which tools to move out of SaaS first, the three-year cost of leaving SaaS, cloud migration without breaking things, dependence on the American cloud, the team password manager, stolen API keys, managed or self-hosted Postgres and choosing a business email provider.
To install a model alongside your other self-hosted tools, see our Leaving SaaS offer; to decide, task by task, what runs on your premises and what goes through an API, AI at the right cost.
Frequently asked questions
How much memory does an 8-billion-parameter model need?
Following Hugging Face’s rule, about 16 GB at standard precision, around 8 GB quantised to 8 bits and less again at 4 bits. Add room for the conversations in progress, which grows with the length of the texts and the number of simultaneous users. A card with 24 to 32 GB leaves a comfortable margin for moderate team use.
Can ChatGPT or Claude run on our own servers?
No. Those models are not distributed: they are only used through their provider, directly or through a major cloud. Your servers run open models, whose weights are published. Their quality varies by task: a test on your own cases tells you whether they are enough.
Is a local model worse than a commercial one?
On some tasks, yes, because a model that fits on a company server is smaller than the largest commercial models. On targeted tasks (sorting, extraction, summarising internal documents), a well-chosen open model can be enough. The right reflex is to compare both on a sample of your real cases before deciding.
Do you need a graphics card to run a model?
Not necessarily for a small model and occasional use: it can run on the processor, more slowly. As soon as a team uses it daily, or the texts are long, a graphics card with enough memory becomes necessary for acceptable response times.
Sources
- Hugging Face, Optimizing LLMs for Speed and Memory (new tab)huggingface.co
- NVIDIA, GeForce RTX 5090, specifications (new tab)nvidia.com
- NVIDIA, L40S, specifications (new tab)nvidia.com
- NVIDIA, H100, specifications (new tab)nvidia.com
- Ollama, FAQ (new tab)docs.ollama.com
Jim · founder of Brain Plus AI
Also in infrastructure and cloud
Choose and price9 min
Private AI in business: the four levels of control
A "private" AI is not a product, it is a degree of control over your data. We distinguish four levels, from an API used as is to a model installed on your premises, and a simple way to choose task by task.
Choose and price8 min
Dedicated server, VPS or cloud: choosing by use
Dedicated server, VPS or cloud: none of them is the right choice for everything. What really sets them apart, the four questions that decide for each use, and why most companies end up combining all three.
Understand9 min
Cloud costs: the five items that inflate the bill
A cloud bill almost never jumps all at once: it grows through items nobody looks at. The five that weigh the most, what makes them invisible, and the routine that keeps them under control without slowing projects down.