On-Premise AI Server: GPU, RAM and Storage You Need
An AI server's requirements are set almost entirely by model size and concurrent users. Here is how to calculate before buying hardware.
Running a language model on your own server means you provide the graphics memory (VRAM), RAM and storage yourself. The most common mistake is buying hardware before knowing the model and workload. The right order: define the use case, choose the model, calculate memory, then pick hardware.
How much VRAM does a model need?
Rule of thumb: parameter count times bytes per parameter. A 16-bit model uses 2 bytes per parameter, while a 4-bit quantised version uses about half a byte. Add room for conversation context (KV cache) and overhead, usually 10 to 30 percent more.
- 7 billion parameters: about 14 GB at 16-bit, about 4 to 5 GB at 4-bit.
- 13 billion: about 26 GB at 16-bit, about 7 to 8 GB at 4-bit.
- 70 billion: about 140 GB at 16-bit, about 35 to 40 GB at 4-bit.
These are estimates; real needs differ by model and by the software running it, so test before ordering.
How do concurrent users affect this?
One user and fifty concurrent users need very different hardware. Each active request uses extra context memory and compute time. Ask: how many active users at peak, how long are typical documents, and how many seconds of response time is acceptable. For a pilot with 5 to 10 users, one mid-range server is often enough.
RAM, CPU and storage
- System RAM: at least equal to total VRAM so the model loads smoothly, ideally double.
- CPU: no need for a fancy one, but enough PCIe lanes for GPUs and NVMe.
- Storage: NVMe SSD for model files (tens of GB each) and a vector database if you use document search. Add room for source documents and backups.
- Power and cooling: GPU servers draw a lot of power and make heat. Check room power and cooling before the hardware arrives.
One big GPU or several small ones?
A model that fits on one GPU is simpler and faster. Splitting one model across several GPUs allows larger models but adds complexity and needs a fast enough interconnect. For many company cases, a mid-size quantised model on one GPU is adequate.
When is renting better than buying?
If the workload is still being tested or use is occasional, renting a GPU server or using a local data centre can be cheaper while you learn. Buying makes sense once a pilot shows stable load and the data truly must stay in your building.
Sentra helps choose models and hardware from pilot to rollout. See the Private LLM service, IT procurement, and data considerations in ChatGPT for business.
Frequently asked questions
What is the minimum GPU for an office AI assistant?
It depends on the model. A small quantised model can run on one GPU with a dozen or so GB of VRAM for a small pilot. Many users or a large model need much more. Calculate from model and load, not from generic numbers.
Is it possible without a GPU?
For small models, yes, with slower answers. For shared use by many people, a GPU is almost always needed.
How long should a pilot run?
A few weeks with prepared test questions is enough to judge accuracy and load before buying hardware.
Does data really never leave the office?
That is the aim of on-premise, but also make sure network access, backups and software updates do not open unwanted outbound paths.
Tell us about the process. We'll help map out the most proportionate approach.
