Running a large open model on one server

A large open model can run on one server inside your own network. The prompts, the documents behind them and the answers stay on a machine you control. A datasheet cannot tell you whether that server is fast enough and accurate enough for your work. You find out by measuring it on your own tasks, before you buy hardware.

One server keeps the model and its data in one place

Most organisations that ask for private AI have the same concern. Contracts, patient notes, source code or financial records should not travel to a third party's model to be read. A model that runs on one server can sit behind your own firewall. It can expose the same kind of interface a hosted API offers, so an application on your network calls it in the same way and nothing else changes.

One server also puts a ceiling on cost. The hardware is bought or rented once, and running it costs power and upkeep rather than a fee per word. That trade only pays if the server keeps up with real use, which is why measurement comes first. Where data should live is a wider question than one machine, and our security page covers it.

Why a large model can fit on one card

A model's weights must sit in fast memory while it runs. For a dense model, every weight is read for every token it writes, so a large dense model needs a lot of memory and a lot of bandwidth.

Many recent open models are built differently. A mixture-of-experts model splits most of its layers into many small expert networks, and a router chooses a few of them for each token. Only a small share of the weights does work for any one token. The model therefore writes about as fast as a much smaller model, but all of its weights still have to be held somewhere fast. So the deciding question is whether the whole model fits in the card's memory.

It matters more than raw compute. If part of the model spills onto the processor and system memory, every token waits for the slowest part. A model that only just fails to fit can run several times slower than one that fits with room to spare.

Quantisation makes it fit, and has to be checked

Quantisation stores the model's weights in fewer bits. A weight held in 16 bits can often be held in 8 or 4, with a small shared scale for each group of values so the rounding stays close to the original. Fewer bits means a smaller model in memory, and fewer bytes to read for every token, so it is usually faster too.

Rounding changes the numbers, and a careless conversion can make the model worse at exactly the tasks you care about. Some formats also need calibration, which means running sample data through the model to learn the range of values each layer sees. Insist on these checks:

• compare the quantised layers against the full-precision ones and report the error, layer by layer;
• run the quantised model and the original on the same set of your own tasks and compare the answers;
• check that the format suits the card, because some cards compute certain low-bit formats natively and others have to unpack them first.

A common approach is to quantise the largest parts of the model hard and leave the smaller, more sensitive parts at higher precision. Whatever the mix, judge it by the answers on your own tasks. A headline score on a public benchmark does not tell you that.

The serving engine decides much of the speed

The serving engine is the software that loads the model, schedules requests and runs the maths on the card. The same model on the same card can run much faster or slower depending on the engine, its version and its settings.

Engines differ in a few ways that matter in practice. Some have kernels tuned for a particular card and a particular low-bit format, and others pad or convert the weights and lose part of the gain. Engines also differ in how they store the conversation so far, the KV cache, and storing it in fewer bits leaves room for longer documents or more users. Many support speculative decoding, where a small draft step guesses the next few tokens and the main model checks the guesses in one pass. That helps most on predictable text such as code and less on open conversation.

Getting a new model running on an engine can also take real work. Support for a new architecture often arrives in stages, and a large model on one card often runs out of memory until the settings are right. Budget time for that, and keep a record of every setting that made it work.

Measure speed and quality on your own work

Three numbers describe speed: how long the first token takes to arrive, how fast the answer is written after that, and how long a long document takes to read in before the answer starts. They behave differently. A chat assistant lives on the first two. A system that summarises long reports lives on the third.

Measure them the way the server will be used:

• use prompts drawn from your real work, of realistic length;
• give every prompt a unique marker, so no two requests are identical;
• run with one user and with the number of users you expect at a busy moment;
• record the time to first token for every user, not only the total output.

Quality needs its own test. Build a set of tasks from your own material with known good answers, run the candidate models on them, and have a person check a sample of the results. Speed only counts once the answers are right.

The common pitfalls

Many problems with a private model come from details that are easy to check and easy to miss.

Memory runs out in more places than one. The card has to hold the weights and the KV cache together. System memory matters as well, because loading and converting a large model can need far more of it than running does. Size both before you buy.

Warm-up distorts the first results. A large model can take minutes to load, and the first requests after a start are often slower while the engine compiles and caches its work. Measure after warm-up, and plan for what happens when the service restarts.

The prefix cache inflates benchmarks. Engines remember the start of recent prompts so they can skip repeated work. That is useful in production. In a benchmark it misleads. Send the same prompt twice and the engine appears far faster than it will be on real traffic. Unique prompts avoid it.

Concurrency has its own limits. More users raise total output until something runs out. Memory is one limit. An engine setting, such as the maximum number of requests in a batch, can also cap how many run at once, and the rest queue. The sign is time to first token rising sharply while total output stays flat.

Settings change behaviour as well as speed. Sampling settings, and whether the model thinks before it answers, can change how reliably it uses tool results. Test tool calls separately from chat.

Operations can undo the work. Automatic system updates can restart the engine or change a driver under it. Downloads can finish with a file missing. Check every downloaded file against its published checksum, and keep the driver and its libraries out of unattended upgrades.

Questions to ask before you buy

Ask anyone proposing a private model, us included, for the measurements behind the proposal. Which tasks were used, and whose material were they drawn from? Were the prompts unique, and was the server warm? How many users at once, and what happened to the first-token time as that number rose? How was the quantised model compared with the original? What is untested?

A good answer comes with the method, the raw results and a plain list of limits. Our private AI service starts the same way. We run candidate models on your material and record speed and quality before you buy anything.