Big models.
Small machines.
Can a private AI server with a few GB of RAM run models far larger than its memory? We measured it under hard cgroup memory caps, cold page cache, on a 2-vCPU AMD EPYC server, and built our own engine to push further.
One equation decides everything.
When a model doesn't fit in RAM, every token re-reads weights from disk. There are only three levers: touch fewer bytes (quantization, mixture-of-experts), read faster (sequential streaming instead of page faults), or verify more tokens per read (speculation). Squeeze pulls all three.
Mixture-of-experts beats dense when RAM is scarce.
Generation speed on the same 2-vCPU server (llama.cpp b12202, mmap, repack off, cold cache).
A 35-billion-parameter model ran inside a 1 GB memory cap, and with 4× the parameters it still generated 2–6× faster than the dense 9B, because only ~3B parameters are active per token.
Squeeze: our Rust streaming engine.
Quantized weights streamed straight from GGUF, page-cache-bypassing reads, double-buffered prefetch, pin-what-fits, and prompt-lookup speculation. Output matches llama.cpp token for token.
Faster than llama.cpp mmap on Qwen3-8B (8.7 GB) inside the same 1 GB cap, for document Q&A: 0.31 vs 0.09 tok/s at 707 MB peak RSS. Plain chat: 1.8×.
Peak RAM running a 604 MB model (Qwen3-0.6B) on an M1 laptop, with output identical to full-RAM inference.
Speed-up on document Q&A from prompt-lookup speculation: 2.35 → 10.25 tok/s at 76 MB RAM, same answer.
When the model fits, small is fast.
Fully in RAM on 1.8 vCPU. These numbers set the size of model we deploy as our private AI.
| Model (Q4_K_M) | File | Prompt tok/s | Generate tok/s |
|---|---|---|---|
| Qwen3.5 0.8B | 0.53 GB | 150 | 27.1 |
| Qwen3.5 2B | 1.28 GB | 68 | 17.2 |
| Qwen3.5 4B | 2.74 GB | 21 | 7.5 |
Honest conclusions
Running far beyond RAM works, but it's slow. Sub-1-tok/s is right for overnight batch jobs (document extraction, migrations, report drafting), not live chat.
For live chat, a small model that fits wins. For big-model quality, MoE plus a modest RAM budget is the sweet spot.
Method: every run used a fresh systemd scope with a hard MemoryMax (page cache included), no swap and a CPU quota, after evicting the model file from the page cache. The harness and engine source live in our lab repo.