How much memory does a local LLM need? A sizing guide

A local model needs memory for its weights, for a KV cache that grows with context and parallel slots, and for runtime overhead. Here are the formulas, three worked examples from official model configs, the numbers we measured on our lab server, and the commands to check what your models really use.

What uses the memory?

Three things, in this order of size for most setups:

1. Weights. The model’s parameters, loaded once per model. A 30-billion-parameter model at 4-bit is about 19 GB whether one person uses it or ten.
2. KV cache. The keys and values the model keeps for every token in its context, so it doesn’t recompute them for each new token. It grows linearly with context length and with the number of requests the server will handle at once.
3. Overhead. The runtime, compute buffers and temporary memory for processing a long prompt. It is smaller, but it spikes during prompt processing, which is exactly when a tight budget fails.

On a discrete GPU, running out usually means the model spills to the CPU and slows to a crawl. On a unified-memory machine like the NVIDIA DGX Spark, where CPU and GPU share one pool, it can take the whole system down: our lab server froze hard, with no out-of-memory error, while a 65k-token prompt was being processed (the full post-mortem).

The formulas

Weights

weights (GB) ≈ parameters (billions) × bits per weight ÷ 8

Use the real bits per weight of the quantisation format, not the name. The llama.cpp project publishes these for Llama 3.1 8B:

The mixed "K" formats vary a little from model to model, so treat these as close estimates. For a mixture-of-experts model, count all parameters: every expert sits in memory even though only a few run for each token.

KV cache

KV cache (bytes) = 2 × layers × KV heads × head dimension × bytes per element × context × slots

The 2 is one key and one value. Bytes per element is 2 for an f16 cache, 1.0625 for Ollama’s 8-bit q8_0 cache and 0.5625 for q4_0 (llama.cpp stores each block of 32 values with a 2-byte scale, so 34 and 18 bytes per 32). Ollama’s documentation says q8_0 uses approximately half the memory of f16 and q4_0 approximately a quarter, and that the cache is only quantised when flash attention is enabled.

Models with sliding-window attention (Gemma 3 and gpt-oss, for example) hold only the last window of tokens on their local layers, so for those layers use the window size instead of the full context. That is why their cache grows much more slowly than the parameter count suggests.

You’ll find the inputs in the model’s config.json on its official model page:

• num_hidden_layers: layers
• num_key_value_heads: KV heads (models using grouped-query attention have far fewer KV heads than attention heads, which is what keeps the cache manageable)
• head_dim, or if it’s missing, hidden_size ÷ num_attention_heads

Overhead

There is no clean formula. Leave a margin on top of the KV figure, then measure. In our lab the textbook KV formula was about 19% optimistic on one model, which is the direction that freezes a server. The calculator below adds 0.5 GB plus 10% of the KV cache for buffers and margin, which put it 3 to 4% above every figure we measured on our main model.

Context length and parallel slots: where budgets go wrong

Ollama’s FAQ is explicit: required memory scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. Each parallel slot gets the full context and reserves its KV cache whether or not anyone is using it.

That is how our lab server ran out. The main model had a 64k context written into its own settings, which overrode the server-wide limit, and the server allowed four parallel slots. One client was using one slot, but Ollama had reserved KV cache for all four: 262,144 tokens’ worth. The model’s footprint grew to about 33 GB, a second model sat beside it, and a long prompt tipped the machine over.

The fix was one slot per model and a 32,768-token context. A week of measurement afterwards (1,342 requests, 52 overlaps, never more than four at once) showed the extra slots had rarely been used. The cost is a queue: a second caller now waits a median of 18 seconds.

Check: if you raise OLLAMA_NUM_PARALLEL from 1 to 4, multiply your KV figure by four before you change anything.

Worked examples from official model configs

All three use the architecture published in each model’s config.json and model card on Hugging Face. Figures are in decimal GB (10⁹ bytes), the unit ollama ps reports.

1. Qwen3-Coder-30B-A3B (the model behind qwen3-coder in Ollama)

• 30.5B parameters in total, 3.3B active per token; 48 layers; 4 KV heads; head dimension 128; native context 262,144 tokens.
• KV per token: 2 × 48 × 4 × 128 × 2 bytes = 98,304 bytes (f16), or about 52,000 bytes with an 8-bit cache.
• Weights at Q4_K_M: about 18.7 GB.

This model has only 4 KV heads, so its cache is small for its size. That is why the arithmetic and our measurements line up so closely here.

2. Qwen3-32B (dense)

• 32.8B parameters; 64 layers; 8 KV heads; head dimension 128; native context 32,768 tokens.
• KV per token: 2 × 64 × 8 × 128 × 2 = 262,144 bytes (f16), almost three times the coder model.
• Weights at Q4_K_M: about 20.1 GB.

Similar weights to the coder model, very different cache. Four slots at f16 cost more memory than the weights.

3. Qwen2.5-72B-Instruct (dense)

• 72.7B parameters; 80 layers; 8 KV heads; head dimension 128 (8,192 ÷ 64); context up to 131,072 tokens.
• KV per token: 2 × 80 × 8 × 128 × 2 = 327,680 bytes (f16).
• Weights: 44.5 GB at Q4_K_M, 77.3 GB at Q8_0, 145.4 GB at F16.

Run your own numbers in the calculator below. It uses the same formulas (including sliding windows), with presets taken from official configs, and adds the buffer margin described above, calibrated against the lab figures. It shows Fits up to 85% of the memory you allow, Tight from 85 to 100%, and Will not fit above that. Estimates are approximate; measure before you commit.

What does quantisation cost you?

Two things change when you drop from 16-bit to 4-bit weights.

Speed goes up. Generating each token means reading the active weights from memory, so on a bandwidth-limited machine fewer bytes means more tokens a second. The DGX Spark's memory bandwidth is 273 GB/s. In llama.cpp's own table for Llama 3.1 8B, text generation ran at about 29 tokens a second at F16 and about 72 at Q4_K_M on the same (unnamed) hardware. In our lab, qwen3-coder at Q4_K_M generated about 60 to 75 tokens a second on the DGX Spark's GPU (about 15 on the CPU, before the GPU worked).

Quality goes down as the bits go down. How much depends on the model and the task, and llama.cpp notes that quantisation "may introduce some accuracy loss". We ran every workhorse model at Q4_K_M, and for routing, extraction, summaries and tool calls the quality was good enough. Before you go below 4 bits, or trust 4-bit with work that needs careful reasoning, run your own tasks through both versions and compare.

The KV cache has its own quantisation switch (OLLAMA_KV_CACHE_TYPE). It is set per Ollama server, not per model. An 8-bit cache roughly halves the cache with what Ollama calls "a very small loss in precision", and we ran one model we didn't trust in a separate instance with an unquantised f16 cache, one slot and a 16k context, to keep its behaviour conservative.

Our lab's measured numbers

That last row is the reason to read the config rather than guess from parameter count: two models of similar size differed fourfold in what extra context cost them. Note too that the 40 GB floor is a policy, not a proven threshold: three of our five freezes happened with more than 80 GB available, and one model that froze on load was eventually sandboxed rather than fixed.

Measure, don't derive

Load the model at the exact context and slot count you plan to run, then read what it took.

1. Record the baseline.

   free -h
   

   Check: note "used" and "available" before any model is loaded.

2. Load the model with a short request at your target context.

   curl -s http://localhost:11434/api/generate \
     -d '{"model":"qwen3-coder","prompt":"hi","stream":false,"options":{"num_ctx":32768}}' > /dev/null
   

   A request with a different num_ctx reloads the model, so use the value your applications will send, or set it once in a Modelfile (PARAMETER num_ctx 32768) or with OLLAMA_CONTEXT_LENGTH.

3. Read what Ollama reports.

   ollama ps
   

   Check: PROCESSOR says 100% GPU (anything showing CPU means it didn't fit or didn't reach the GPU), CONTEXT shows the context you asked for, and SIZE is the figure for your budget.

4. Get exact bytes from the API.

   curl -s http://localhost:11434/api/ps | jq '.models[] | {name, size_gb: (.size/1e9), vram_gb: (.size_vram/1e9), context_length, quant: .details.quantization_level}'
   

   This needs jq; without it, pipe to python3 -m json.tool and read the fields by eye. Check: size_vram close to size means the model is fully on the GPU. Older Ollama versions omit context_length; read it from ollama ps instead.

5. Cross-check with the GPU driver.

   nvidia-smi --query-gpu=name,memory.used,memory.total --format=csv
   nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
   

   Check: on a discrete GPU, memory.used rises by roughly the model's size. On the DGX Spark and other integrated-GPU systems, NVIDIA documents that nvidia-smi shows "Memory-Usage: Not Supported" while still listing per-process memory, so use the second command plus free -h.

6. Measure the KV cost directly. Repeat steps 2 to 4 at double the context (and again with your planned slot count). Check: the difference in size_vram is your real cost per extra context or slot. If it disagrees with the formula, trust the measurement and keep the margin.

7. Watch it under a long prompt. Send the largest prompt you'll admit and run free -h during processing. Check: available memory never drops into the floor you've set.

A budget template you can copy

Fill the Measured column from step 4, not from the formula. Then make the server enforce it: cap OLLAMA_NUM_PARALLEL, OLLAMA_CONTEXT_LENGTH and OLLAMA_MAX_LOADED_MODELS, and reject prompts larger than the context before they reach the model (how we did that).

If you're weighing a private model against a cloud service in the first place, our decision guide to private AI, ChatGPT and Copilot covers data handling and residency.

Key takeaways

• Budget weights, KV cache and overhead separately; the KV cache scales with context × slots.
• In Ollama every parallel slot reserves the full context, so four slots cost four times the cache even when one person is using the server.
• Read layers, KV heads and head dimension from the model's config.json: similar-sized models can differ fourfold in cache cost.
• 4-bit weights need a little over half the memory of 8-bit and run faster on bandwidth-limited hardware; test quality on your own tasks.
• Measure with ollama ps, /api/ps, nvidia-smi and free -h, then enforce the budget with server limits and a floor of free memory.

Sources

• Ollama FAQ (parallel requests, context length, K/V cache quantisation): https://docs.ollama.com/faq
• Ollama API, list running models (/api/ps): https://docs.ollama.com/api/ps
• llama.cpp, quantize README (bits per weight and speed by format): https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
• Qwen3-Coder-30B-A3B-Instruct model card and config: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct
• Qwen3-32B model card and config: https://huggingface.co/Qwen/Qwen3-32B
• Qwen2.5-72B-Instruct model card and config: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct
• NVIDIA DGX Spark specifications: https://www.nvidia.com/en-us/products/workstations/dgx-spark/
• NVIDIA DGX Spark known issues (nvidia-smi on integrated GPUs): https://docs.nvidia.com/dgx/dgx-spark/known-issues.html
• Catharsis lab series, Part 3: why a local AI server freezes: /blog/local-ai-server-freezes-unified-memory/