\nQ4_K_M<\/td>\n 4.89<\/td>\n 18.7 GB<\/td>\n 20.1 GB<\/td>\n 44.5 GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\nThe mixed "K" formats vary a little from model to model, so treat these as close estimates. For a mixture-of-experts model, count all<\/strong> parameters: every expert sits in memory even though only a few run for each token.<\/p>\n\n\n\nKV cache<\/strong><\/p>\n\n\n\nKV cache (bytes) = 2 \u00d7 layers \u00d7 KV heads \u00d7 head dimension \u00d7 bytes per element \u00d7 context \u00d7 slots\n<\/code><\/pre>\n\n\n\nThe 2 is one key and one value. Bytes per element is 2 for an f16 cache, 1.0625 for Ollama’s 8-bit q8_0<\/code> cache and 0.5625 for q4_0<\/code> (llama.cpp stores each block of 32 values with a 2-byte scale, so 34 and 18 bytes per 32). Ollama’s documentation says q8_0<\/code> uses approximately half the memory of f16<\/code> and q4_0<\/code> approximately a quarter, and that the cache is only quantised when flash attention is enabled.<\/p>\n\n\n\nModels with sliding-window attention (Gemma 3 and gpt-oss, for example) hold only the last window of tokens on their local layers, so for those layers use the window size instead of the full context. That is why their cache grows much more slowly than the parameter count suggests.<\/p>\n\n\n\n
You’ll find the inputs in the model’s config.json<\/code> on its official model page:<\/p>\n\n\n\n\n\nnum_hidden_layers<\/code>: layers<\/li>\n\n\nnum_key_value_heads<\/code>: KV heads (models using grouped-query attention have far fewer KV heads than attention heads, which is what keeps the cache manageable)<\/li>\n\n\nhead_dim<\/code>, or if it’s missing, hidden_size \u00f7 num_attention_heads<\/code><\/li>\n\n<\/ul>\n\n\n\nOverhead<\/strong><\/p>\n\n\n\nThere is no clean formula. Leave a margin on top of the KV figure, then measure. In our lab the textbook KV formula was about 19% optimistic on one model, which is the direction that freezes a server. The calculator below adds 0.5 GB plus 10% of the KV cache for buffers and margin, which put it 3 to 4% above every figure we measured on our main model.<\/p>\n\n\n\n
Context length and parallel slots: where budgets go wrong<\/h2>\n\n\n\n Ollama’s FAQ is explicit: required memory scales with OLLAMA_NUM_PARALLEL<\/code> \u00d7 OLLAMA_CONTEXT_LENGTH<\/code>. Each parallel slot gets the full context and reserves its KV cache whether or not anyone is using it.<\/p>\n\n\n\nThat is how our lab server ran out. The main model had a 64k context written into its own settings, which overrode the server-wide limit, and the server allowed four parallel slots. One client was using one slot, but Ollama had reserved KV cache for all four: 262,144 tokens’ worth. The model’s footprint grew to about 33 GB, a second model sat beside it, and a long prompt tipped the machine over.<\/p>\n\n\n\n
The fix was one slot per model and a 32,768-token context. A week of measurement afterwards (1,342 requests, 52 overlaps, never more than four at once) showed the extra slots had rarely been used. The cost is a queue: a second caller now waits a median of 18 seconds.<\/p>\n\n\n\n
Check:<\/strong> if you raise OLLAMA_NUM_PARALLEL<\/code> from 1 to 4, multiply your KV figure by four before you change anything.<\/p>\n\n\n\nWorked examples from official model configs<\/h2>\n\n\n\n All three use the architecture published in each model’s config.json<\/code> and model card on Hugging Face. Figures are in decimal GB (10\u2079 bytes), the unit ollama ps<\/code> reports.<\/p>\n\n\n\n1. Qwen3-Coder-30B-A3B (the model behind qwen3-coder<\/code> in Ollama)<\/strong><\/p>\n\n\n\n\n\n30.5B parameters in total, 3.3B active per token; 48 layers; 4 KV heads; head dimension 128; native context 262,144 tokens.<\/li>\n\n\n KV per token: 2 \u00d7 48 \u00d7 4 \u00d7 128 \u00d7 2 bytes = 98,304 bytes (f16), or about 52,000 bytes with an 8-bit cache.<\/li>\n\n\n Weights at Q4_K_M: about 18.7 GB.<\/li>\n\n<\/ul>\n\n\n\n\n\n\nSetting<\/th>\n KV cache (8-bit)<\/th>\n Weights + KV<\/th>\n We measured<\/th>\n<\/tr>\n<\/thead>\n \n\n32k context \u00d7 1 slot<\/td>\n 1.7 GB<\/td>\n 20.4 GB<\/td>\n 20.3 GB<\/strong><\/td>\n<\/tr>\n\n64k context \u00d7 4 slots<\/td>\n 13.7 GB<\/td>\n 32.4 GB<\/td>\n about 33 GB<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\nThis model has only 4 KV heads, so its cache is small for its size. That is why the arithmetic and our measurements line up so closely here.<\/p>\n\n\n\n
2. Qwen3-32B (dense)<\/strong><\/p>\n\n\n\n\n\n32.8B parameters; 64 layers; 8 KV heads; head dimension 128; native context 32,768 tokens.<\/li>\n\n\n KV per token: 2 \u00d7 64 \u00d7 8 \u00d7 128 \u00d7 2 = 262,144 bytes (f16), almost three times the coder model.<\/li>\n\n\n Weights at Q4_K_M: about 20.1 GB.<\/li>\n\n<\/ul>\n\n\n\n\n\n\nSetting<\/th>\n KV cache (f16)<\/th>\n Weights + KV, before overhead<\/th>\n<\/tr>\n<\/thead>\n \n\n32k \u00d7 1 slot<\/td>\n 8.6 GB<\/td>\n 28.7 GB<\/td>\n<\/tr>\n \n32k \u00d7 4 slots<\/td>\n 34.4 GB<\/td>\n 54.4 GB<\/td>\n<\/tr>\n \n32k \u00d7 4 slots, 8-bit cache<\/td>\n 18.3 GB<\/td>\n 38.3 GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\nSimilar weights to the coder model, very different cache. Four slots at f16 cost more memory than the weights.<\/p>\n\n\n\n
3. Qwen2.5-72B-Instruct (dense)<\/strong><\/p>\n\n\n\n\n\n72.7B parameters; 80 layers; 8 KV heads; head dimension 128 (8,192 \u00f7 64); context up to 131,072 tokens.<\/li>\n\n\n KV per token: 2 \u00d7 80 \u00d7 8 \u00d7 128 \u00d7 2 = 327,680 bytes (f16).<\/li>\n\n\n Weights: 44.5 GB at Q4_K_M, 77.3 GB at Q8_0, 145.4 GB at F16.<\/li>\n\n<\/ul>\n\n\n\n\n\n\nSetting<\/th>\n Weights + KV, before overhead<\/th>\n On ~120 GB usable, keeping 40 GB free<\/th>\n<\/tr>\n<\/thead>\n \n\nQ4_K_M, 32k \u00d7 1 slot, f16 cache<\/td>\n 55.2 GB<\/td>\n Fits, with room for a small second model<\/td>\n<\/tr>\n \nQ4_K_M, 128k \u00d7 1 slot, f16 cache<\/td>\n 87.4 GB<\/td>\n Breaks the 40 GB floor<\/td>\n<\/tr>\n \nQ8_0, 32k \u00d7 1 slot, f16 cache<\/td>\n 88.0 GB<\/td>\n Breaks the floor<\/td>\n<\/tr>\n \nF16, any context<\/td>\n over 145 GB<\/td>\n Won’t load<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\nRun your own numbers in the calculator below. It uses the same formulas (including sliding windows), with presets taken from official configs, and adds the buffer margin described above, calibrated against the lab figures. It shows Fits<\/strong> up to 85% of the memory you allow, Tight<\/strong> from 85 to 100%, and Will not fit<\/strong> above that. Estimates are approximate; measure before you commit.<\/p>\n\n\n\n\n