Why Does a Local AI Server Freeze? Sizing Memory for LLMs (Part 3 of 11)

In its first month of testing, our lab AI server froze five times with no kernel error and no warning. This is what caused it, the memory budget we settled on, and the steps to size one for your own server.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The problem: why did the server keep freezing?

The lab server is an NVIDIA DGX Spark, a compact desktop AI system built on the GB10 Grace Blackwell Superchip with 128 GB of unified memory, one pool shared by the CPU and GPU, roughly 120 GB of it usable. That lets a desk-sized machine hold a 30-billion-parameter model, and makes everything run short at once when the models take too much. Over seven weeks it carried test workflows, a Telegram agent, the LLM gateway our test clients used (Part 2) and a multi-agent stock-analysis service on public market data.

In its first month the server froze five times. Each time ping, SSH and the gateway all went down together. After a restart, the previous boot’s journal just stopped mid-operation, with no kernel panic, no NVIDIA driver Xid error and no out-of-memory kill. Holding the power button was the only way back.

For a business this is the worst kind of failure. One machine runs everything, so one freeze stops every workflow at once, and with no error message there is nothing to fix.

What we were aiming for

• No more freezes caused by how memory was set up, with a written budget everyone could check.
• Any freeze that did happen should recover by itself and be noticed straight away, even if the server couldn’t report it.
• A model we suspected but couldn’t diagnose should be testable without putting everything else at risk.
• Every number in the budget should come from a measurement, not a guess.

How we approached it

The options pulled in different directions. We could give every model a bigger context so prompts stopped being cut off, or cap it so memory stayed predictable. We could keep several parallel slots so callers rarely waited, or use one slot and accept a queue. We could try to root-cause the suspect model, or contain it.

We chose predictability every time. Contexts are capped, each model gets one slot, and the suspect model runs in its own sandbox. We accepted that some callers would wait and that long jobs would have to fit a smaller window. Because nothing on the server could log its own death, we also decided that detection had to live somewhere else.

What we built

The challenges below explain how we found each cause; this is the setup we ended with.

Pinned software and one way in

Ollama, the server that runs the models, is pinned to an exact version, and changes go through the repository and a deploy script (Part 6), never a web UI. Ollama listens only on the server itself (loopback). Every client goes through the gateway, which requires authentication. Keep-alive went from 30 seconds to 5 minutes, so models swap in and out less often.

A sandbox for the risky model

A second, loopback-only Ollama instance runs the suspect model alone, with conservative settings: flash attention off, an unquantised f16 KV cache (the working memory a model keeps for every token of its context), one slot, a 16k context and a 2-minute keep-alive. Its container has a 48 GB memory limit, but on unified memory that only limits the process’s system memory, so the one-slot, 16k setting is the real guard for GPU allocations. Only gateway keys that name the model can call it, at up to 20 requests a minute. A 20-minute phased soak test, from a cold load through long prompts, tool calls and concurrent requests to an idle unload, finished with 53 requests, no errors, no GPU alarms and no kernel messages.

A watchdog and an outside heartbeat

A hardware watchdog (systemd’s RuntimeWatchdogSec=60s) has the init process keep resetting a hardware timer. If the kernel hangs, the board reboots itself after 60 seconds, so with boot time and an automatic GPU-access restore a freeze becomes a three to four minute outage. Because a watchdog can miss a stall, a separate device also checks the server and raises an alert when it stops answering.

A written memory budget

Capacity is a chain of numbers, so changing one means recalculating the rest.

Detection that gets read

The per-minute sampler now warns on low free memory, high GPU memory, growing swap or a prompt near the largest reserved context. Severe levels send a Telegram alert, as does any prompt bigger than the gateway would ever admit, a second line of defence in case something bypasses the gateway. The dashboard shows memory pressure and each model’s reserved context.

Challenges we hit

A silent upgrade and a model that hung while loading

The Ollama container had been recreated from a container-management web UI with "recreate with pull" on the :latest tag, which quietly upgraded Ollama and pruned "unused" images, including the one OCR needed. Three days later came the first freeze. Five days before that, because the Telegram agent’s prompts were being cut off silently at 16k, we had raised the two main models’ context to 65,536 tokens in each model’s own settings file, the Modelfile.

The sampler’s last lines before freezes one, two and three matched. nemotron-3-nano:30b had loaded within the previous 60 seconds, twice as a swap from another model, with more than 80 GB available and swap near zero. That model loaded about 30 times in the period and hung 3 times, apparently at random, always while loading. It is a hybrid Mamba and mixture-of-experts model with 31.6 B parameters, 3 B active per token, taking 24 GB at Q4_K_M quantisation (weights stored at about 4 bits each), and it was running with an 8-bit cache, flash attention and four slots. Hybrid state-space models use code paths plain transformers never touch. The server died before it could log anything, so we never found the root cause. That is why it got the sandbox.

A crash loop and a side door

The fourth freeze came the next morning, with nemotron not loaded. The tracing service, Langfuse, had restarted 72 times that boot, the last time 2 seconds before the freeze, because a half-finished deployment crashed whenever the gateway sent it traces. Removing the trace callbacks stopped the loop. Three minutes after that reboot, a test client’s IDE plugin loaded nemotron again by calling Ollama directly, past the gateway. We renamed the model’s tags so the old names no longer resolved, and that is why Ollama now listens on loopback only.

The freeze we fully explained

Five days after the sandbox went in, the server froze in the early hours, and the newly added watchdog didn’t recover it. An overnight stock analysis was running, and its debate transcript grows with every call, so the prompts climbed from 50k to 52k, 54k, 57k, 59k, 61k, 63k and finally 65,007 tokens. The model serving it, qwen3-coder, still had num_ctx 65536 in its Modelfile. num_ctx is Ollama’s context window, the most tokens one request can hold, and the per-model value overrode the server-wide OLLAMA_CONTEXT_LENGTH=16384.

With OLLAMA_NUM_PARALLEL=4, Ollama gives every slot the full num_ctx and reserves KV cache for all of them. Four slots at 64k took the model to about 33 GB, roughly 19 GB of weights plus roughly 14 GB of KV cache, though only one client was using one slot. OLLAMA_MAX_LOADED_MODELS=2 let a second model stay loaded beside it. The last readings showed 46.7 GB on the GPU, 65.5 GB of RAM in use and 3.2 GB of swap. The next 65k prompt started processing, and 28 seconds later the server stopped.

KV reserved ≈ slots × context length × bytes of cache per token

For qwen3-coder (48 layers, 4 KV heads of 128 dimensions) an 8-bit cache costs roughly 50 KB per token. Four slots of 65,536 tokens is 262,144 tokens, or about 14 GB, before prompt processing allocates temporary memory on top.

Nothing caught it. The sampler’s log went unread and hung with the host. The gateway passed prompts of any size, because by default LiteLLM treats max_input_tokens as model metadata and enforces it only if you turn on its pre-call checks. Why the watchdog stayed quiet we never proved: it only reboots when the init process stops resetting the timer, so something was evidently still doing that while everything else starved. That is why the heartbeat lives on a separate device.

A reload that takes the GPU away

After the second freeze the GPU was unusable until a second reboot. On this server systemctl daemon-reload (and daemon-reexec, edit, enable or disable) rebuilds the container cgroups and drops their permission to use the GPU. Ollama starts a new runner for every model load, so the model lands on the CPU and nvidia-smi inside the container fails with Failed to initialize NVML. It is a known issue in the NVIDIA Container Toolkit, documented with workarounds such as its CDI mode (the Container Device Interface). We now prefer cron to systemd timers, run a restore script every minute that restarts only broken GPU containers, reload systemd only when a unit file changed, and verify the end state after deploys.

Power-offs that were most likely heat

Two days after the fifth freeze, and again the next day, the server turned off, with about 100 GB of memory free. The one-minute readings beforehand were 88.0 °C and 87.7 °C, below the critical trip point, but an average hides the peak. Measured after the restart, one ordinary stock analysis took the chip from 48 °C to 95 °C in four minutes. Better ventilation brought idle from about 50 °C to about 37 °C, and a recorder now writes a HOT line to disk at 85 °C or above, sampling every 2 seconds.

Calculated numbers that were wrong

In one week, several calculated budgets failed. An embedding model configured for 4,096 tokens was trained on 2,048, and Ollama silently dropped the rest. Doubling the context cost 3.22 GB on qwen3:14b but 0.80 GB on gemma3:12b, and the textbook KV formula was about 19% optimistic, the direction that freezes a server.

The result

One model’s footprint went from about 33 GB at four 64k slots to 22.3 GB at an interim two slots, and 20.3 GB at one slot and 32k. The resident pair measures 37.8 GB, leaving at least 40 GB free. That step of about 2 GB per 32k slot matches the arithmetic. A week of measurement (1,342 requests, 52 overlaps, never more than 4 at once) showed the extra slots were rarely used anyway.

The costs are real. A second caller now waits in Ollama’s queue, a median of 18 seconds and 58 at the 90th percentile. Long jobs have to summarise or retrieve less to fit 32k, and only two models stay loaded. The nemotron freezes were never root-caused, and the 40 GB floor is a policy rather than a proven predictor, since three recorded freezes happened with more than 80 GB available.

Do it yourself: size a memory budget for Ollama

What you need: a Linux server with a GPU or unified memory, Docker with Compose, and, if you use NVIDIA GPUs, the NVIDIA Container Toolkit. Allow about two hours.

1. Pin Ollama, give it the GPU and keep it private. Refer to the image by digest and publish the port on loopback only. The devices block below is the NVIDIA form; other vendors differ.

   services:
     ollama:
       image: ollama/ollama@sha256:<digest>
       restart: unless-stopped
       ports:
         - "127.0.0.1:11434:11434"
       volumes:
         - ollama:/root/.ollama
       deploy:
         resources:
           reservations:
             devices:
               - driver: nvidia
                 count: all
                 capabilities: [gpu]
   volumes:
     ollama:
   

   Check: docker compose config --images prints ollama/ollama@sha256:…, docker compose exec ollama nvidia-smi -L lists the GPU, and curl -m 5 http://<server>:11434 from another machine is refused.

2. Set slots, context and keep-alive explicitly under the same service.

       environment:
         OLLAMA_NUM_PARALLEL: "1"
         OLLAMA_CONTEXT_LENGTH: "32768"
         OLLAMA_MAX_LOADED_MODELS: "2"
         OLLAMA_KEEP_ALIVE: "5m"
   

   Check: docker compose exec ollama env | grep OLLAMA lists all four.

3. Find hidden overrides. Run docker compose exec ollama ollama show <model> --modelfile | grep num_ctx for every model. Any value there beats OLLAMA_CONTEXT_LENGTH, and a client can still override both per request with options.num_ctx. Check: every model either has no num_ctx line or shows the value you intend.

4. Build tags with the context fixed in them. Inside the container, write a two-line Modelfile, FROM <model> and PARAMETER num_ctx 32768, then run ollama create <model>-32k -f Modelfile. Check: ollama show <model>-32k --parameters shows num_ctx 32768.

5. Measure the real footprint. Send each model one short request, then run curl -s localhost:11434/api/ps. Check: note size_vram and context_length for every loaded model (older Ollama versions omit context_length; then read it from ollama ps). These are your budget, not the arithmetic.

6. Measure the KV cost. Load a 16k and a 32k tag of the same model in turn and compare size_vram. The difference is the cost of 16k more context per slot. If the formula disagrees, trust the measurement and add a margin.

7. Write the budget down. Resident models plus a fixed free floor must fit in memory. Check: with both models loaded and a long prompt processing, free -g stays above the floor.

8. Sandbox a risky model. Run a second Ollama service with its own model volume, ports: ["127.0.0.1:11435:11434"] and these settings. KV cache type is global per server, which is why it needs its own instance.

       environment:
         OLLAMA_FLASH_ATTENTION: "0"
         OLLAMA_KV_CACHE_TYPE: "f16"
         OLLAMA_NUM_PARALLEL: "1"
         OLLAMA_CONTEXT_LENGTH: "16384"
         OLLAMA_KEEP_ALIVE: "2m"
   

   Point only the gateway at it. Check: /api/ps on the main instance never lists that model.

9. Add a watchdog and an outside heartbeat. First run sudo wdctl to confirm the board has a hardware watchdog. If it does, create a drop-in so package upgrades don’t overwrite it, and apply it with a planned reboot rather than a re-exec, given the GPU issue above.

   sudo mkdir -p /etc/systemd/system.conf.d
   printf '[Manager]\nRuntimeWatchdogSec=60s\n' | sudo tee /etc/systemd/system.conf.d/watchdog.conf
   

   On a separate device, schedule a check every minute that alerts after a few missed answers. Check: after the reboot sudo wdctl shows a 60-second timeout, and unplugging the server’s network raises the alert.

Key takeaways

• On unified memory, reserved memory is what freezes the machine. Watch slots, context length and cache precision together.
• A model’s own settings file can override your server-wide limit, so build tags with the context fixed in them.
• Pin your images and never let a UI pull :latest.
• Contain what you can’t root-cause, and monitor from outside the box.
• Average temperatures hide peaks, and calculated budgets hide surprises. Measure before you trust a number.