In its first month of testing, our lab AI server froze five times with no kernel error and no warning. This is what caused it, the memory budget we settled on, and the steps to size one for your own server.<\/p>\n\n\n\n
Where this fits \u00b7 Running Local AI in Production \u00b7 Part 3 of 11<\/strong><\/p>\n\n\n\n You are reading Part 3 of 11 in a series where we built a private AI server in our lab and ran realistic test workloads on it for seven weeks. Each part stands on its own. The previous part covered one LLM gateway as the only way to reach a model: keys, budgets and tracing; this one covers five hard freezes and the memory budget that ended them.<\/p>\n\n\n\n \u2190 Part 2: How Do You Control Access to Local AI Models? Use a Gateway<\/a> \u00b7 Series guide<\/a> \u00b7 Part 4: What Happens When a Prompt Is Too Big for a Local LLM? \u2192<\/a><\/p>\n\n\n\n Just want it working? Follow the step-by-step setup guide<\/a>: local AI with Ollama and LiteLLM, from a bare server to a secure gateway.<\/p>\n\n\n\n This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.<\/em><\/p>\n\n\n\n The lab server is an NVIDIA DGX Spark, a compact desktop AI system built on the GB10 Grace Blackwell Superchip with 128 GB of unified memory<\/strong>, one pool shared by the CPU and GPU, roughly 120 GB of it usable. That lets a desk-sized machine hold a 30-billion-parameter model, and makes everything run short at once when the models take too much. Over seven weeks it carried test workflows, a Telegram agent, the LLM gateway our test clients used (Part 2<\/a>) and a multi-agent stock-analysis service on public market data.<\/p>\n\n\n\n In its first month the server froze five times. Each time ping, SSH and the gateway all went down together. After a restart, the previous boot’s journal just stopped mid-operation, with no kernel panic, no NVIDIA driver Xid error and no out-of-memory kill. Holding the power button was the only way back.<\/p>\n\n\n\n For a business this is the worst kind of failure. One machine runs everything, so one freeze stops every workflow at once, and with no error message there is nothing to fix.<\/p>\n\n\n\n The options pulled in different directions. We could give every model a bigger context so prompts stopped being cut off, or cap it so memory stayed predictable. We could keep several parallel slots so callers rarely waited, or use one slot and accept a queue. We could try to root-cause the suspect model, or contain it.<\/p>\n\n\n\n We chose predictability every time. Contexts are capped, each model gets one slot, and the suspect model runs in its own sandbox. We accepted that some callers would wait and that long jobs would have to fit a smaller window. Because nothing on the server could log its own death, we also decided that detection had to live somewhere else.<\/p>\n\n\n\n The challenges below explain how we found each cause; this is the setup we ended with.<\/p>\n\n\n\n Ollama, the server that runs the models, is pinned to an exact version, and changes go through the repository and a deploy script (Part 6), never a web UI. Ollama listens only on the server itself (loopback). Every client goes through the gateway, which requires authentication. Keep-alive went from 30 seconds to 5 minutes, so models swap in and out less often.<\/p>\n\n\n\n A second, loopback-only Ollama instance runs the suspect model alone, with conservative settings: flash attention off, an unquantised f16 KV cache<\/strong> (the working memory a model keeps for every token of its context), one slot, a 16k context and a 2-minute keep-alive. Its container has a 48 GB memory limit, but on unified memory that only limits the process’s system memory, so the one-slot, 16k setting is the real guard for GPU allocations. Only gateway keys that name the model can call it, at up to 20 requests a minute. A 20-minute phased soak test, from a cold load through long prompts, tool calls and concurrent requests to an idle unload, finished with 53 requests, no errors, no GPU alarms and no kernel messages.<\/p>\n\n\n\n A hardware watchdog (systemd’s Capacity is a chain of numbers, so changing one means recalculating the rest.<\/p>\n\n\n\nSee all 11 parts of the series<\/summary>\n\n
\n\n
TL;DR<\/h3>\n\n\n\n
\n\n
Jargon buster: 15 terms used in this post, explained in plain English<\/summary>\n\n
\n\n
In this post<\/h3>\n\n\n\n
\n\n
The problem: why did the server keep freezing?<\/h2>\n\n\n\n
\n\n\n\nWhat we were aiming for<\/h2>\n\n\n\n
\n\n
\n\n\n\nHow we approached it<\/h2>\n\n\n\n
\n\n\n\nWhat we built<\/h2>\n\n\n\n
Pinned software and one way in<\/h3>\n\n\n\n
A sandbox for the risky model<\/h3>\n\n\n\n
A watchdog and an outside heartbeat<\/h3>\n\n\n\n
RuntimeWatchdogSec=60s<\/code>) has the init process keep resetting a hardware timer. If the kernel hangs, the board reboots itself after 60 seconds, so with boot time and an automatic GPU-access restore a freeze becomes a three to four minute outage. Because a watchdog can miss a stall, a separate device also checks the server and raises an alert when it stops answering.<\/p>\n\n\n\nA written memory budget<\/h3>\n\n\n\n