Is Private AI Worth It? Lessons From Seven Weeks in Our Lab (Part 11 of 11)

The last post in the series: what our private AI lab set out to prove, the platform we ended up with, ten lessons that started as real failures, where one local server falls short, and a checklist for planning your own.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The problem: can one private server do real AI work?

Many organisations would like to use AI on emails, invoices, documents and internal knowledge, but are uneasy about sending that data to a cloud provider. The obvious alternative is to run models on hardware you own. The question this series set out to answer was whether that is practical on a single compact server, and what it takes to keep it reliable.

The answer wasn’t obvious at the start. In seven weeks the lab server had five hard freezes, a monitoring sensor that was blind for more than a day, an agent prompt too big to say "hi", and two power-offs most likely caused by heat. Each of those would be an outage, a silent failure or a risk to data in a real deployment.

What we were aiming for

• Business-style data stays on the server, with no silent route to a cloud service.
• A server that doesn’t freeze, with limits that are measured rather than guessed.
• Every model call traceable to a named client, and every rule enforced by the machine, not by memory or good intentions.
• A clear-eyed view of what local AI can’t do, so the choice between local and cloud is an informed one.

How we approached it

We built a lab rather than writing a paper, and ran realistic test workloads on synthetic and sample data. We could have bought more hardware or used cloud APIs for the hard parts. Instead we kept to an NVIDIA DGX Spark, a compact desktop AI system built on the GB10 Grace Blackwell Superchip with 128 GB of unified memory (roughly 120 GB usable) shared by the CPU and GPU, and ran it the way you would run a production system. That meant one gateway, measured budgets, git-based deployment, monitoring and written rules, as covered in Part 1 onwards. When something broke, we found the cause, fixed it and turned the fix into a rule.

What we built: the whole platform in one view

Every client (workflows, a chat window, a Telegram agent, lab devices and a stock-analysis experiment) calls one gateway with its own key. The gateway checks the key and the input budget, compacts long conversations, routes auto requests, queues work and records a trace. Behind it, Ollama serves the local models, two knowledge-base workers answer from documentation, and an OCR service loads on demand. Monitoring watches memory, heat and refusals, with an external heartbeat.

Almost all of it lives in one git repository and is deployed by script. The few exceptions, such as the agent’s own settings, are copied before each change, logged and watched by an alert.

Challenges we hit: ten lessons from real failures

1. Reserved memory freezes the server, not the weights

The worst failure was a hard freeze that needed a power cycle, with no kernel, GPU or out-of-memory error. A stock analysis’s transcript had grown to 65,007 tokens. The model serving it, qwen3-coder, had a 64k context built in, and Ollama gives every parallel slot the full context and reserves KV cache, the model’s working memory, for each. Four slots took that one model to about 33 GB, roughly 19 GB of weights plus roughly 14 GB of KV cache. The next 65k prompt began processing, and 28 seconds later the server stopped. Monitoring on the same host hung with it, and a hardware watchdog didn’t fire, so we added an external heartbeat on a separate device. With 32k contexts and one slot, the same model takes 20.3 GB (Part 3).

2. Measure, don’t calculate

Three budgets were wrong in one week because someone worked them out instead of observing them. The embedding model clamps at 2,048 tokens, not the 4,096 we assumed, so a 9,071-token request came back "successful" having read 2,048. Dense data runs closer to 2 characters per token than 3. Doubling context cost 3.22 GB on one model and 0.80 GB on another. Load the model, read what it took, and check the arithmetic against that.

3. One door for every model

Early freezes came partly from test clients calling Ollama directly, loading models nobody planned for with no record of which client asked. Now every call goes through LiteLLM, and everything except the gateway listens on the server itself. No local model may fall back to a cloud model, because a fallback is set per model and would send anyone’s request off the server (Part 2).

4. Refuse loudly rather than truncate quietly

Ollama silently cuts a prompt that is too long. We wrote our own admission check, because LiteLLM only enforces context limits with pre-call checks on. Then we found it admitting prompts up to 322 times larger than it counted. Tool definitions weren’t counted, so a short message with twelve tool schemas counted 11 tokens and cost 3,540, and digits were counted in threes when these models spend one token per digit. Of 76 test samples, 59 were unsafe, and none after the fix. Compaction at the gateway now keeps the system prompt, recent turns and every tool call with its result, and stores anything trimmed (Part 4).

5. An agent’s fixed prompt is a budget

The Telegram agent once refused "hi" at 26,484 tokens against a 26,000 budget, because it loaded its framework’s developer guide from the wrong working directory. Earlier, about 90 admin tool schemas had pushed its fixed prompt to 32–35k. Cutting tools from 37 to 24 brought it to 14.9k (Part 9).

6. Sensors must fail loudly

The per-minute capacity probe was blind for more than a day because it ran curl in a container image that doesn’t include it, and swallowed the error, reporting "nothing loaded". A probe that can’t get an answer now publishes unknown (null), never 0. A swap warning that fired every minute for a day now judges swap by how fast it grows (Part 7).

7. Pin everything and deploy from git

The freezes began when a container management UI recreated Ollama from a :latest tag and quietly upgraded it, and the same tool later pruned the OCR image. Every image is now pinned, :latest is blocked at commit, and that UI is view-only. The deploy script detects drift and restores GPU access afterwards, because a service-manager reload can strip it, and updates run only in a maintenance window (Part 6).

8. AI agents need guardrails the machine enforces

AI coding agents helped build the lab, but an instruction file only works for a session that opens it. This is how the lab runs now. Agents get scoped, unprivileged accounts, and their changes go through the repository and review. Secrets stay out of any file an agent can read, files agents tend to edit are immutable between deploys, and local-to-cloud fallbacks are refused at commit time.

9. "Off" switches can lie

The agent’s simulated sensor feed, set to enabled: false, was still sending 442 events an hour, each a 15k-token turn, and replies took up to two minutes. The integration switches itself on when configured. Parking its watch lists brought replies back to 3–7 seconds. Check "off" by watching traffic.

10. Heat is a capacity limit too

The server switched itself off twice with about 100 GB of memory free, most likely from heat, though we didn’t prove it. Sampling every 2 seconds after the restart, an ordinary stock analysis took the hottest zone from 48 °C to 95 °C in four minutes, which the one-minute average had hidden. Better ventilation took idle from about 50 °C to about 37 °C, and the stock-analysis tool now sends its deep calls to a free cloud model when the server runs hot, which is allowed only because its data is public (Part 8).

The result: it works, within honest limits

The server stopped freezing. The warm pair fits in 37.8 GB with at least 40 GB kept free, oversized prompts are refused or compacted rather than cut, and every call is traced to a named client. The chat agent answers in 3–7 seconds, and documents never take a path off the server.

The limits are real. One slot means a second caller waits, with a median of 18 seconds and a 90th percentile of 58 over seven days of test load. Only two models stay warm, and auto routing only got from 9 seconds to 0.4 once the router stopped forcing model swaps. The workhorses are 30B-class quantised models (weights stored at lower precision so they fit), good at routing, extraction, summaries and tool use but not frontier models. Scored by agreement with later price moves at five trading days, the stock-analysis agents agreed 30% of the time across their first ten calls, while always answering "Buy" agreed 40%. Ten calls prove nothing, but now we know instead of assuming. That was an evaluation of AI agents, not financial advice.

Cloud still has a place for public, non-sensitive work. The stock-analysis experiment may use free OpenRouter models for its two deep calls per run, on a $0 credit limit, compacted to the local budget, with a local fallback and every call logged. Nothing else goes there.

Next, roughly in order, we plan to roll out a newer Ollama one instance at a time after its sandbox run, move to Langfuse v4 and replace the object store, as the upstream project is no longer maintained. We will load the agent’s tools per conversation, add scheduled smoke tests through every entry point and workflow, choose the deep-thinking model once each candidate has 20 scored calls, and turn event feeds into digests. More capacity starts with a measurement, and if demand keeps growing, a second server is the honest answer to the queue. A heat-aware gate that briefly holds batch work, never interactive traffic, is designed and waiting on the ventilation results.

Very little of what we learned was about the models themselves. It was about memory, measurement, plain operational habits and keeping agents inside sensible fences, and the failures taught us the most. That’s the end of the series, and thank you for reading it.

Do it yourself: plan your own private AI server

What you need: a server with a GPU or unified memory large enough for your two busiest models plus generous headroom, Linux, Docker and Git, and time to work through the series. Each step links the part that covers it in depth.

1. Decide what must stay private. List your workloads and mark which data may never leave your hardware (Part 1).

   Check: every workload has a data class and a "local only" or "cloud allowed" label.

2. Size memory, not compute. For each model you’ll keep warm, add the weights to context × slots of KV cache, then keep a large share free (Part 3).

   Check: after loading, curl -s 127.0.0.1:11434/api/ps reports sizes close to your estimate.

3. Cap Ollama. Bind it to loopback and set OLLAMA_NUM_PARALLEL and OLLAMA_CONTEXT_LENGTH deliberately, as in Part 1’s steps and Part 3. Changing the service reloads systemd, which can strip GPU access from running GPU containers; see the GPU caveat in Part 3 first.

   Check: nothing outside the server can reach Ollama’s port.

4. Put one gateway in front with a virtual key per client and no shared admin key (Part 2).

   Check: a request without a valid key is refused, and usage shows per key.

5. Set an input budget and compaction for each model, below its context (Part 4).

   Check: an oversized prompt is refused or compacted, never silently cut.

6. Add routing only when you have several models, keeping the classifier small (Part 5).

   Check: auto answers quickly without evicting a warm model.

7. Run it from git. Pin every image by digest, block :latest, and deploy by script in maintenance windows (Part 6).

   Check: every image: line in docker compose config contains @sha256:.

8. Watch it from outside. Trace every call, publish "unknown" when a sensor fails, and add an external heartbeat (Part 7).

   Check: switch the server off in a test window and confirm the alarm arrives.

9. Add workloads one at a time. Agents with measured prompts (Part 9), workflows and knowledge bases (Part 10), and multi-agent jobs (Part 8), with local-to-cloud fallbacks refused at commit.

   Check: each workload passes a smoke test, streamed and with tools, before the next one starts.

10. Back up nightly to a separate target, alert on any failed run, and test a restore before relying on it (Part 6).

    Check: you have restored a backup onto a machine other than the server, and the restored services start and answer.

Key takeaways

• On unified memory, budget for context × slots on top of the weights, and let the machine enforce it.
• Measure every number the system depends on, and measure again whenever you change anything.
• Send every model call through one gateway, with per-client keys, budgets, tracing and no silent route to the cloud.
• Prefer a clear refusal to a quiet truncation, and make every sensor say "unknown" when it doesn’t know.
• Give AI agents scoped access, keep secrets out of their reach, and gate their changes through the repository.
• Local AI buys privacy, predictable cost and control, at the price of queueing, smaller models and real operations work.