Every model call on our lab’s private AI server goes through one gateway: one key per client, a token budget checked before the GPU sees anything, a trace for every request, and no local model allowed to fall back to the cloud. Here is how we built it, what went wrong, and how to set it up yourself.<\/p>\n\n\n\n
Where this fits \u00b7 Running Local AI in Production \u00b7 Part 2 of 11<\/strong><\/p>\n\n\n\n You are reading Part 2 of 11 in a series where we built a private AI server in our lab and ran realistic test workloads on it for seven weeks. Each part stands on its own. The previous part covered why we built the lab on our own hardware, what ran on it and what that choice cost; this one covers one LLM gateway as the only way to reach a model: keys, budgets and tracing.<\/p>\n\n\n\n \u2190 Part 1: Local AI or Cloud AI? Why We Built a Private AI Server<\/a> \u00b7 Series guide<\/a> \u00b7 Part 3: Why Does a Local AI Server Freeze? Sizing Memory for LLMs \u2192<\/a><\/p>\n\n\n\n Just want it working? Follow the step-by-step setup guide<\/a>: local AI with Ollama and LiteLLM, from a bare server to a secure gateway.<\/p>\n\n\n\n This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.<\/em><\/p>\n\n\n\n In Part 1<\/a> we built a private AI server in our lab, with local models served by Ollama. Its first, hand-built version had a basic control problem: a client could reach a model three ways. It could go through LiteLLM<\/strong>, the gateway meant to be the front door. It could go straight to Ollama, which listened on the lab network. Or it could use a second backend inside Open WebUI, the chat UI, which talked to Ollama directly.<\/p>\n\n\n\n Only the first path knew who was calling. On the second day of our reorganisation, minutes after the server came back from a hard freeze, a test laptop’s IDE plugin called Ollama directly and loaded Keys were just as loose. Nearly everything used the gateway’s admin key<\/strong>, which can create and delete keys and call every model, and copies of it had spread into old IDE templates and a workflow. In a real business that is a privacy and reliability risk at once: you can’t tell who sent what, you can’t limit anyone, and one leaked key opens everything.<\/p>\n\n\n\n An LLM gateway<\/strong> is a proxy between clients and models. Every client talks to one OpenAI-compatible URL, and the gateway decides which backend answers. It is the one place where identity, policy, budgets, concurrency and tracing can all live, so a path that skips it skips all five. We chose LiteLLM because it is open source, speaks the OpenAI API that most tools already support, and has virtual keys<\/strong>, each with its own alias, model list, rate limit and optional budget.<\/p>\n\n\n\n We moved the clients first and closed the side doors after, because closing a path before moving its users breaks them. For budgets, we expected LiteLLM’s built-in settings to do the job, and planned our own check only if they didn’t. For cloud models, we chose a rule enforced before every change over trusting reviewers to spot a risky line.<\/p>\n\n\n\n The stock-analysis experiment and the Telegram agent got their own keys, we switched off the chat UI’s direct Ollama backend, and each knowledge base became an ordinary model behind the gateway. Then we bound every service that only other containers need to the host’s loopback<\/strong> interface, the address only the machine itself can reach. Everything except the gateway now listens on the server itself, the gateway and dashboards need authentication, and Ollama’s own access log is a second witness for anything that slips past.<\/p>\n\n\n\nSee all 11 parts of the series<\/summary>\n\n
\n\n
TL;DR<\/h3>\n\n\n\n
\n\n
Jargon buster: 15 terms used in this post, explained in plain English<\/summary>\n\n
\n\n
In this post<\/h3>\n\n\n\n
\n\n
The problem: three ways in, one key for everything<\/h2>\n\n\n\n
nemotron-3-nano:30b<\/code>, the model whose loading had come just before the earlier freezes. We had removed it from the gateway’s routing the day before, which did nothing for a client that never used the gateway.<\/p>\n\n\n\n
\n\n\n\nWhat we were aiming for<\/h2>\n\n\n\n
\n\n
\n\n\n\nHow we approached it<\/h2>\n\n\n\n
\n\n\n\nWhat we built<\/h2>\n\n\n\n
Closing the side doors<\/h3>\n\n\n\n
One key per client<\/h3>\n\n\n\n