How Do You Control Access to Local AI Models? Use a Gateway (Part 2 of 11)

Every model call on our lab’s private AI server goes through one gateway: one key per client, a token budget checked before the GPU sees anything, a trace for every request, and no local model allowed to fall back to the cloud. Here is how we built it, what went wrong, and how to set it up yourself.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The problem: three ways in, one key for everything

In Part 1 we built a private AI server in our lab, with local models served by Ollama. Its first, hand-built version had a basic control problem: a client could reach a model three ways. It could go through LiteLLM, the gateway meant to be the front door. It could go straight to Ollama, which listened on the lab network. Or it could use a second backend inside Open WebUI, the chat UI, which talked to Ollama directly.

Only the first path knew who was calling. On the second day of our reorganisation, minutes after the server came back from a hard freeze, a test laptop’s IDE plugin called Ollama directly and loaded nemotron-3-nano:30b, the model whose loading had come just before the earlier freezes. We had removed it from the gateway’s routing the day before, which did nothing for a client that never used the gateway.

Keys were just as loose. Nearly everything used the gateway’s admin key, which can create and delete keys and call every model, and copies of it had spread into old IDE templates and a workflow. In a real business that is a privacy and reliability risk at once: you can’t tell who sent what, you can’t limit anyone, and one leaked key opens everything.

What we were aiming for

• One door: every model call goes through the gateway, and nothing else on the network can reach a model.
• One key per client, with its own model list and limits, and the admin key used for administration only.
• Oversized prompts refused or trimmed before they reach the GPU, never silently truncated.
• No path, deliberate or accidental, by which local data reaches a cloud model, and a trace for every call.

How we approached it

An LLM gateway is a proxy between clients and models. Every client talks to one OpenAI-compatible URL, and the gateway decides which backend answers. It is the one place where identity, policy, budgets, concurrency and tracing can all live, so a path that skips it skips all five. We chose LiteLLM because it is open source, speaks the OpenAI API that most tools already support, and has virtual keys, each with its own alias, model list, rate limit and optional budget.

We moved the clients first and closed the side doors after, because closing a path before moving its users breaks them. For budgets, we expected LiteLLM’s built-in settings to do the job, and planned our own check only if they didn’t. For cloud models, we chose a rule enforced before every change over trusting reviewers to spot a risky line.

What we built

Closing the side doors

The stock-analysis experiment and the Telegram agent got their own keys, we switched off the chat UI’s direct Ollama backend, and each knowledge base became an ordinary model behind the gateway. Then we bound every service that only other containers need to the host’s loopback interface, the address only the machine itself can reach. Everything except the gateway now listens on the server itself, the gateway and dashboards need authentication, and Ollama’s own access log is a second witness for anything that slips past.

One key per client

Model lists are opt-in, so the riskiest model, by then in its own sandbox, is only available to keys that ask for it. Interactive keys get a lower requests-per-minute ceiling than batch tools, so one runaway loop can’t fill the queue. We removed every copy and moved each client to its own key. The admin key is used by LiteLLM and admin commands only.

AI coding agents are treated like a new contractor: scoped, unprivileged access, no secrets in any file they can read, and every change through the repository and review. The live gateway configuration is read-only between deploys (Part 6).

A budget check at the gate

After the freezes described in Part 3, each model gets a 32,768-token context window per request, shared by the prompt and the answer. At that limit Ollama doesn’t refuse; it truncates silently, and an agent loses part of its own reasoning without knowing.

We built an admission check: a small pre-call hook in the gateway that counts each request against the model’s budget and refuses anything over it with an OpenAI-style 400 context_length_exceeded error, naming the count, the budget and what to do. The input budgets are 26,000 tokens for the four 32k models and the auto router, leaving at least 6,768 tokens for the reply, and 13,000 for the two models with a 16k window. Later, refusing became compacting at the gateway (Part 4).

Queueing and tracing

Each heavy model has max_parallel_requests: 2. In the LiteLLM version we ran, a request over that cap waited for a free slot; current releases reject it at once with a 429 instead. Setting it to 1 would only have moved the queue from Ollama into LiteLLM, so we made queueing visible instead (Part 7). Every request leaves a row in LiteLLM’s spend log and a trace in Langfuse, an open-source tool for observing LLM calls.

No cloud fallback, and one narrow exception

LiteLLM fallbacks retry a failed or slow request on another model. We use them only between the two resident local models, gemma4 and qwen3-coder. A pre-commit egress check refuses:

1. any fallback (ordinary, context-window or content-policy) from a local model to a cloud model;
2. any default_fallbacks entry that names a cloud model, because a default applies to local models too;
3. any OpenRouter deployment whose model id doesn’t end in :free;
4. any fallback naming a model that doesn’t exist, because a typo would silently switch off the first rule.

The one exception covers public data only. TradingAgents, the stock-analysis experiment (Part 8), makes about 30 local analyst calls per run plus two "deep" calls, and those two may use a model on OpenRouter’s free tier, only for the lab’s public-market test data. Free tiers may log or train on prompts, so nothing else goes to them.

Four controls keep that exception narrow:

• the cloud models are granted to the stock-analysis key only;
• the OpenRouter key has a $0 credit limit;
• each sets allow_fallbacks: false, so OpenRouter can’t re-route to another provider of the same model (it doesn’t stop a model fallback configured elsewhere);
• the only fallback goes from each free model to local qwen3-coder, with prompts compacted to the local budget first, and every call leaves a log line saying where it ran.

Challenges we hit

The built-in budget setting doesn’t enforce anything

We assumed LiteLLM’s per-model max_input_tokens would stop oversized prompts. By default it is only model metadata. LiteLLM enforces it only when pre-call checks are on (enable_pre_call_checks), and then it filters deployments rather than telling the caller what went wrong. That is why we wrote our own hook. Even then, a budget in the wrong place did nothing: the hook reads model_info, and a value under litellm_params looks right but is ignored. A pre-commit check now refuses a local model without a declared budget, and the guard re-checks the running gateway.

The token counter was wrong by 322 times

The gate had been letting through prompts it should have refused, because it never counted tool definitions. A short message carrying a dozen tool schemas counted as 11 tokens but cost 3,540. The counter also grouped digits three to a token, while these models spend one token per digit. Of 76 samples, 59 were unsafe before the fix.

Traces that never arrived

The Langfuse callbacks had sat under the wrong configuration section since the first build, and LiteLLM ignores them there without complaint, so no traces had ever been recorded. Moved to the right section, every trace batch crashed the Langfuse web container. Langfuse v3 needs Postgres, ClickHouse, Redis, S3-compatible blob storage and a worker, and ours had been installed without the last three. It restarted 72 times before we took the callbacks out and completed the stack.

Keys leak in quiet ways

In LiteLLM an empty model list means every model, cloud ones included, which is how a dashboard key quietly gained access to everything. Any key with every-model access must now be listed in a file in the repository, so the commit is the approval. A load test once fell back to the admin key and raised "admin key used", the one alert that must never be noise, so tests now use a dedicated key limited to local models with a $1 per 30 days tripwire. Open WebUI stores its key in its environment and its database, and the database copy wins at start-up.

A fallback can send a document to the cloud

Picture a document-reading model with a short timeout and a cloud fallback. Whenever the single local slot stays busy past the timeout, LiteLLM re-sends the request, meaning the document itself, to the cloud provider, and the caller can’t tell. A fallback belongs to a model, not a key, so it applies to every caller. The lab’s vision model has no cloud fallback and no timeout; a busy slot means the call waits.

The result

In the first test of the gate, a 45,183-token request was refused in milliseconds and never reached the GPU. After the counting fix, none of the 76 samples was unsafe. Traces now arrive in Langfuse, and a soak test ran with no restarts. From another lab machine the database and model ports were refused while the gateway answered. A scheduled guard compares gateway and Ollama logs, alerts on admin-key use, flags unapproved every-model keys and counts refusals per key. We added that last check after the Telegram agent had 20 requests refused over five days and nothing said so (Part 9).

One door has costs. The gateway is a single point of failure: a configuration change means a restart of about 10 seconds, or about 30 for a recreate. Budgets sit below the model card. With one slot per model, a colliding caller waited a median of 18 seconds over seven days of test traffic, with 90 per cent of waits under 58 seconds. Every new client needs a key and a model list, on purpose. And alerts are not enforcement: scoped access, read-only configuration and pre-commit checks are what prevent problems.

Do it yourself: put LiteLLM in front of Ollama

What you need: the server from Part 1 with Ollama on loopback, Python 3, Postgres (a distro package or the official container), and about two hours.

1. Prepare the database and secrets. Create a service account, a Postgres role and a database, then an environment file only that account can read. LiteLLM’s admin key, the master key, must start with sk-.

   sudo useradd --system --home /opt/litellm litellm
   sudo -u postgres createuser -P litellm && sudo -u postgres createdb -O litellm litellm
   sudo install -d -m 750 -o litellm /etc/litellm
   printf '%s=%s\n%s=%s\n' LITELLM_MASTER_KEY "sk-$(openssl rand -hex 32)" \
     DATABASE_URL 'postgresql://litellm:<db-password>@127.0.0.1:5432/litellm' \
     | sudo install -m 600 -o litellm /dev/stdin /etc/litellm/litellm.env
   

   Check: sudo -u postgres psql -lqt | grep litellm lists the database, and sudo ls -l /etc/litellm/litellm.env shows -rw-------.

2. Install a pinned proxy in its own virtual environment. The extra_proxy extra brings the Prisma client the proxy needs once a database is configured.

   sudo python3 -m venv /opt/litellm
   sudo /opt/litellm/bin/pip install 'litellm[proxy,extra_proxy]==<version>'
   

   Check: /opt/litellm/bin/pip show litellm prisma prints both, with the version you pinned. Read the release notes before any upgrade, because pip doesn’t run database migrations for you.

3. Write /etc/litellm/config.yaml. Use ollama_chat/ so tool calls work, and set the budget in model_info.

   model_list:
     - model_name: qwen3-coder
       litellm_params:
         model: ollama_chat/qwen3-coder-32k
         api_base: http://127.0.0.1:11434
         max_parallel_requests: 2
       model_info:
         max_input_tokens: 26000
   router_settings:
     enable_pre_call_checks: true
   general_settings:
     master_key: os.environ/LITELLM_MASTER_KEY
     database_url: os.environ/DATABASE_URL
   

   Add no fallbacks or default_fallbacks that point at a cloud model. On current releases a request over max_parallel_requests gets an immediate 429, so pin the version you tested, raise the cap and let Ollama queue, or handle 429 in clients. For a quick start, enable_pre_call_checks filters out models too small for a request. For a clear error and a count that includes tool definitions, write the hook in Part 4’s steps and register it under litellm_settings: callbacks. Check: the file is committed to Git.

4. Run the gateway as a systemd service under the unprivileged account, bound to loopback.

   [Service]
   User=litellm
   EnvironmentFile=/etc/litellm/litellm.env
   ExecStart=/opt/litellm/bin/litellm --config /etc/litellm/config.yaml --host 127.0.0.1 --port 4000
   Restart=on-failure
   

   Save it as /etc/systemd/system/litellm.service with [Install] WantedBy=multi-user.target, then sudo systemctl enable --now litellm. Put a TLS reverse proxy in front for other machines, or firewall port 4000 to known hosts; keys are bearer tokens, so never send them over plain HTTP. Check: curl -s localhost:4000/health/liveliness answers that it is alive.

5. Mint one key per client, on the server itself, so the master key never crosses the network. Load it first with set -a; . /etc/litellm/litellm.env; set +a in a root shell.

   curl -s localhost:4000/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H 'Content-Type: application/json' \
     -d '{"key_alias":"chat-ui","models":["qwen3-coder"],"rpm_limit":30}'
   

   Check: the JSON contains a key. Store it only in that client’s configuration.

6. Send a test request with the client key.

   curl -s localhost:4000/v1/chat/completions -H "Authorization: Bearer $CLIENT_KEY" \
     -H 'Content-Type: application/json' \
     -d '{"model":"qwen3-coder","messages":[{"role":"user","content":"Say hello."}]}'
   

   Check: you get an answer, the call appears in the spend log under chat-ui, and asking for a model not on the key’s list returns an authorisation error.

7. Test the budget. Send a prompt of around 40,000 words, generated with a short script. Check: you get an error, not an answer, and Ollama’s log (journalctl -u ollama) shows no new POST /api/chat line for it. Then send a short message with a dozen tool definitions. With only the built-in checks it will almost certainly get through, because LiteLLM’s counter ignores tool schemas; Part 4’s hook closes that gap.

8. Refuse cloud fallbacks before commit. Add a pre-commit script that parses config.yaml and fails if any fallback leads from an ollama model to another provider. Check: add a bad fallback on purpose; the commit is refused.

9. Confirm only the gateway is reachable. From another machine, run curl -m 5 http://<server>:11434, then request the gateway’s health path through your TLS proxy. Check: the first is refused or times out and the second answers.

Key takeaways

• One door makes control possible. Bind everything else to loopback and keep the model server’s log as a second witness.
• The admin key is for administration only, not even for your own tests.
• LiteLLM’s max_input_tokens is metadata by default. Switch on pre-call checks or enforce it yourself, and count tool definitions.
• Fallbacks belong to the model, not the key. Refuse local-to-cloud fallbacks in CI rather than hoping a review catches them.
• If you allow cloud models at all, keep them narrow: one key, public data only, no provider re-routing, a local fallback and a log line for every call.