How Do You Set Up Local AI? A Step-by-Step LiteLLM Guide

A step-by-step guide to running AI on your own Linux server: Ollama kept private, LiteLLM as the only way in, a key per app, a chat UI, tracing and a locked-down server. Every step ends with a check.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The practical companion to our 11-part series: what to build, in what order, and how to prove it works.

What you will end up with

At the top are your apps, scripts, coding assistants and a chat UI (Open WebUI). All of them talk to one OpenAI-compatible address, the LiteLLM gateway, which checks each caller’s key, models, rate limits and budget, refuses anything too big, and logs and traces every call. Behind it sits Ollama, the model server, listening only on the server’s loopback address (127.0.0.1), running the models on the GPU. Nothing leaves the server unless you deliberately allow it.

It is for IT teams and developers comfortable on a Linux command line. Local AI fits sensitive data, a fixed cost and control over when a model changes; the cloud still fits public data, bursty workloads and the largest models. Many teams use both, kept apart by rules the gateway enforces.

Before you start

Hardware. What matters is memory the GPU can use: VRAM, or the shared pool on a unified-memory machine. Rough figures for 4-bit models with one request slot; measure your own in Step 2.

The "plan for" column adds the KV cache, the memory reserved for context, which grows with context length and parallel slots. In the lab, a 30B mixture-of-experts coding model measured 20.3 GB at 32k with one slot, but about 33 GB (roughly 19 GB of weights plus 14 GB of KV cache) with four slots at 64k. On unified memory everything shares one pool, so leave a generous free floor. Our lab server was an NVIDIA DGX Spark, a compact desktop AI system built on the GB10 Grace Blackwell Superchip with 128 GB of unified memory shared by the CPU and GPU (the operating system reports roughly 120 GB as usable). We had already benchmarked two of them linked together; this guide needs only one, and any Linux server with enough GPU memory works.

Software.

• A current long-term-support Debian or Ubuntu release.
• A working NVIDIA driver (nvidia-smi lists the GPU) and the NVIDIA Container Toolkit.
• Docker Engine with the Compose v2 plugin (docker compose), for Open WebUI and Langfuse.
• Python 3.11 or later, for LiteLLM.
• For remote access, a domain name for TLS, or a firewall.

Time. Allow a day for Steps 1 to 6, plus half a day for tracing and lockdown.

Step 1: Install Ollama and keep it private

The install script creates an ollama service account and a systemd service; OLLAMA_VERSION pins the release, so upgrades happen when you choose.

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=<version> sh
systemctl status ollama --no-pager

Set memory settings explicitly; defaults change between releases. Current versions pick the default context from the GPU memory they detect: 4k below 24 GB, 32k up to 48 GB and 256k above that. Run sudo systemctl edit ollama and add:

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_CONTEXT_LENGTH=32768"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_KEEP_ALIVE=30m"

Then sudo systemctl restart ollama. Why each one matters:

• OLLAMA_HOST pins the listener to IPv4 loopback (the default, made explicit; a wildcard address exposes every model).
• OLLAMA_NUM_PARALLEL is request slots per model. Each slot gets the full context, so memory grows with context × slots. With one slot a second caller waits; in the lab the median wait was 18 seconds.
• OLLAMA_CONTEXT_LENGTH is the context window, shared by prompt and answer.
• OLLAMA_MAX_LOADED_MODELS (default three per GPU) stops a stray request loading a third model.
• OLLAMA_KEEP_ALIVE (default five minutes) is how long an idle model stays loaded.

Caveat: systemctl edit, like any systemd reload (enable and disable included), can strip GPU access from running NVIDIA containers until they restart. Do it in a maintenance window and check each with docker exec <container> nvidia-smi -L.

Or in a container. Pin the image by digest, publish on loopback only and reserve the GPU, or it quietly runs on the CPU. Then prefix the ollama commands below with docker compose exec ollama.

services:
  ollama:
    image: ollama/ollama@sha256:<digest>
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      OLLAMA_NUM_PARALLEL: "1"
      OLLAMA_CONTEXT_LENGTH: "32768"
      OLLAMA_MAX_LOADED_MODELS: "2"
      OLLAMA_KEEP_ALIVE: "30m"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  ollama:

Check: ss -ltn | grep 11434 shows 127.0.0.1:11434 and nothing else. systemctl show ollama --property=Environment lists all five settings. From a second machine, curl -m 5 http://<server-address>:11434 is refused or times out.

Step 2: Choose models and measure them

Start with two models: a general one for questions and drafting, and a coding one for code and agents. Here, a 12B Gemma model and a 30B mixture-of-experts Qwen coder, which activates only a few of its sub-networks per token, so it answers about as fast as a much smaller model. Check the Ollama library for current tags.

A model’s own context setting beats the server-wide one: in the lab, a 65,536-token value in a model’s settings quietly overrode the server limit and contributed to the freezes. So build tags with the context fixed:

ollama pull gemma3:12b
ollama pull qwen3-coder:30b
printf 'FROM gemma3:12b\nPARAMETER num_ctx 32768\n' > general.Modelfile
printf 'FROM qwen3-coder:30b\nPARAMETER num_ctx 32768\n' > coder.Modelfile
ollama create general-32k -f general.Modelfile
ollama create coder-32k -f coder.Modelfile

Send each model one request and read what it holds:

for m in general-32k coder-32k; do
  curl -s 127.0.0.1:11434/api/generate -d "{\"model\":\"$m\",\"prompt\":\"Hi\",\"stream\":false}"
done
ollama ps
curl -s 127.0.0.1:11434/api/ps | python3 -m json.tool

Write your memory budget down in Git: measured resident models plus a fixed free floor must fit. In the lab the resident pair measured 37.8 GB, and we keep at least 40 GB of the shared pool free. On a discrete GPU, leave roughly 10 per cent of VRAM spare.

Check: ollama ps shows 100% GPU under PROCESSOR and 32768 under CONTEXT for both models (older versions lack that column; read context_length from /api/ps). ollama show coder-32k --parameters lists num_ctx 32768. If PROCESSOR shows any CPU share, fix the driver or the container’s GPU access before you go on.

Step 3: Put LiteLLM in front as the only door

LiteLLM is an open-source gateway that speaks the OpenAI API. Identity, model access, budgets and tracing all live there, so any path that skips it skips all of them. It keeps keys and spend logs in Postgres.

First, the database, a service account and the secrets. LiteLLM’s admin key, the master key, must start with sk-. The salt key encrypts credentials stored in the database and must never change once set. Systemd reads the environment file as root before starting the service, so it can be root-only (mode 600).

sudo apt install -y postgresql python3-venv
sudo useradd --system --create-home --home-dir /opt/litellm --shell /usr/sbin/nologin litellm
sudo -u postgres createuser -P litellm
sudo -u postgres createdb -O litellm litellm
sudo install -d -m 750 -o root -g litellm /etc/litellm
printf '%s=%s\n' \
  LITELLM_MASTER_KEY "sk-$(openssl rand -hex 32)" \
  LITELLM_SALT_KEY "sk-$(openssl rand -hex 32)" \
  DATABASE_URL 'postgresql://litellm:<db-password>@127.0.0.1:5432/litellm' \
  | sudo install -m 600 -o root -g root /dev/stdin /etc/litellm/litellm.env

Install a pinned version into a virtual environment owned by the service account, because the proxy generates its Prisma database client on first start. The extra_proxy extra brings that client; without it the proxy fails at start-up.

sudo -u litellm python3 -m venv /opt/litellm/venv
sudo -u litellm /opt/litellm/venv/bin/pip install 'litellm[proxy,extra_proxy]==<version>'

Write /etc/litellm/config.yaml (owner root, group litellm, mode 640):

model_list:
  - model_name: general
    litellm_params:
      model: ollama_chat/general-32k
      api_base: http://127.0.0.1:11434
    model_info:
      max_input_tokens: 26000
  - model_name: coder
    litellm_params:
      model: ollama_chat/coder-32k
      api_base: http://127.0.0.1:11434
    model_info:
      max_input_tokens: 26000
router_settings:
  enable_pre_call_checks: true
litellm_settings:
  drop_params: true
  request_timeout: 600
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

The deliberate choices:

• ollama_chat/ uses Ollama’s chat endpoint, which returns native tool calls; with ollama/, agents get text instead.
• max_input_tokens: 26000 leaves 6,768 tokens of the 32k window for the answer. On its own it is only metadata; enable_pre_call_checks and Step 5 make it count.
• No max_parallel_requests: current releases reject a request over that cap at once with a 429 instead of queueing it. With one slot, Ollama already queues the second caller.
• No fallbacks or default_fallbacks (see Step 8).
• One litellm_settings block only. Later steps add to it; a second litellm_settings: line silently replaces the first.

Run it as a systemd service bound to loopback. Save this as /etc/systemd/system/litellm.service:

[Unit]
Description=LiteLLM gateway
After=network-online.target postgresql.service ollama.service
Wants=network-online.target

[Service]
User=litellm
Group=litellm
EnvironmentFile=/etc/litellm/litellm.env
ExecStart=/opt/litellm/venv/bin/litellm --config /etc/litellm/config.yaml --host 127.0.0.1 --port 4000
Restart=on-failure
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Then sudo systemctl daemon-reload && sudo systemctl enable --now litellm (the Step 1 GPU caveat applies). The first start takes a minute while the proxy creates its database tables.

Check: curl -s 127.0.0.1:4000/health/liveliness answers that it is alive, and curl -s 127.0.0.1:4000/health/readiness reports the database as connected. ss -ltn | grep 4000 shows 127.0.0.1:4000 only. journalctl -u litellm -n 50 shows no Prisma errors.

Step 4: Give every app its own key

Virtual keys are LiteLLM’s per-app keys, each with an alias, models, rate limits and an optional budget, so you can see who sent what, limit each and revoke one alone. Keep the master key for administration: in the lab, one load test that fell back to it set off the "admin key used" alert, so even tests got their own key.

Mint keys on the server, in a root shell, so the master key never crosses the network:

set -a; . /etc/litellm/litellm.env; set +a
curl -s 127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"key_alias":"chat-ui","models":["general","coder"],"rpm_limit":60,"tpm_limit":200000}'
curl -s 127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"key_alias":"invoice-app","models":["general"],"rpm_limit":20,"tpm_limit":100000,"max_budget":1,"budget_duration":"30d"}'

Always list the models: an empty list means every model, including any cloud model added later. Give batch keys lower rate limits than interactive ones, so a runaway loop can’t fill the queue. max_budget is only a tripwire for cloud spend. Each key is shown once; store it only in that client’s configuration.

Check: each response contains a key. A call with the invoice key to coder is refused as not allowed for that key. With APP_KEY set to one app’s key, curl -s 127.0.0.1:4000/v1/models -H "Authorization: Bearer $APP_KEY" lists only that key’s models.

Step 5: Stop oversized prompts before they reach the GPU

By default Ollama doesn’t refuse a prompt bigger than the context window. It drops what doesn’t fit, logs a truncation warning and answers anyway, so an agent silently loses part of its own reasoning. You want a clear refusal before the GPU does any work.

The quick option is already in the config: with enable_pre_call_checks: true, LiteLLM won’t send a request over max_input_tokens. But its counter is generic, not the model’s tokenizer, and older releases ignored tool definitions: in the lab, a short message carrying a dozen tool schemas counted as 11 tokens but cost 3,540.

The better option is a small pre-call hook, saved as /etc/litellm/context_guard.py next to config.yaml (owner root, group litellm, mode 640):

import json

from fastapi import HTTPException
from litellm import token_counter
from litellm.integrations.custom_logger import CustomLogger

BUDGETS = {"general": 26000, "coder": 26000}
MARGIN = 1.15  # safety factor; calibrate per model against usage.prompt_tokens


class ContextGuard(CustomLogger):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        messages = data.get("messages")
        if not messages:
            return data
        n = token_counter(model="", messages=messages)
        if data.get("tools"):
            n += token_counter(model="", text=json.dumps(data["tools"]))
        n = int(n * MARGIN)
        budget = BUDGETS.get(data.get("model"), 13000)
        if n > budget:
            raise HTTPException(
                status_code=400,
                detail=f"context_length_exceeded: about {n} tokens is over the "
                f"{budget}-token budget; shorten the input",
            )
        return data


guard_instance = ContextGuard()

Register the instance (not the class) by adding this line to the existing litellm_settings block, then restart LiteLLM:

  callbacks: context_guard.guard_instance

The hook returns data to allow a request and raises an HTTPException to refuse it. Its generic counter under-counts digit-heavy text, where these models spend about one token per digit, so in production count with each model’s own tokenizer and chat template (Part 4) and compare with each response’s usage.prompt_tokens.

Check: with APP_KEY set, send a prompt of about 40,000 words, then a short message with many tool definitions.

python3 - <<'EOF'
import json
tool = lambda i: {"type": "function", "function": {"name": f"tool_{i}",
    "description": "Look up a record in the sample system. " * 40,
    "parameters": {"type": "object", "properties": {"id": {"type": "string"}}}}}
json.dump({"model": "general", "messages": [{"role": "user", "content": "word " * 40000}]},
          open("big.json", "w"))
json.dump({"model": "general", "messages": [{"role": "user", "content": "hi"}],
           "tools": [tool(i) for i in range(150)]}, open("tools.json", "w"))
EOF
for f in big.json tools.json; do
  curl -s -w '\nHTTP %{http_code}\n' 127.0.0.1:4000/v1/chat/completions \
    -H "Authorization: Bearer $APP_KEY" -H 'Content-Type: application/json' -d @"$f"
done

Both come back within milliseconds as HTTP 400 with context_length_exceeded in the error message, and journalctl -u ollama --since "5 min ago" shows no new POST /api/chat line. With only the built-in check, the first is refused; on older releases the second gets through.

Step 6: Connect a chat UI and your first app

Point Open WebUI at the gateway with its own key, never at Ollama, so every chat is attributed and traced. Because LiteLLM listens only on loopback, run Open WebUI with host networking, also on loopback, with its secrets in a mode-600 file:

sudo install -d -m 750 /srv/open-webui
printf '%s=%s\n' \
  OPENAI_API_KEY '<chat-ui-key-from-step-4>' \
  WEBUI_SECRET_KEY "$(openssl rand -hex 32)" \
  | sudo install -m 600 /dev/stdin /srv/open-webui/webui.env

Then /srv/open-webui/compose.yaml:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui@sha256:<digest>
    restart: unless-stopped
    network_mode: host
    env_file: [webui.env]
    environment:
      HOST: "127.0.0.1"
      PORT: "8080"
      ENABLE_OLLAMA_API: "false"
      OPENAI_API_BASE_URL: "http://127.0.0.1:4000/v1"
    volumes:
      - open-webui:/app/backend/data
volumes:
  open-webui:

Start it with sudo docker compose -f /srv/open-webui/compose.yaml up -d; the first account you create becomes the administrator. Open WebUI saves most of these settings to its database on first start, and that copy then wins, so make later changes in its admin settings.

For your own apps, anything that speaks the OpenAI API works, such as the openai Python package:

import os

from openai import OpenAI

key = os.environ["APP_KEY"]
client = OpenAI(base_url="http://127.0.0.1:4000/v1", api_key=key)
reply = client.chat.completions.create(
    model="general",
    messages=[{"role": "user", "content": "Write a two-line status update."}],
)
print(reply.choices[0].message.content)

Apps on other machines use the HTTPS address from Step 8.

Check: ss -ltn | grep 8080 shows 127.0.0.1:8080. Open WebUI lists only general and coder, and a chat gets an answer. In LiteLLM’s spend logs the chat appears under chat-ui and the script’s call under its own alias.

Step 7: See what is happening

Start with the gateway: journalctl -u litellm -f for live errors, the spend logs for every call’s key alias, model and tokens, and the admin UI at /ui for usage per key. Ollama’s log is a second witness: a POST /api/chat with no matching gateway request means something found a side door.

For per-request detail, add Langfuse, an open-source tool that records each call as a trace with prompt, answer, tokens and timing. Langfuse v3 needs Postgres, ClickHouse, Redis, S3-compatible blob storage and a worker; the official Docker Compose file brings them all. Before the first start:

• Replace every value marked # CHANGEME with output from openssl rand -hex 32 (the 64 hex characters the encryption key needs).
• Publish the web app on 127.0.0.1:3000:3000 and the object store on 127.0.0.1:9090:9000; the other services are already on loopback.
• Remove the Postgres port mapping (127.0.0.1:5432:5432): it clashes with the LiteLLM database, and the containers reach each other on their own network.
• Cap Docker’s log size in the daemon settings.

Then connect LiteLLM. Current releases need the Langfuse Python SDK v4 in the gateway’s venv, and that SDK needs Langfuse server 3.63.0 or newer; check both for your pinned release.

sudo -u litellm /opt/litellm/venv/bin/pip install 'langfuse>=4.7,<5'
printf '%s=%s\n' \
  LANGFUSE_PUBLIC_KEY '<public-key-from-langfuse>' \
  LANGFUSE_SECRET_KEY '<secret-key-from-langfuse>' \
  LANGFUSE_HOST 'http://127.0.0.1:3000' \
  | sudo tee -a /etc/litellm/litellm.env > /dev/null

Add the callbacks to the same litellm_settings block. In the lab they once sat in the wrong section, and LiteLLM ignored them silently. The block now reads:

litellm_settings:
  drop_params: true
  request_timeout: 600
  callbacks: context_guard.guard_instance
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

Restart LiteLLM. Then watch refusals per key (a run of context_length_exceeded means a client’s prompt has grown), time to first token, queue waits, unexpected models in ollama ps, free memory against your floor, and GPU temperature. In the lab, one ordinary test run took the sensors from 48 °C to 95 °C in four minutes.

Check: the Langfuse containers stay up for an hour without restarts, ss -ltn shows ports 3000 and 9090 on 127.0.0.1 only, and one chat through the gateway appears as a trace within a minute.

Step 8: Lock it down

Everything listens on loopback; add one controlled way in.

Remote access through TLS. Keys are bearer tokens, so never send them over plain HTTP. Put a TLS reverse proxy in front that passes only the API and the liveness check, keeping LiteLLM’s admin UI and management routes off the network. A Caddy example:

ai.example.com {
	@api path /v1/* /health/liveliness
	handle @api {
		reverse_proxy 127.0.0.1:4000
	}
	handle {
		respond 403
	}
}

chat.example.com {
	reverse_proxy 127.0.0.1:8080
}

An allow-list beats blocking admin paths one by one, because new releases add management routes. Open WebUI has its own logins. Or publish nothing and let a host firewall allow only the networks that need it (allow SSH first). Ports Docker publishes on all interfaces bypass host firewalls such as ufw, another reason to publish every container port on 127.0.0.1.

No silent cloud fallback. A LiteLLM fallback retries a failed or slow request on another model for every caller, because it belongs to the model, not the key. A local model with a short timeout and a cloud fallback quietly sends whole prompts to the cloud whenever the local slot is busy, and free cloud tiers may log or train on them. Let local models fall back only to local ones, enforced by a pre-commit check on fallbacks and default_fallbacks. Grant any cloud model to one key, for public data only.

Secrets out of reach. Environment files are root-only and stay out of Git (commit placeholder templates). Nothing a person, app or AI agent can read holds the master key.

Least privilege for AI coding agents. Treat one like a new contractor: an unprivileged account with no sudo and no docker group membership (which equals full control of the host), no access to secret files, and every change through Git and a review, never straight onto the server.

Write the data rule down: "Data never leaves the server, and every model call goes through the gateway." Loopback binding, per-key model lists and the fallback check enforce it.

Check: from another machine, ports 11434, 4000, 8080, 3000, 9090 and 5432 are refused or time out, while https://ai.example.com/health/liveliness answers. https://ai.example.com/ui and https://ai.example.com/key/list return 403. A commit that adds a local-to-cloud fallback is refused. As the agent’s account, cat /etc/litellm/litellm.env fails with "Permission denied".

Step 9: Keep it running

Pin everything: Ollama, LiteLLM, and every image by digest (image@sha256:...), never :latest. In the lab, a container recreated from a floating tag silently upgraded the model server days before the first freeze. docker compose config --images should list only @sha256: references.

Keep config in Git: the Ollama override, LiteLLM config and hook, units, compose files, env templates, Modelfiles and memory budget, with a pre-commit hook that refuses unpinned images, private keys and cloud fallbacks.

Change through a deploy, not by hand. A short script pulls the repository, stops on any hand edit to the live files, copies the new files in and restarts only what changed, reloading systemd only for unit changes.

Update in a controlled way. Read the release notes and upgrade one component at a time in a maintenance window; LiteLLM updates its database schema when the new version starts, so back up first. Smoke-test streamed and plain answers, tools, an oversized prompt and one call per app. Soak for a day, with the previous pin ready for rollback.

Back up, and test the restore. Back up nightly to a separate target, with an alert when a backup fails or doesn’t run: the LiteLLM database (pg_dump), Open WebUI’s data volume, Langfuse’s databases if you keep traces, and the config repository. Models can be rebuilt from the Modelfiles. Test restores into a scratch database; an untested backup is a hope.

Watch from outside. A hard freeze takes the server’s own monitoring with it; in the lab an on-board watchdog hung with the host. A separate device should check the health address every minute and alert after a few misses.

Watch temperatures. Long generations are sustained full load: record GPU and system temperatures every few seconds and check airflow before adding work.

Check: after a deploy the config checkout is clean, and a second deploy of the same commit changes nothing. Unplugging the server’s network raises the outside alert. A gateway pointed at a restored database lists your keys at /key/list.

Best-practice checklist

• Ollama on loopback, with slots, context, loaded models and keep-alive set; num_ctx fixed in tags; measured models plus a free floor fit.
• LiteLLM is the only way in, on loopback, unprivileged; only root can read the master key.
• Every app, UI and test has its own key, model list and rate limits; Open WebUI’s Ollama connection is off.
• Oversized prompts are refused, tool definitions included; local models use ollama_chat/.
• No local-to-cloud fallback, enforced at commit.
• Every call leaves a log row and a trace; refusals per key raise an alert.
• Remote access only through TLS with only the API allowed, or a firewall.
• AI coding agents run unprivileged, can’t read secrets and change things through Git.
• Pinned versions, config in Git, smoke-tested and soaked upgrades.
• Tested nightly backups, an outside heartbeat and recorded temperatures.

Troubleshooting

Where to go deeper

• Running Local AI in Production: The Complete Series Guide: all 11 parts, including routing, agents and workflows.
• How Do You Control Access to Local AI Models? Use a Gateway: Steps 3, 4 and 8 in depth.
• Why Does a Local AI Server Freeze? Sizing Memory for LLMs: Steps 1 and 2 in depth.
• What Happens When a Prompt Is Too Big for a Local LLM?: Step 5 in depth.
• How Do You Keep an AI Server Stable? GitOps for Local AI: Step 9 in depth.
• What Is Your Local AI Server Actually Doing? Observability Basics: Step 7 in depth.