benchmarked two of them linked together<\/a>; this guide needs only one, and any Linux server with enough GPU memory works.<\/p>\n\n\n\nSoftware.<\/strong><\/p>\n\n\n\n\n\n- A current long-term-support Debian or Ubuntu release.<\/li>\n\n\n
- A working NVIDIA driver (
nvidia-smi<\/code> lists the GPU) and the NVIDIA Container Toolkit.<\/li>\n\n\n- Docker Engine with the Compose v2 plugin (
docker compose<\/code>), for Open WebUI and Langfuse.<\/li>\n\n\n- Python 3.11 or later, for LiteLLM.<\/li>\n\n\n
- For remote access, a domain name for TLS, or a firewall.<\/li>\n\n<\/ul>\n\n\n\n
Time.<\/strong> Allow a day for Steps 1 to 6, plus half a day for tracing and lockdown.<\/p>\n\n\n\n
\n\n\n\nStep 1: Install Ollama and keep it private<\/h2>\n\n\n\n
The install script creates an ollama<\/code> service account and a systemd service; OLLAMA_VERSION<\/code> pins the release, so upgrades happen when you choose.<\/p>\n\n\n\ncurl -fsSL https:\/\/ollama.com\/install.sh | OLLAMA_VERSION=<version> sh\nsystemctl status ollama --no-pager\n<\/code><\/pre>\n\n\n\nSet memory settings explicitly; defaults change between releases. Current versions pick the default context from the GPU memory they detect: 4k below 24 GB, 32k up to 48 GB and 256k above that. Run sudo systemctl edit ollama<\/code> and add:<\/p>\n\n\n\n[Service]\nEnvironment="OLLAMA_HOST=127.0.0.1:11434"\nEnvironment="OLLAMA_NUM_PARALLEL=1"\nEnvironment="OLLAMA_CONTEXT_LENGTH=32768"\nEnvironment="OLLAMA_MAX_LOADED_MODELS=2"\nEnvironment="OLLAMA_KEEP_ALIVE=30m"\n<\/code><\/pre>\n\n\n\nThen sudo systemctl restart ollama<\/code>. Why each one matters:<\/p>\n\n\n\n\n\nOLLAMA_HOST<\/code> pins the listener to IPv4 loopback (the default, made explicit; a wildcard address exposes every model).<\/li>\n\n\nOLLAMA_NUM_PARALLEL<\/code> is request slots per model. Each slot gets the full context<\/strong>, so memory grows with context \u00d7 slots. With one slot a second caller waits; in the lab the median wait was 18 seconds.<\/li>\n\n\nOLLAMA_CONTEXT_LENGTH<\/code> is the context window, shared by prompt and answer.<\/li>\n\n\nOLLAMA_MAX_LOADED_MODELS<\/code> (default three per GPU) stops a stray request loading a third model.<\/li>\n\n\nOLLAMA_KEEP_ALIVE<\/code> (default five minutes) is how long an idle model stays loaded.<\/li>\n\n<\/ul>\n\n\n\nCaveat: systemctl edit<\/code>, like any systemd reload (enable<\/code> and disable<\/code> included), can strip GPU access from running NVIDIA containers until they restart. Do it in a maintenance window and check each with docker exec <container> nvidia-smi -L<\/code>.<\/p>\n\n\n\nOr in a container.<\/strong> Pin the image by digest, publish on loopback only and reserve the GPU, or it quietly runs on the CPU. Then prefix the ollama<\/code> commands below with docker compose exec ollama<\/code>.<\/p>\n\n\n\nservices:\n ollama:\n image: ollama\/ollama@sha256:<digest>\n restart: unless-stopped\n ports:\n - "127.0.0.1:11434:11434"\n volumes:\n - ollama:\/root\/.ollama\n environment:\n OLLAMA_NUM_PARALLEL: "1"\n OLLAMA_CONTEXT_LENGTH: "32768"\n OLLAMA_MAX_LOADED_MODELS: "2"\n OLLAMA_KEEP_ALIVE: "30m"\n deploy:\n resources:\n reservations:\n devices:\n - driver: nvidia\n count: all\n capabilities: [gpu]\nvolumes:\n ollama:\n<\/code><\/pre>\n\n\n\nCheck:<\/strong> ss -ltn | grep 11434<\/code> shows 127.0.0.1:11434<\/code> and nothing else. systemctl show ollama --property=Environment<\/code> lists all five settings. From a second machine, curl -m 5 http:\/\/<server-address>:11434<\/code> is refused or times out.<\/p>\n\n\n\n
\n\n\n\nStep 2: Choose models and measure them<\/h2>\n\n\n\n
Start with two models: a general one for questions and drafting, and a coding one for code and agents. Here, a 12B Gemma model and a 30B mixture-of-experts Qwen coder, which activates only a few of its sub-networks per token, so it answers about as fast as a much smaller model. Check the Ollama library for current tags.<\/p>\n\n\n\n
A model’s own context setting beats the server-wide one: in the lab, a 65,536-token value in a model’s settings quietly overrode the server limit and contributed to the freezes. So build tags with the context fixed:<\/p>\n\n\n\n
ollama pull gemma3:12b\nollama pull qwen3-coder:30b\nprintf 'FROM gemma3:12b\\nPARAMETER num_ctx 32768\\n' > general.Modelfile\nprintf 'FROM qwen3-coder:30b\\nPARAMETER num_ctx 32768\\n' > coder.Modelfile\nollama create general-32k -f general.Modelfile\nollama create coder-32k -f coder.Modelfile\n<\/code><\/pre>\n\n\n\nSend each model one request and read what it holds:<\/p>\n\n\n\n
for m in general-32k coder-32k; do\n curl -s 127.0.0.1:11434\/api\/generate -d "{\\"model\\":\\"$m\\",\\"prompt\\":\\"Hi\\",\\"stream\\":false}"\ndone\nollama ps\ncurl -s 127.0.0.1:11434\/api\/ps | python3 -m json.tool\n<\/code><\/pre>\n\n\n\nWrite your memory budget down in Git: measured resident models plus a fixed free floor must fit. In the lab the resident pair measured 37.8 GB, and we keep at least 40 GB of the shared pool free. On a discrete GPU, leave roughly 10 per cent of VRAM spare.<\/p>\n\n\n\n
Check:<\/strong> ollama ps<\/code> shows 100% GPU<\/code> under PROCESSOR and 32768 under CONTEXT for both models (older versions lack that column; read context_length<\/code> from \/api\/ps<\/code>). ollama show coder-32k --parameters<\/code> lists num_ctx 32768<\/code>. If PROCESSOR shows any CPU share, fix the driver or the container’s GPU access before you go on.<\/p>\n\n\n\n
\n\n\n\nStep 3: Put LiteLLM in front as the only door<\/h2>\n\n\n\n
LiteLLM is an open-source gateway that speaks the OpenAI API. Identity, model access, budgets and tracing all live there, so any path that skips it skips all of them. It keeps keys and spend logs in Postgres.<\/p>\n\n\n\n
First, the database, a service account and the secrets. LiteLLM’s admin key, the master key, must start with sk-<\/code>. The salt key encrypts credentials stored in the database and must never change once set. Systemd reads the environment file as root before starting the service, so it can be root-only (mode 600).<\/p>\n\n\n\nsudo apt install -y postgresql python3-venv\nsudo useradd --system --create-home --home-dir \/opt\/litellm --shell \/usr\/sbin\/nologin litellm\nsudo -u postgres createuser -P litellm\nsudo -u postgres createdb -O litellm litellm\nsudo install -d -m 750 -o root -g litellm \/etc\/litellm\nprintf '%s=%s\\n' \\\n LITELLM_MASTER_KEY "sk-$(openssl rand -hex 32)" \\\n LITELLM_SALT_KEY "sk-$(openssl rand -hex 32)" \\\n DATABASE_URL 'postgresql:\/\/litellm:<db-password>@127.0.0.1:5432\/litellm' \\\n | sudo install -m 600 -o root -g root \/dev\/stdin \/etc\/litellm\/litellm.env\n<\/code><\/pre>\n\n\n\nInstall a pinned version into a virtual environment owned by the service account, because the proxy generates its Prisma database client on first start. The extra_proxy<\/code> extra brings that client; without it the proxy fails at start-up.<\/p>\n\n\n\nsudo -u litellm python3 -m venv \/opt\/litellm\/venv\nsudo -u litellm \/opt\/litellm\/venv\/bin\/pip install 'litellm[proxy,extra_proxy]==<version>'\n<\/code><\/pre>\n\n\n\nWrite \/etc\/litellm\/config.yaml<\/code> (owner root, group litellm<\/code>, mode 640):<\/p>\n\n\n\nmodel_list:\n - model_name: general\n litellm_params:\n model: ollama_chat\/general-32k\n api_base: http:\/\/127.0.0.1:11434\n model_info:\n max_input_tokens: 26000\n - model_name: coder\n litellm_params:\n model: ollama_chat\/coder-32k\n api_base: http:\/\/127.0.0.1:11434\n model_info:\n max_input_tokens: 26000\nrouter_settings:\n enable_pre_call_checks: true\nlitellm_settings:\n drop_params: true\n request_timeout: 600\ngeneral_settings:\n master_key: os.environ\/LITELLM_MASTER_KEY\n database_url: os.environ\/DATABASE_URL\n<\/code><\/pre>\n\n\n\nThe deliberate choices:<\/p>\n\n\n\n
\n\nollama_chat\/<\/code> uses Ollama’s chat endpoint, which returns native tool calls; with ollama\/<\/code>, agents get text instead.<\/li>\n\n\nmax_input_tokens: 26000<\/code> leaves 6,768 tokens of the 32k window for the answer. On its own it is only metadata; enable_pre_call_checks<\/code> and Step 5 make it count.<\/li>\n\n\n- No
max_parallel_requests<\/code>: current releases reject a request over that cap at once with a 429<\/code> instead of queueing it. With one slot, Ollama already queues the second caller.<\/li>\n\n\n- No
fallbacks<\/code> or default_fallbacks<\/code> (see Step 8).<\/li>\n\n\n- One
litellm_settings<\/code> block only. Later steps add to it; a second litellm_settings:<\/code> line silently replaces the first.<\/li>\n\n<\/ul>\n\n\n\nRun it as a systemd service bound to loopback. Save this as \/etc\/systemd\/system\/litellm.service<\/code>:<\/p>\n\n\n\n[Unit]\nDescription=LiteLLM gateway\nAfter=network-online.target postgresql.service ollama.service\nWants=network-online.target\n\n[Service]\nUser=litellm\nGroup=litellm\nEnvironmentFile=\/etc\/litellm\/litellm.env\nExecStart=\/opt\/litellm\/venv\/bin\/litellm --config \/etc\/litellm\/config.yaml --host 127.0.0.1 --port 4000\nRestart=on-failure\nNoNewPrivileges=true\n\n[Install]\nWantedBy=multi-user.target\n<\/code><\/pre>\n\n\n\nThen sudo systemctl daemon-reload && sudo systemctl enable --now litellm<\/code> (the Step 1 GPU caveat applies). The first start takes a minute while the proxy creates its database tables.<\/p>\n\n\n\nCheck:<\/strong> curl -s 127.0.0.1:4000\/health\/liveliness<\/code> answers that it is alive, and curl -s 127.0.0.1:4000\/health\/readiness<\/code> reports the database as connected. ss -ltn | grep 4000<\/code> shows 127.0.0.1:4000<\/code> only. journalctl -u litellm -n 50<\/code> shows no Prisma errors.<\/p>\n\n\n\n
\n\n\n\nStep 4: Give every app its own key<\/h2>\n\n\n\n
Virtual keys<\/strong> are LiteLLM’s per-app keys, each with an alias, models, rate limits and an optional budget, so you can see who sent what, limit each and revoke one alone. Keep the master key for administration: in the lab, one load test that fell back to it set off the "admin key used" alert, so even tests got their own key.<\/p>\n\n\n\nMint keys on the server, in a root shell, so the master key never crosses the network:<\/p>\n\n\n\n
set -a; . \/etc\/litellm\/litellm.env; set +a\ncurl -s 127.0.0.1:4000\/key\/generate \\\n -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application\/json' \\\n -d '{"key_alias":"chat-ui","models":["general","coder"],"rpm_limit":60,"tpm_limit":200000}'\ncurl -s 127.0.0.1:4000\/key\/generate \\\n -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application\/json' \\\n -d '{"key_alias":"invoice-app","models":["general"],"rpm_limit":20,"tpm_limit":100000,"max_budget":1,"budget_duration":"30d"}'\n<\/code><\/pre>\n\n\n\nAlways list the models: an empty list means every model, including any cloud model added later. Give batch keys lower rate limits than interactive ones, so a runaway loop can’t fill the queue. max_budget<\/code> is only a tripwire for cloud spend. Each key is shown once; store it only in that client’s configuration.<\/p>\n\n\n\nCheck:<\/strong> each response contains a key<\/code>. A call with the invoice key to coder<\/code> is refused as not allowed for that key. With APP_KEY<\/code> set to one app’s key, curl -s 127.0.0.1:4000\/v1\/models -H "Authorization: Bearer $APP_KEY"<\/code> lists only that key’s models.<\/p>\n\n\n\n\n\n
Thinking about private AI for your organisation?<\/strong> We design, build and secure on-premises AI platforms for Australian organisations, with the guardrails this series describes built in from the start.<\/p>\n\n\n\n