What Is Your Local AI Server Actually Doing? Observability Basics (Part 7 of 11)

If nobody can see inside a private AI server, it freezes without warning. In our lab we built tracing, dashboards, an hourly guard and a live view of who uses which model, and learned to make sensors fail loudly. Steps to build it yourself are included.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The problem: a server we couldn’t see into

The lab server is the NVIDIA DGX Spark from Part 1, with its unified memory shared by the CPU and GPU. Running models on your own hardware gives you control, and also the whole job of watching them.

We found out how little we could see after the hard freezes described in Part 3. Reviewing the worst one, we found the warning signs had already been recorded: a per-minute memory sensor had written everything to a log file that nobody read. The hourly checker had no memory rules, the dashboard showed nothing about memory pressure, and the gateway would accept a prompt of any size. On top of that, tracing showed zero traces. For anyone running AI on their own hardware, that means outages arrive without warning and can’t be explained afterwards.

What we were aiming for

• Every model call recorded, with its prompt, answer, model, tokens and timing.
• One front page that shows health, heat and usage at a glance, on a phone as well as a desktop.
• Changes explained in plain language, so someone who didn’t build the server knows what happened and what to do.
• Sensors that fail loudly, so a broken reading can never look like a quiet one.

How we approached it

Every model call in the lab goes through one gateway, LiteLLM (Part 2). That made the gateway the natural tap: trace there, and read its spend log for usage, instead of instrumenting each client.

For everything else we preferred small open-source tools over a large monitoring platform, and wrote scripts only where nothing off the shelf fitted. Two rules applied throughout: monitoring tools must not be able to change anything, and every number on screen must say when it is unknown.

What we built

Tracing every model call

The gateway records each request in Langfuse, an open-source tracing tool. A trace is the full record of one request. Langfuse version 3 needs a web service, a background worker, Postgres, ClickHouse (an analytics database), Redis (a queue and cache) and S3-compatible object storage. Leave any one out and ingestion fails, which is what happened to us.

Metrics, logs and a front page

A lightweight metrics tool records host, container and GPU metrics with one-minute history, and alerts the lab’s alert chat when CPU, memory, GPU or disk stay high or its agent goes offline. A web log viewer gives live, searchable container logs, so finding out why something restarted doesn’t need an SSH session. A self-hosted start page shows health tiles and a header with GPU load, power and temperature (amber at 75 °C, red at 85 °C). An image-update checker shows newer tags but changes nothing; updates still go through the repo (Part 6).

None of these tools can change anything. They reach Docker through a socket proxy that allows read-only (GET) API calls only. Mounting the socket file read-only isn’t enough: that protects the file, not the API behind it, and a container that can write to the Docker API effectively controls the host. Everything except the gateway listens on the server itself (loopback), and the gateway and dashboards need authentication.

An hourly guard and a plain-English event log

Metrics say how busy the server is, not that someone changed it. A guard script runs hourly from cron, compares the server with its previous run and records an event for anything different: models on each Ollama instance against an allow-list built from the gateway config in git, callers that bypassed the gateway, admin-key use and new keys, drift from the last deployed commit, new containers or ports, and backups, reboots and GPU repairs. Each event is info, warn or alert, and alerts go to the alert chat, de-duplicated. The guard only reports, with one exception: an unapproved model is renamed after a grace period, never deleted.

Raw events were hard to act on, so an event log page colour-codes them (red: act now, amber: have a look, blue: expected), folds repeats into one row ("×12 in the last day"), and opens each one to what it means, what to do, the evidence and the runbook: "Files on the server differ from the commit last deployed — someone edited the server by hand. The next deploy of this target refuses to run until it is resolved."

Who is using which model, and who is waiting

The gateway’s spend log records each request’s key, the model asked for, where it ran, the time and the tokens. A page republished every minute shows what each Ollama instance has loaded, one card per caller with each model asked for and where it ran (green on the server, amber in the cloud, red failed), and a caller-by-model grid for the last 5 minutes, hour and day.

The main Ollama instance serves one request per model at a time. In the LiteLLM release we ran, the gateway’s max_parallel_requests cap made a second caller wait, so a queue showed only as latency. Current releases reject a request over the cap at once with a 429; pin your version, let Ollama queue instead, or handle 429 in clients. The spend log only learns about a request once it ends, so a small gateway hook records each request as it starts and finishes, and the page shows one lane per model: the request being answered, then each caller waiting and for how long.

A heat recorder

A thermal-watch script runs every minute from cron (not a systemd timer, because on this server a systemd reload strips GPU access from containers). It samples the hottest thermal zone and the GPU every 2 seconds, writes a summary line per minute plus a HOT line for every sample at or above 85 °C, and fsyncs every write, because here a shutdown is a sudden power-off and the last lines matter most.

Challenges we hit

Tracing had never worked

There were two causes, and the first hid the second. LiteLLM only registers logging callbacks from litellm_settings, and ours sat under general_settings, where they were silently ignored. And only Langfuse’s web service and databases were running: health and auth checks returned 200, while POST /api/public/ingestion returned 500.

Fixing the first made the second worse. With callbacks active, every batch of traces crashed the web service, and the next morning the server froze for the fourth time; Langfuse had restarted 72 times on that boot, the last two seconds before the freeze. We removed the callbacks that day, finished the stack with Redis, an object store and the worker that evening, and turned callbacks back on only after a test trace landed in ClickHouse within 45 seconds.

Two days later, ClickHouse’s internal logs had grown to 41 GB, mostly a query profiler log nobody used.

The memory sensor that couldn’t see

The day before the worst freeze, we extended the per-minute memory sensor to record how much context each loaded model had reserved; that reservation, the KV cache holding the model’s working state, caused the freezes. The sensor asked Ollama by running curl inside the container, and the Ollama image doesn’t include curl. For more than a day it reported a maximum context of 0 and a model list of []. "Nothing loaded" meant a probe that had never run.

Every layer swallowed the failure: a shell heredoc ending in 2>/dev/null || true, with a Python except Exception: pass inside. We rewrote it to query Ollama’s API from the host, tag every snapshot probe: ok | partial | failed, publish null and "unknown" instead of [] and 0, and raise "capacity sensor blind" after three failed minutes. A model that has fallen back to CPU now shows as where: cpu, not as 0.0 GB of GPU memory. We tested each loud path by injecting the failure on purpose.

The guard read "no answer" as "nothing there"

When an Ollama instance failed to answer ollama list, the guard saw an empty list, logged "model removed" for every tag, silenced that instance’s alerts and reset its rename clock. Now a failed listing raises one warning and the last known tags carry forward, and an instance that disappears keeps being reported until it escalates to an alert.

Dashboards that misled

The dashboard took four redesigns in two weeks. It kept serving a stale prerendered page across restarts; the deploy script’s revalidate call went to an address the dashboard rejected, through curl without -f, so the failure read as success. One counter said "Alerts 50 open" when it meant 50 hourly alert events. And a pasted tile left icon: twice in one mapping: the dashboard’s YAML parser rejected it for about ten minutes, while our pre-commit check, using PyYAML, accepted duplicates silently.

Power-offs a one-minute average couldn’t explain

Two days after the worst freeze, the server powered itself off, and again the next day, with no shutdown record, no kernel error and plenty of free memory. The metrics tool’s one-minute readings just before were 88.0 °C and 87.7 °C, against an idle of about 50 °C. That is why we built the heat recorder.

The result

Tracing works. A test stock analysis (Part 8) is one trace with one entry per model call, and its 216,561 tokens matched the gateway’s records exactly. With the profiler off and 7- to 30-day retention, ClickHouse’s data directory went from 41 GB to 366 MB.

The queue is visible. Over seven days there were 1,342 requests and 52 overlaps, with a median wait of about 18 seconds and a 90th percentile of 58. Heat is measurable too. After the restart, one ordinary test analysis took the thermal zones from 48 °C to 95 °C in four minutes and back to 50 °C within one. That came after the power-offs, so heat is the likely cause, not a proven one. Better ventilation brought idle to about 37 °C.

Unknown now says unknown. A failed read of model budgets keeps the last known numbers marked stale_since, and while a gateway restart was pending the dashboard showed two models as "NOT POLICED", which was the true state.

The costs are real. Tracing alone added Redis, an object store, a worker and ClickHouse housekeeping, and noise needs a budget: one proposed warning would have added about 120 events a day. Still to do: thresholds on the queue view, a gateway "thermal gate" that holds batch callers while the server is hot, and replacing the object store behind Langfuse, as the upstream project is no longer maintained.

Do it yourself: observability in an afternoon

What you need: a Linux server running Ollama and LiteLLM (Part 2’s steps), Docker Compose, an NVIDIA GPU for the nvidia-smi commands (use your vendor’s tool otherwise), and about half a day.

1. Deploy Langfuse v3 with all its parts from the project’s official Docker Compose file: web, worker, Postgres, ClickHouse, Redis and S3-compatible storage. Before the first up, replace every secret marked # CHANGEME with a long random value (openssl rand -hex 32), change the published ports to loopback, and cap Docker’s log size (max-size and max-file in the daemon’s logging options):

   # in the official compose file
   langfuse-web:
     ports: ["127.0.0.1:3000:3000"]
   minio:
     ports: ["127.0.0.1:9090:9000"]
   

   Check: all six containers stay up for an hour with no restarts, ss -tlnp shows ports 3000 and 9090 on 127.0.0.1 only, and the web UI lets you create a project and API keys.

2. Turn on LiteLLM callbacks in the right section. Current LiteLLM needs the Langfuse Python SDK v4 in its environment, which in turn needs Langfuse server 3.63.0 or newer:

   /opt/litellm/bin/pip install "langfuse>=4.7,<5"
   

   Give the gateway LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST (your Langfuse URL) as environment variables, then add:

   litellm_settings:
     success_callback: ["langfuse"]
     failure_callback: ["langfuse"]
   

   Check: send one chat request through the gateway, and a trace appears in Langfuse within a minute.

3. Cap ClickHouse’s own logs with a file such as clickhouse/config.d/system-logs.xml, mounted into the ClickHouse container at /etc/clickhouse-server/config.d/system-logs.xml (the official compose file doesn’t do this for you):

   <clickhouse>
     <trace_log remove="remove"/>
     <query_log><ttl>event_date + INTERVAL 14 DAY DELETE</ttl></query_log>
     <text_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></text_log>
   </clickhouse>
   

   Check: trace data grows by design, so measure the system tables instead. Over a week this stays flat:

   SELECT table, formatReadableSize(sum(bytes_on_disk))
   FROM system.parts WHERE database = 'system' GROUP BY table;
   

4. Put a read-only proxy in front of the Docker socket and point monitoring tools at tcp://dockerproxy:2375 instead of the socket. We used the widely used Tecnativa proxy, pinned by digest:

   dockerproxy:
     image: ghcr.io/tecnativa/docker-socket-proxy@sha256:<digest>
     environment: { CONTAINERS: "1", IMAGES: "1", EVENTS: "1", INFO: "1", POST: "0" }
     volumes: ["/var/run/docker.sock:/var/run/docker.sock"]
   

   Never publish its port to the network; for an agent using network_mode: host, publish it on 127.0.0.1:2375:2375 only. Check: from a container on the same network, a POST to /containers/create on the proxy returns 403, while a GET to /containers/json returns a list.

5. Add metrics and one front page. Run a metrics tool with GPU support (any tool with an NVIDIA or ROCm collector) and a dashboard (any self-hosted start page) with one tile per container, and route alerts to one chat. Check: the GPU graph moves when you run a model, and stopping the metrics agent sends an "offline" alert.

6. Record heat every 2 seconds. Save this as /usr/local/sbin/thermal-watch; it stops after 56 seconds, so a slow minute can’t overlap the next run:

   #!/bin/sh
   end=$(( $(date +%s) + 56 ))
   while [ "$(date +%s)" -lt "$end" ]; do
     z=$(cat /sys/class/thermal/thermal_zone*/temp | sort -n | tail -1)
     g=$(nvidia-smi --query-gpu=temperature.gpu,power.draw,utilization.gpu --format=csv,noheader,nounits)
     echo "$(date -u +%FT%TZ) zone_c=$((z/1000)) gpu=$g" >> /var/log/thermal.log
     sync /var/log/thermal.log; sleep 2
   done
   

   Install it in root’s crontab as * * * * * flock -n /run/thermal.lock /usr/local/sbin/thermal-watch. On some unified-memory systems power.draw reads [N/A]; that’s expected. Check: the log grows by about 25 to 28 lines a minute with no missing minutes, and a heavy prompt shows the temperature climbing.

7. Make probes fail loudly. Query Ollama’s /api/ps from the host and write failed when it doesn’t answer:

   if out=$(curl -fsS --max-time 5 "$OLLAMA_URL/api/ps"); then echo "probe=ok $out"; else echo "probe=failed models=null"; fi
   

   Check: stop Ollama, and the probe prints probe=failed, never an empty model list.

8. Check config with a strict parser. In your pre-commit hook, run only yamllint‘s duplicate-key rule, so its style rules don’t flag your existing files:

   yamllint -d '{rules: {key-duplicates: enable}}' dashboards/*.yaml
   

   Check: a test file with the same key twice fails with a "duplication of key" error.

Key takeaways

• Check that tracing actually records traces. "Configured" and "working" were a long way apart.
• Never let a failed reading look like a quiet one. Publish unknown, null, failed and stale_since, never a plausible zero.
• Put the number that predicts trouble on screen. On a unified-memory server that’s reserved context and memory pressure, not GPU utilisation.
• Record at the resolution of the problem. Heat spikes last minutes, so sample every 2 seconds and fsync each write.
• Give monitoring tools genuinely read-only access through a GET-only socket proxy, not a read-only mount of the socket file.
• Explain every event in plain language, and test config with the same parser production uses.