How Do You Keep an AI Server Stable? GitOps for Local AI (Part 6 of 11)

A silent image upgrade, a pruned container image and a backup that had quietly stopped pushed us to run our lab AI server from a git repo. Here is what we built, the GPU trap we fell into, how to set up AI coding agents safely, and the steps to build the same thing yourself.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The problem: a server nobody could explain

The lab server is the NVIDIA DGX Spark from Part 1. A few weeks into testing it ran a couple of dozen containers, over a dozen models and four Docker Compose stacks, all set up by hand.

That started to cost us. There had been three hard freezes in two weeks (Part 3 covers them). Shortly before them, the model server had been silently upgraded: someone recreated it from a :latest image in the container-management UI, which pulled a newer Ollama release, and the freezes followed. The same click pruned old images, including the one the OCR service needed. Nothing had a history or a rollback, and the off-box backup had stopped without anyone noticing. For a business, that’s the difference between a quick rollback and a day of guesswork.

What we were aiming for

• Lose nothing. No data, model or working configuration should disappear because of a change.
• Make every change reviewable and reversible, with a record of what changed, why and how to undo it.
• Let anyone with the repo, or any AI session working from it, understand the server without being able to break it by accident.
• Allow no surprises from outside the repo: no automatic upgrades and no silent backup failures.

How we approached it

On one server, a cluster orchestrator would add more moving parts than it removes. We needed history, review and rollback, which git already gives you: a repo as the single source of truth, a deploy script as the only way in, and nightly snapshots that expose any difference between the two.

The ground rules: remove nothing, only disable; keep the container UI view-only; recreate at most one service at a time; and require a backup gate, a verified off-box archive less than 24 hours old, before anything that could lose data.

First we stopped the bleeding with small, logged changes. We pinned Ollama to the version already running, raised its keep-alive from 30 seconds to 5 minutes to cut model churn, and made the GPU-restore script check every GPU container, not only Ollama, every minute from cron. Why GPU access needs restoring at all is the first challenge below.

What we built

   git repo (source of truth)
        │  pull
        ▼
   server checkout ── deploy.sh <stack> ──► sync files into place ──► compose up (that stack only)
        │                                                         └─► verify GPU access
        └── nightly state snapshot ──► state/*.txt ──► commit   (drift becomes visible)

A deploy script as the only way in

On the first day we copied every stack and the host configuration into the repo and hash-checked each file against the live copy. Adopting them recreated nothing. From then on deploy.sh was the only way configuration reaches the server. For one target it:

1. Pulls the repo, fast-forward only, failing loudly if the server’s copy has diverged.
2. Checks for drift against the commit last deployed to that target, not the latest commit. A normal repo change shows as pending. A live file that differs from what was deployed means a hand edit, and the deploy stops. You either copy the live file into the repo or rerun with --accept-drift, which first saves the differing files to a pre-change folder.
3. Plans with a compose dry run and refuses to recreate a container without --allow-recreate ("this is an agreed window"). --no-recreate is for config-only changes, and --service <name> limits a recreate to one service.
4. Syncs files, never touching data directories, .env files or model weights, and refuses to run if a variable in the template is missing from the live .env.
5. Verifies GPU access, then logs one line to a deploy log committed to the repo.

Pinned images and visible updates

Every image is pinned, and a check refuses floating tags. Before the pinning window we pulled every registry tag and confirmed it matched the image ID already running, so pinning couldn’t quietly upgrade anything, then recreated containers one at a time. Updates are visible but never automatic: an update checker reaches Docker through a socket proxy that allows read-only (GET) calls only, and can only propose a one-line tag bump for review.

Records, snapshots and pre-commit checks

A rules file at the repo root loads automatically in every AI coding session, each rule next to the incident that caused it. INVENTORY.md says what should run and why. CHANGELOG.md is append-only: what, why, who and how to roll back.

A nightly snapshot writes what is actually running into state/ (containers with digests, models, ports, config-file hashes and workflows) and commits only on a real change. Exported test workflows must hold no literal secret, or the export fails.

Every commit also runs six checks, each for a mistake we had already made once:

When a model was linked to the freezes, we renamed its tags to …-disabled rather than deleting them, so bringing it back was a copy command, not a re-download. Retired services go into a compose profile called disabled.

Backups that fail loudly

We rebuilt the backup so it can restore onto a fresh machine without copying anything you can download again. Each archive carries a restore manifest: the NVIDIA driver and Container Toolkit versions, the Docker version, the package list and every Ollama tag’s Modelfile, so custom fixed-context tags can be rebuilt exactly. Our recommendation is the same for any server like this: run it nightly to a separate target, alert on any failed run, check integrity on a schedule, and test a restore before you rely on it.

Rules for AI coding agents

AI coding agents draft changes well, but they take the shortest path, which on a live server means editing files in place. So the safe path has to be the only one:

• Scoped, unprivileged, reviewable access. Agents use an unprivileged account that can read the repo and propose changes; changes reach the server only through the deploy script, after review. No root and no shared admin accounts.
• No secrets in files the agent’s account can read. If an agent needs to test a model, it gets its own narrowly scoped gateway key.
• Immutable live files. The gateway config, its guard and every compose file carry chattr +i; deploy.sh lifts it only for its own sync and restores it on every exit path. This catches accidents; the access model is what enforces the rule.
• A server that explains its rules. A login banner and an AGENTS.md wherever an agent might start say the server is managed from a repo, and if you can’t use the repo, stop and ask.
• Alerts that mean what they say. Admin-key calls are told apart from missing or malformed keys, and any config change inside a Docker volume, where deploy.sh can’t see it, raises an alert.

Challenges we hit

A systemd reload takes the GPU away

On this server (cgroup v2, the systemd cgroup driver and the NVIDIA container runtime), systemctl daemon-reload removes GPU device permission from every running GPU container. So do daemon-reexec, enable, disable and systemctl edit, which reloads for you. It’s a known issue in the NVIDIA Container Toolkit. A process that already holds the GPU keeps working, but Ollama starts a new process for each model it loads, that process can’t reach the GPU, and the model quietly runs on the CPU. The symptoms are models at "100% CPU", a load average of 10 to 20, and Failed to initialize NVML from nvidia-smi inside the container. Enabling a systemd timer for backups broke Ollama and the OCR API within 60 seconds.

So we schedule with cron, and after any reload the GPU restore tests GPU access in each GPU container and restarts the ones that lost it, even mid-request. Early on, deploy.sh host reloaded systemd on every host deploy. One afternoon in week two, eight script-only host deploys restarted Ollama eight times, and one restart turned a test caller’s request into an Ollama 500 error; the gateway retried, and the answer took 28 seconds instead of about 20. The deploy now reloads systemd only when a file under the systemd folders changed.

The OS can trigger this too. In week three an automatic package refresh ran its own daemon-reload, and the autoheal restarted three services. Automatic refreshes, security updates included, now wait for a maintenance window.

Tools that reported success when they hadn’t checked

The GPU restore reported success whenever its cron copy held the lock. Now it says "checked nothing", and the deploy verifies GPU access itself.

Deploys that interleaved or didn’t take effect

Parallel deploys interleaved, so the script now takes a lock. And a config-only deploy didn’t always apply: when a file is bind-mounted as a single file, the sync replaces it with a rename (a new inode) and the container keeps seeing the old content. Those containers need a restart, in the right order.

Drift alerts don’t stop anyone

Drift detection notices a hand edit after the fact but doesn’t prevent one, and an AI agent never reads the ops chat. So the agent rules rely on permissions, not alerts.

The result

We tested the loop with a deliberate hand edit: it showed up in the next state commit, the next deploy refused to run, and --accept-drift --no-recreate put it right. Pinning every image changed nothing that was running. Leaving model weights out made the backup archive about a third smaller.

Three days in, we updated the host’s OS, kernel, NVIDIA driver and Container Toolkit, mainly for better memory handling, and treated it like any change: a fresh backup, the GPU idle, the before-state saved and the old kernel kept as a fallback. Afterwards every GPU container had GPU access, dry runs showed no drift, and it soaked for three days before the next change.

The cost is speed: a one-line change is now a commit, six checks, a review, a dry run and a deploy, which is still far cheaper than recovering from a freeze. Two gaps remain: configuration inside Docker volumes is invisible to deploy.sh, and GPU injection should move to the toolkit’s Container Device Interface (CDI) mode, which avoids the reload problem instead of repairing it.

Do it yourself: a GitOps repo for one AI server

What you need: one Linux server running Docker Compose 2.17 or later (for --dry-run), with the NVIDIA Container Toolkit if you use NVIDIA GPUs, a private Git remote, and about an afternoon.

1. Lay out the repo, one folder per stack plus host config and state:

   stacks/<stack>/compose.yaml   stacks/<stack>/.env.template
   host/cron.d/   host/systemd/   scripts/deploy.sh   state/   AGENTS.md   CHANGELOG.md
   

   Check: after copying the live files in, diff -r against the live folder shows nothing.

2. Pin every image by digest, using what is already running. Record each container’s image ID first, so you can prove the pin changed nothing:

   docker inspect --format '{{.Image}}' ollama > before-ollama.txt
   docker image inspect --format '{{index .RepoDigests 0}}' "ollama/ollama:$TAG"   # TAG = the tag running now
   # compose.yaml →  image: ollama/ollama@sha256:<digest>
   

   Changing image: is a config change, so Compose will recreate the container; do it one service at a time in a maintenance window. Check: docker compose config --images lists only @sha256: references, and after the recreate docker inspect --format '{{.Image}}' ollama prints the same image ID as before-ollama.txt.

3. Record what was deployed. At the end of each deploy, write the commit to state/deployed-<stack>, and check drift against it:

   S=mystack   # the stack's folder name
   git show "$(cat "state/deployed-$S"):stacks/$S/compose.yaml" | diff -q - "/srv/$S/compose.yaml"
   

   Check: hand-edit the live file, and the deploy stops and names it.

4. Deploy one stack at a time: git pull --ff-only, the drift check, docker compose --dry-run up -d, the sync, then docker compose up -d --no-recreate unless a recreate was explicitly allowed. Wrap it in flock so two deploys can’t interleave. Check: running the same deploy twice reports nothing to change.

5. Reload systemd only when units changed, then verify the GPU (NVIDIA example):

   git diff --name-only "$OLD" "$NEW" | grep -q '^host/systemd/' && sudo systemctl daemon-reload
   docker exec <gpu-container> nvidia-smi -L
   

   Check: nvidia-smi -L lists the GPU in every GPU container after a host deploy. Failed to initialize NVML means the reload took GPU access away; restart that container.

6. Snapshot nightly from cron. Write docker ps --format '{{.Names}} {{.Image}}', docker exec ollama ollama list and sha256sum of the live config into state/, and commit only if git status --porcelain state/ prints something. Cron has a minimal environment, so set PATH in the script, commit with git -c user.name=state-bot -c user.email=<address>, and push with a deploy key that can write only to this repo. Check: a quiet night produces no commit; a hand edit shows up the next morning.

7. Add a pre-commit hook (.git/hooks/pre-commit, executable) that refuses private keys and any image not pinned by digest, and lets clean commits through:

   #!/bin/sh
   added=$(git diff --cached -U0 | grep '^+')
   if printf '%s\n' "$added" | grep -Eq 'BEGIN [A-Z ]*PRIVATE KEY'; then
     echo "refused: private key in commit" >&2; exit 1
   fi
   if printf '%s\n' "$added" | grep -E '^\+[[:space:]]*image:' | grep -vq '@sha256:'; then
     echo "refused: image not pinned by digest" >&2; exit 1
   fi
   exit 0
   

   Check: a compose file with image: redis:7 or a bare image: nginx is rejected, and an ordinary commit with no image changes goes through.

8. Fence in AI agents. Create an unprivileged user that can read the repo only, keep .env files at mode 600 owned by another account, set chattr +i (as root) on live config, and write the rules in AGENTS.md. Check: as that user, appending to a live config file fails with "Operation not permitted", and reading a .env fails with "Permission denied".

Key takeaways

• Pin every image, and compare image IDs before and after so you know pinning didn’t change anything.
• Detect drift against what was last deployed, not the latest commit, so you can tell a hand edit from a pending change.
• On a GPU server, treat a systemd reload as a production event. Reload only when systemd files change, and check GPU access afterwards.
• Disable and label instead of deleting, so rolling back is a rename rather than a re-download.
• Make backup failure loud, check the backup’s age before risky work, and test a restore before you rely on it.
• Alerts aren’t enough for AI agents. Give them scoped, unprivileged accounts, keep secrets out of their reach, and send every change through the repo.