nvidia-smi<\/code> inside the container. Enabling a systemd timer for backups broke Ollama and the OCR API within 60 seconds.<\/p>\n\n\n\nSo we schedule with cron, and after any reload the GPU restore tests GPU access in each GPU container and restarts the ones that lost it, even mid-request. Early on, deploy.sh host<\/code> reloaded systemd on every host deploy. One afternoon in week two, eight script-only host deploys restarted Ollama eight times, and one restart turned a test caller’s request into an Ollama 500 error; the gateway retried, and the answer took 28 seconds instead of about 20. The deploy now reloads systemd only when a file under the systemd folders changed.<\/p>\n\n\n\nThe OS can trigger this too. In week three an automatic package refresh ran its own daemon-reload<\/code>, and the autoheal restarted three services. Automatic refreshes, security updates included, now wait for a maintenance window.<\/p>\n\n\n\nTools that reported success when they hadn’t checked<\/h3>\n\n\n\n
The GPU restore reported success whenever its cron copy held the lock. Now it says "checked nothing", and the deploy verifies GPU access itself.<\/p>\n\n\n\n
Deploys that interleaved or didn’t take effect<\/h3>\n\n\n\n
Parallel deploys interleaved, so the script now takes a lock. And a config-only deploy didn’t always apply: when a file is bind-mounted as a single file<\/em>, the sync replaces it with a rename (a new inode) and the container keeps seeing the old content. Those containers need a restart, in the right order.<\/p>\n\n\n\nDrift alerts don’t stop anyone<\/h3>\n\n\n\n
Drift detection notices a hand edit after the fact but doesn’t prevent one, and an AI agent never reads the ops chat. So the agent rules rely on permissions, not alerts.<\/p>\n\n\n\n
\n\n\n\nThe result<\/h2>\n\n\n\n
We tested the loop with a deliberate hand edit: it showed up in the next state commit, the next deploy refused to run, and --accept-drift --no-recreate<\/code> put it right. Pinning every image changed nothing that was running. Leaving model weights out made the backup archive about a third smaller.<\/p>\n\n\n\nThree days in, we updated the host’s OS, kernel, NVIDIA driver and Container Toolkit, mainly for better memory handling, and treated it like any change: a fresh backup, the GPU idle, the before-state saved and the old kernel kept as a fallback. Afterwards every GPU container had GPU access, dry runs showed no drift, and it soaked for three days before the next change.<\/p>\n\n\n\n
The cost is speed: a one-line change is now a commit, six checks, a review, a dry run and a deploy, which is still far cheaper than recovering from a freeze. Two gaps remain: configuration inside Docker volumes is invisible to deploy.sh<\/code>, and GPU injection should move to the toolkit’s Container Device Interface (CDI) mode, which avoids the reload problem instead of repairing it.<\/p>\n\n\n\n
\n\n\n\nDo it yourself: a GitOps repo for one AI server<\/h2>\n\n\n\n
What you need:<\/strong> one Linux server running Docker Compose 2.17 or later (for --dry-run<\/code>), with the NVIDIA Container Toolkit if you use NVIDIA GPUs, a private Git remote, and about an afternoon.<\/p>\n\n\n\n\n\n- \n
Lay out the repo<\/strong>, one folder per stack plus host config and state:<\/p>\nstacks\/<stack>\/compose.yaml stacks\/<stack>\/.env.template\nhost\/cron.d\/ host\/systemd\/ scripts\/deploy.sh state\/ AGENTS.md CHANGELOG.md\n<\/code><\/pre>\nCheck:<\/em> after copying the live files in, diff -r<\/code> against the live folder shows nothing.<\/p>\n<\/li>\n\n\n- \n
Pin every image by digest<\/strong>, using what is already running. Record each container’s image ID first, so you can prove the pin changed nothing:<\/p>\ndocker inspect --format '{{.Image}}' ollama > before-ollama.txt\ndocker image inspect --format '{{index .RepoDigests 0}}' "ollama\/ollama:$TAG" # TAG = the tag running now\n# compose.yaml \u2192 image: ollama\/ollama@sha256:<digest>\n<\/code><\/pre>\nChanging image:<\/code> is a config change, so Compose will recreate the container; do it one service at a time in a maintenance window.\nCheck:<\/em> docker compose config --images<\/code> lists only @sha256:<\/code> references, and after the recreate docker inspect --format '{{.Image}}' ollama<\/code> prints the same image ID as before-ollama.txt<\/code>.<\/p>\n<\/li>\n\n\n- \n
Record what was deployed.<\/strong> At the end of each deploy, write the commit to state\/deployed-<stack><\/code>, and check drift against it:<\/p>\nS=mystack # the stack's folder name\ngit show "$(cat "state\/deployed-$S"):stacks\/$S\/compose.yaml" | diff -q - "\/srv\/$S\/compose.yaml"\n<\/code><\/pre>\nCheck:<\/em> hand-edit the live file, and the deploy stops and names it.<\/p>\n<\/li>\n\n\n- \n
Deploy one stack at a time:<\/strong> git pull --ff-only<\/code>, the drift check, docker compose --dry-run up -d<\/code>, the sync, then docker compose up -d --no-recreate<\/code> unless a recreate was explicitly allowed. Wrap it in flock<\/code> so two deploys can’t interleave.\nCheck:<\/em> running the same deploy twice reports nothing to change.<\/p>\n<\/li>\n\n\n- \n
Reload systemd only when units changed<\/strong>, then verify the GPU (NVIDIA example):<\/p>\ngit diff --name-only "$OLD" "$NEW" | grep -q '^host\/systemd\/' && sudo systemctl daemon-reload\ndocker exec <gpu-container> nvidia-smi -L\n<\/code><\/pre>\nCheck:<\/em> nvidia-smi -L<\/code> lists the GPU in every GPU container after a host deploy. Failed to initialize NVML<\/code> means the reload took GPU access away; restart that container.<\/p>\n<\/li>\n\n\n- \n
Snapshot nightly from cron.<\/strong> Write docker ps --format '{{.Names}} {{.Image}}'<\/code>, docker exec ollama ollama list<\/code> and sha256sum<\/code> of the live config into state\/<\/code>, and commit only if git status --porcelain state\/<\/code> prints something. Cron has a minimal environment, so set PATH<\/code> in the script, commit with git -c user.name=state-bot -c user.email=<address><\/code>, and push with a deploy key that can write only to this repo.\nCheck:<\/em> a quiet night produces no commit; a hand edit shows up the next morning.<\/p>\n<\/li>\n\n\n- \n
Add a pre-commit hook<\/strong> (.git\/hooks\/pre-commit<\/code>, executable) that refuses private keys and any image not pinned by digest, and lets clean commits through:<\/p>\n#!\/bin\/sh\nadded=$(git diff --cached -U0 | grep '^+')\nif printf '%s\\n' "$added" | grep -Eq 'BEGIN [A-Z ]*PRIVATE KEY'; then\n echo "refused: private key in commit" >&2; exit 1\nfi\nif printf '%s\\n' "$added" | grep -E '^\\+[[:space:]]*image:' | grep -vq '@sha256:'; then\n echo "refused: image not pinned by digest" >&2; exit 1\nfi\nexit 0\n<\/code><\/pre>\nCheck:<\/em> a compose file with image: redis:7<\/code> or a bare image: nginx<\/code> is rejected, and an ordinary commit with no image changes goes through.<\/p>\n<\/li>\n\n\n- \n
Fence in AI agents.<\/strong> Create an unprivileged user that can read the repo only, keep .env<\/code> files at mode 600<\/code> owned by another account, set chattr +i<\/code> (as root) on live config, and write the rules in AGENTS.md<\/code>.\nCheck:<\/em> as that user, appending to a live config file fails with "Operation not permitted", and reading a .env<\/code> fails with "Permission denied".<\/p>\n<\/li>\n\n<\/ol>\n\n\n\n