Our first smart router made every request slower and pushed other clients’ models out of memory. This is how we rebuilt it in the lab, with rules before models, resident models first and clear refusals, and how to build one yourself.<\/p>\n\n\n\n
Where this fits \u00b7 Running Local AI in Production \u00b7 Part 5 of 11<\/strong><\/p>\n\n\n\n You are reading Part 5 of 11 in a series where we built a private AI server in our lab and ran realistic test workloads on it for seven weeks. Each part stands on its own. The previous part covered fitting every prompt to a local model’s context window; this one covers routing requests across models on a single GPU without constant model swaps.<\/p>\n\n\n\n \u2190 Part 4: What Happens When a Prompt Is Too Big for a Local LLM?<\/a> \u00b7 Series guide<\/a> \u00b7 Part 6: How Do You Keep an AI Server Stable? GitOps for Local AI \u2192<\/a><\/p>\n\n\n\n Just want it working? Follow the step-by-step setup guide<\/a>: local AI with Ollama and LiteLLM, from a bare server to a secure gateway.<\/p>\n\n\n\n This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.<\/em><\/p>\n\n\n\n The lab server is the NVIDIA DGX Spark from Part 1<\/a>, with roughly 120 GB of usable unified memory<\/strong>, one pool shared by the CPU and GPU. As Part 3<\/a> explained, every model loaded into that pool reserves memory for its context, and loading a different model means pushing another one out. On one GPU, a model swap costs seconds, and a cold load can take more than a minute.<\/p>\n\n\n\n Many different callers share the server. IDE plugins send code, test users ask questions in Open WebUI, and a Telegram agent (Part 9<\/a>) sends long prompts full of tool definitions. None of them should need to know which model is loaded, so the gateway (LiteLLM, Part 2<\/a>) offers a model called The first router was a FastAPI service of about 380 lines. It sent the last three user turns to A review found more. The router kept its own model list, which drifted from the gateway’s every time a model was added or renamed. It didn’t know what was loaded or how long the prompt was, so it would send a 30,000-token prompt to a model with a 16,384-token window. And nothing stopped it choosing a paid cloud model. It just happened not to, and one config edit could have changed that.<\/p>\n\n\n\n The obvious fix was a better classifier, but timing showed the classifier itself was fast, 13 ms per call once loaded. The cost was where it ran and what it evicted. So we looked at the problem as memory first and accuracy second.<\/p>\n\n\n\n We considered running the classifier on the CPU, and chose a separate small Ollama instance instead, because it also gives other helper models, such as embedding models, a stable home. We considered letting The main Ollama instance now holds two models ( The router reads the model list from the gateway’s Adding a model now means adding one gateway entry.<\/p>\n\n\n\n Slash commands win.<\/strong> If the last user message starts with a command, the caller has chosen. An explicit name still goes through the gateway’s per-key model list and rate limit, so a key can only reach models it is already allowed to call.<\/p>\n\n\n\nSee all 11 parts of the series<\/summary>\n\n
\n\n
TL;DR<\/h3>\n\n\n\n
\n\n
auto<\/code> and a small router picks the local model. Its first version made every auto<\/code> call take 9 seconds instead of 2, and pushed other clients’ models out of memory.<\/li>\n\n\nJargon buster: 15 terms used in this post, explained in plain English<\/summary>\n\n
\n\n
In this post<\/h3>\n\n\n\n
\n\n
The problem: why did routing make everything slower?<\/h2>\n\n\n\n
auto<\/code>, and behind it sits a small service we call the smart router.<\/p>\n\n\n\nqwen3:0.6b<\/code>, a 0.6-billion-parameter model, for a one-word label (CODE, AGENTIC, QA or SIMPLE), mapped CODE to qwen3-coder and everything else to gemma4, and forwarded the request. The classifier ran on the main Ollama instance, which at the time held one model at a time. So each auto<\/code> request loaded the classifier, which evicted the big model, and then the big model loaded again. An auto<\/code> request took 9 seconds against 2 seconds calling the model directly, and if another test client was using a different model at that moment, the router evicted theirs too.<\/p>\n\n\n\n
\n\n\n\nWhat we were aiming for<\/h2>\n\n\n\n
\n\n
\n\n\n\nHow we approached it<\/h2>\n\n\n\n
auto<\/code> choose from five or more models, and chose two, because both can stay loaded and nothing gets evicted. And instead of making the model smarter, we put plain rules in front of it, so the model only sees the cases rules can’t settle.<\/p>\n\n\n\n
\n\n\n\nWhat we built<\/h2>\n\n\n\n
Two resident models and a separate classifier<\/h3>\n\n\n\n
OLLAMA_MAX_LOADED_MODELS=2<\/code>): qwen3-coder<\/strong> for code at 20.3 GB and gemma4<\/strong> for chat, questions and agent work at 17.5 GB, measured together at 37.8 GB with one slot each. Everything else on disk is "cold": callable by name, never chosen automatically. A separate small Ollama instance on the same pinned version runs only utility models (the server also runs a sandbox instance for a risky model, see Part 3<\/a>). It holds qwen3:0.6b<\/code> with a 24-hour keep-alive, and its weights are hard-linked from the main instance, so they take no extra disk.<\/p>\n\n\n\nOne model list, read from the gateway<\/h3>\n\n\n\n
\/model\/info<\/code> endpoint at start-up and every five minutes, keeping the last good list if a refresh fails. Each model entry carries a few tags:<\/p>\n\n\n\nmodel_info:\n site_tier: warm # warm | cold | cloud | sandbox (only warm is auto-routable)\n capabilities: [code, fim, chat]\n context_window: 32768\n cost_class: free # free | metered\n routable: true # false = callable only by explicit name\n<\/code><\/pre>\n\n\n\nA routing pipeline that does the cheap steps first<\/h3>\n\n\n\n