How Do You Run Several AI Models on One GPU Without Slowdowns? (Part 5 of 11)

Our first smart router made every request slower and pushed other clients’ models out of memory. This is how we rebuilt it in the lab, with rules before models, resident models first and clear refusals, and how to build one yourself.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The problem: why did routing make everything slower?

The lab server is the NVIDIA DGX Spark from Part 1, with roughly 120 GB of usable unified memory, one pool shared by the CPU and GPU. As Part 3 explained, every model loaded into that pool reserves memory for its context, and loading a different model means pushing another one out. On one GPU, a model swap costs seconds, and a cold load can take more than a minute.

Many different callers share the server. IDE plugins send code, test users ask questions in Open WebUI, and a Telegram agent (Part 9) sends long prompts full of tool definitions. None of them should need to know which model is loaded, so the gateway (LiteLLM, Part 2) offers a model called auto, and behind it sits a small service we call the smart router.

The first router was a FastAPI service of about 380 lines. It sent the last three user turns to qwen3:0.6b, a 0.6-billion-parameter model, for a one-word label (CODE, AGENTIC, QA or SIMPLE), mapped CODE to qwen3-coder and everything else to gemma4, and forwarded the request. The classifier ran on the main Ollama instance, which at the time held one model at a time. So each auto request loaded the classifier, which evicted the big model, and then the big model loaded again. An auto request took 9 seconds against 2 seconds calling the model directly, and if another test client was using a different model at that moment, the router evicted theirs too.

A review found more. The router kept its own model list, which drifted from the gateway’s every time a model was added or renamed. It didn’t know what was loaded or how long the prompt was, so it would send a 30,000-token prompt to a model with a 16,384-token window. And nothing stopped it choosing a paid cloud model. It just happened not to, and one config edit could have changed that.

What we were aiming for

• Routing adds well under a second, and never causes a model swap on its own.
• One list of models, kept in the gateway, so the router can’t drift.
• Cloud, experimental and oversized choices never happen automatically. A prompt that doesn’t fit gets a clear error, never silent truncation.
• Every routing decision is visible in the traces, so we can check it was right.

How we approached it

The obvious fix was a better classifier, but timing showed the classifier itself was fast, 13 ms per call once loaded. The cost was where it ran and what it evicted. So we looked at the problem as memory first and accuracy second.

We considered running the classifier on the CPU, and chose a separate small Ollama instance instead, because it also gives other helper models, such as embedding models, a stable home. We considered letting auto choose from five or more models, and chose two, because both can stay loaded and nothing gets evicted. And instead of making the model smarter, we put plain rules in front of it, so the model only sees the cases rules can’t settle.

What we built

Two resident models and a separate classifier

The main Ollama instance now holds two models (OLLAMA_MAX_LOADED_MODELS=2): qwen3-coder for code at 20.3 GB and gemma4 for chat, questions and agent work at 17.5 GB, measured together at 37.8 GB with one slot each. Everything else on disk is "cold": callable by name, never chosen automatically. A separate small Ollama instance on the same pinned version runs only utility models (the server also runs a sandbox instance for a risky model, see Part 3). It holds qwen3:0.6b with a 24-hour keep-alive, and its weights are hard-linked from the main instance, so they take no extra disk.

One model list, read from the gateway

The router reads the model list from the gateway’s /model/info endpoint at start-up and every five minutes, keeping the last good list if a refresh fails. Each model entry carries a few tags:

model_info:
  site_tier: warm           # warm | cold | cloud | sandbox (only warm is auto-routable)
  capabilities: [code, fim, chat]
  context_window: 32768
  cost_class: free          # free | metered
  routable: true            # false = callable only by explicit name

Adding a model now means adding one gateway entry.

A routing pipeline that does the cheap steps first

Slash commands win. If the last user message starts with a command, the caller has chosen. An explicit name still goes through the gateway’s per-key model list and rate limit, so a key can only reach models it is already allowed to call.

Rules decide the clear cases in effectively zero time. A tools array means AGENTIC, because only an agent sends tool definitions. Code fences, stack traces, file paths with extensions, def, class or import, SELECT … FROM, error names such as TypeError: and call syntax like df.dropna(...) mean CODE. Twelve words or fewer, with no question mark and no request to explain, draft or summarise, means SIMPLE. A question or a prose task such as "explain" or "translate", with no sign of code, means QA.

The tiny classifier handles the rest, using a prompt with examples and parsing that tolerates near-misses (more on that below).

Labels become candidates. Each label maps to a capability: CODE needs code, AGENTIC needs tools, QA and SIMPLE need chat. A candidate must be routable, in the warm tier, have the capability, and be large enough for the estimated prompt plus the answer cap. Cloud, sandbox and cold models are never chosen automatically, even if someone marks one routable by mistake. The estimate is deliberately rough, characters divided by four plus the caller’s max_tokens or the label’s default cap (1,024 for SIMPLE, 4,096 for QA and AGENTIC, 8,192 for CODE).

Unlike the gateway’s exact count in Part 4, this only has to pick a model, so being off by 20% costs nothing. The exact count and admission check happen at the gateway (Part 4), which applies its own budget to auto so an oversized request is refused at the first hop.

Loaded models come first. The router asks Ollama what is loaded (/api/ps, cached for five seconds) and puts a loaded candidate first, then breaks ties by a preference order and the larger window. Capability still beats residency: a code request goes to qwen3-coder even if only gemma4 is in memory. If Ollama can’t be reached, the router treats nothing as loaded.

When nothing fits, it says so. The router returns HTTP 413 context_too_large, naming the smallest model that could take the prompt (callable with /model:<name>) or saying there is none. The gateway’s own check returns the 400 from Part 4; the two codes come from two different hops.

Every decision in the traces

Each forwarded request carries its decision: the label, the reason (for example rule: language + code noun; CODE → qwen3-coder (resident)), whether a loaded model was used, whether the caller chose, whether a rule or the classifier decided, and the estimated size. In the tracing tool (Langfuse, Part 7) the router sets filterable metadata, a trace name such as auto → qwen3-coder, and tags such as router:label:CODE and router:via:rule, merged with anything the caller sent.

Challenges we hit

A tiny model that called plain questions code

With the original prompt, qwen3:0.6b labelled plain questions as CODE 9 times out of 20 on a realistic sample ("What is the capital of France?", "Explain how DNS works", "How do tides work?") and 10 out of 20 on a held-out set of another 20. The prompt listed CODE first and gave definitions without examples. We listed QA first, added "if unsure, QA", and sent ten example question-and-answer pairs before the real message, because a model this small follows examples far better than definitions. It then scored 20 out of 20 on the sample and 16 out of 20 on the held-out set. The four misses are either caught by the rules anyway or are QA-versus-AGENTIC confusions that both go to gemma4. Near-misses such as SIMPLEX or Label: QA are now read correctly, and anything unreadable defaults to QA. In practice the rules now settle all 40 prompts in both test sets, so the classifier only sees genuinely ambiguous messages; its accuracy matters for those.

Everyday words that look like code

Some signals are weak alone: a language name ("python", "sql"), a code noun ("function", "regex", "bug") or a code verb ("refactor", "fix"). Everyday text is full of them. "I have a query about tide times", "fix a flat tyre" and "Express your answer as a fraction" must never become CODE, so weak signals count only in pairs. A single one, as in "Is Python a snake?", goes to the classifier.

A field name that took routing down

The first deployment named the tier field tier. LiteLLM already uses model_info.tier for its own values, so it rejected every model entry and auto was down for about five minutes until we renamed it with a prefix. When you add your own metadata to another product’s config, prefix the field names.

Stale windows and a third warm model

A few days later, while writing the memory budget into policy after a hard freeze (Part 3), we found two routing bugs. qwen3-coder and gemma4 still advertised a 65,536-token window after being capped at 32,768, so the router was sizing prompts against capacity that no longer existed. And qwen3:14b and gemma3:12b were still tagged warm. A second large model loaded beside qwen3-coder is exactly what we found during the freeze, and a third would have evicted the warm pair. Both moved to cold, with a specialist model, so auto can now choose only qwen3-coder or gemma4. After a reboot gemma4 took about 70 seconds to load cold and qwen3-coder 10.5 seconds, and before one hard freeze qwen3-coder had been loaded, unloaded and reloaded within five minutes. We can’t say that churn caused the freeze, but a router should never add to it.

A routing note that agents copied

Each auto reply began with a short note, such as `[router → gemma4: rule: question; QA → gemma4 (resident)]`. Testers in Open WebUI liked knowing which model answered. To an agent, though, the note is just text. It gets saved in the history, uses context on every later turn and gets repeated, and the test Telegram agent started its replies with [router → …]. Requests that carry tools now skip the note unless a setting turns it back on.

The result

Routing an auto request went from 9 seconds to 0.4 to 0.5 seconds. In the same evening’s measurements both big models stayed resident, the classifier loaded on the main instance zero times, and a direct qwen3-coder call answered in 2.5 seconds. Most of the gain came from not swapping models, not from a smarter decision.

The trade-offs are deliberate. auto chooses between two models, not five, because more choice would mean more swapping. Rules need upkeep when new everyday text looks like code, and 113 offline unit tests, including both test sets, make that safe. Each resident model has one slot, so a second caller to the same model still waits in Ollama’s queue. Routing decides which queue a request joins; it doesn’t shorten it.

Do it yourself: rules-first routing with a small resident set

What you need: a GPU server running Ollama in Docker behind a LiteLLM proxy, two models that fit in memory together, and a small Python web service such as FastAPI. Allow a day.

1. Keep two models warm. Set OLLAMA_MAX_LOADED_MODELS=2 and a keep-alive of at least a few minutes on the main instance. Check: after one request to each, curl -s localhost:11434/api/ps lists both.

2. Give the classifier its own instance, with its own model volume, on another loopback port.

     ollama-helpers:
       image: ollama/ollama@sha256:<digest>
       restart: unless-stopped
       ports:
         - "127.0.0.1:11436:11434"
       volumes:
         - ollama-helpers:/root/.ollama
       environment:
         OLLAMA_KEEP_ALIVE: "24h"
         OLLAMA_MAX_LOADED_MODELS: "2"
   

   Pull qwen3:0.6b into it. Hard-linking weights from the main instance saves disk only if both volumes sit on the same filesystem. Check: the main instance’s /api/ps never lists the classifier.

3. Tag models in the gateway. Add prefixed custom fields to each entry’s model_info, as in the example above, and mark only your two resident models as warm and routable. Check: GET /model/info shows the fields, and the proxy starts without rejecting any entry.

4. Write the router. The core is one function: the cheapest decision first, then filter, then prefer what is loaded.

   def route(req):
       label = slash_command(req) or rules(req) or classify(req)  # classifier last
       need = CAPABILITY[label]                     # "code", "tools" or "chat"
       size = rough_estimate(req) + req.get("max_tokens", DEFAULT_CAP[label])
       loaded = loaded_models()                     # Ollama /api/ps, cached 5 s
       candidates = [m for m in gateway_models()    # gateway /model/info
                     if m.routable and m.site_tier == "warm"
                     and need in m.capabilities and m.context_window >= size]
       if not candidates:
           return error(413, "context_too_large", suggest=smallest_fit(size))
       candidates.sort(key=lambda m: (m.name not in loaded, PREFERENCE.index(m.name)))
       return forward(req, candidates[0])
   

5. Write the rules first. Tools present means agent, code markers mean code, very short text means small talk, and a question means QA. Count weak signals only in pairs. Check: unit tests with everyday sentences like "fix a flat tyre" all come out as non-code.

6. Add the classifier last. Send only undecided messages, with ten labelled examples ahead of them, and default to QA when the reply can’t be parsed. Check: on a set of 20 plain questions, none comes back as CODE, and a prompt bigger than every window gets a 413, never a truncated answer.

7. Expose it as auto in the gateway’s model_list. The openai/ provider insists on an API key, so pass one even if the router ignores it (load a real one with os.environ/ if the router checks keys), and include /v1 in the base URL.

     - model_name: auto
       litellm_params:
         model: openai/auto
         api_base: http://127.0.0.1:<router-port>/v1
         api_key: none   # any non-empty value; the router ignores it
       model_info:
         max_input_tokens: 26000
   

   Check: a request for auto returns an answer, and the trace shows which model served it.

8. Test for thrash. Run two clients that alternate 20 code prompts and 20 chat prompts. Check: after warm-up, /api/ps still lists the same two models and Ollama’s log shows no new model loads.

Key takeaways

• On one GPU, the expensive part of routing is a model swap. Measure end-to-end latency before tuning the classifier.
• Use rules first and a small model last, and give that model examples rather than definitions.
• Keep one list of models in the gateway. A second list will drift.
• Choose by capability first and let residency break ties.
• Never fall back to the cloud silently, and refuse clearly instead of truncating.
• Separate what people see from what agents receive, because a helpful note to a person is just extra tokens to an agent.