{"id":1095,"date":"2026-09-07T09:30:00","date_gmt":"2026-09-06T23:30:00","guid":{"rendered":"https:\/\/catharsis.net.au\/blog\/local-llm-context-budget\/"},"modified":"2026-11-03T20:27:55","modified_gmt":"2026-11-03T09:27:55","slug":"local-llm-context-budget","status":"publish","type":"post","link":"https:\/\/catharsis.net.au\/blog\/local-llm-context-budget\/","title":{"rendered":"What Happens When a Prompt Is Too Big for a Local LLM? (Part 4 of 11)"},"content":{"rendered":"\n

On a local AI server, a prompt too big for the model gets silently truncated, and raising the window for every parallel slot is what froze our lab server. We built admission control and gateway-side compaction, then found our token counts were out by up to 322 times.<\/p>\n\n\n\n

\n\n

Where this fits \u00b7 Running Local AI in Production \u00b7 Part 4 of 11<\/strong><\/p>\n\n\n\n

You are reading Part 4 of 11 in a series where we built a private AI server in our lab and ran realistic test workloads on it for seven weeks. Each part stands on its own. The previous part covered five hard freezes and the memory budget that ended them; this one covers fitting every prompt to a local model’s context window.<\/p>\n\n\n\n

\u2190 Part 3: Why Does a Local AI Server Freeze? Sizing Memory for LLMs<\/a>  \u00b7  Series guide<\/a>  \u00b7  Part 5: How Do You Run Several AI Models on One GPU Without Slowdowns? \u2192<\/a><\/p>\n\n\n\n

Just want it working? Follow the step-by-step setup guide<\/a>: local AI with Ollama and LiteLLM, from a bare server to a secure gateway.<\/p>\n\n\n\n

See all 11 parts of the series<\/summary>\n\n
    \n\n
  1. Local AI or Cloud AI? Why We Built a Private AI Server<\/a>: why we built the lab on our own hardware, what ran on it and what that choice cost<\/li>\n\n\n
  2. How Do You Control Access to Local AI Models? Use a Gateway<\/a>: one LLM gateway as the only way to reach a model: keys, budgets and tracing<\/li>\n\n\n
  3. Why Does a Local AI Server Freeze? Sizing Memory for LLMs<\/a>: five hard freezes and the memory budget that ended them<\/li>\n\n\n
  4. What Happens When a Prompt Is Too Big for a Local LLM?<\/strong> (you are here): fitting every prompt to a local model’s context window<\/li>\n\n\n
  5. How Do You Run Several AI Models on One GPU Without Slowdowns?<\/a>: routing requests across models on a single GPU without constant model swaps<\/li>\n\n\n
  6. How Do You Keep an AI Server Stable? GitOps for Local AI<\/a>: running the server from a Git repository: deploys, drift, pinning and safe AI agents<\/li>\n\n\n
  7. What Is Your Local AI Server Actually Doing? Observability Basics<\/a>: making the server observable: traces, dashboards, usage and heat<\/li>\n\n\n
  8. Can Local AI Agents Analyse Stocks? How We Tested and Scored Them<\/a>: a multi-agent stock-analysis experiment and how we scored the agents<\/li>\n\n\n
  9. Why Was Our Telegram AI Agent So Slow? The Hidden Prompt Cost<\/a>: a Telegram AI agent on local models, and why its fixed prompt became the budget<\/li>\n\n\n
  10. Can AI Automate Business Workflows Without Sending Data Out?<\/a>: document, email and knowledge-base workflows that keep data on the server<\/li>\n\n\n
  11. Is Private AI Worth It? Lessons From Seven Weeks in Our Lab<\/a>: the lessons from seven weeks, the honest limits and what comes next<\/li>\n\n<\/ol>\n\n<\/details>\n\n<\/div><\/div>\n\n\n\n

    This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.<\/em><\/p>\n\n\n\n

    \n\n

    TL;DR<\/h3>\n\n\n\n