{"id":1089,"date":"2026-08-17T09:30:00","date_gmt":"2026-08-16T23:30:00","guid":{"rendered":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/"},"modified":"2026-11-03T20:53:10","modified_gmt":"2026-11-03T09:53:10","slug":"why-run-ai-on-your-own-hardware","status":"publish","type":"post","link":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/","title":{"rendered":"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11)"},"content":{"rendered":"\n<p class=\"has-pale-cyan-blue-background-color has-background wp-block-paragraph\">We built a private AI server in our lab and ran realistic test workloads on it for seven weeks. This post covers why we went local, what ran on the server, what that choice cost, and how to set up the same base server yourself.<\/p>\n\n\n\n<div class=\"wp-block-group has-background\" style=\"background-color:#f4f6f8;padding-top:28px;padding-right:32px;padding-bottom:28px;padding-left:32px\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-container-core-group-is-layout-40de7cf3 wp-block-group-is-layout-constrained\">\n\n<p class=\"wp-block-paragraph\"><strong>Where this fits \u00b7 Running Local AI in Production \u00b7 Part 1 of 11<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:17px\">You are reading Part 1 of 11 in a series where we built a private AI server in our lab and ran realistic test workloads on it for seven weeks. Each part stands on its own. This one covers why we built the lab on our own hardware, what ran on it and what that choice cost.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:16px\"><a href=\"https:\/\/catharsis.net.au\/blog\/running-local-ai-in-production-series-guide\/\" target=\"_blank\" rel=\"noreferrer noopener\">Series guide<\/a> &nbsp;\u00b7&nbsp; <a href=\"https:\/\/catharsis.net.au\/blog\/llm-gateway-one-door-for-every-model\/\" target=\"_blank\" rel=\"noreferrer noopener\">Part 2: How Do You Control Access to Local AI Models? Use a Gateway \u2192<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:16px\">Just want it working? Follow the <a href=\"https:\/\/catharsis.net.au\/blog\/local-ai-litellm-setup-guide\/\" target=\"_blank\" rel=\"noreferrer noopener\">step-by-step setup guide<\/a>: local AI with Ollama and LiteLLM, from a bare server to a secure gateway.<\/p>\n\n\n\n<details class=\"wp-block-details is-layout-flow wp-block-details-is-layout-flow\"><summary>See all 11 parts of the series<\/summary>\n\n<ol class=\"wp-block-list\">\n\n<li><strong>Local AI or Cloud AI? Why We Built a Private AI Server<\/strong> (you are here): why we built the lab on our own hardware, what ran on it and what that choice cost<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/llm-gateway-one-door-for-every-model\/\" target=\"_blank\" rel=\"noreferrer noopener\">How Do You Control Access to Local AI Models? Use a Gateway<\/a>: one LLM gateway as the only way to reach a model: keys, budgets and tracing<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/local-ai-server-freezes-unified-memory\/\" target=\"_blank\" rel=\"noreferrer noopener\">Why Does a Local AI Server Freeze? Sizing Memory for LLMs<\/a>: five hard freezes and the memory budget that ended them<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/local-llm-context-budget\/\" target=\"_blank\" rel=\"noreferrer noopener\">What Happens When a Prompt Is Too Big for a Local LLM?<\/a>: fitting every prompt to a local model&#8217;s context window<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/smart-routing-single-gpu-local-llm\/\" target=\"_blank\" rel=\"noreferrer noopener\">How Do You Run Several AI Models on One GPU Without Slowdowns?<\/a>: routing requests across models on a single GPU without constant model swaps<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/gitops-for-a-gpu-ai-server\/\" target=\"_blank\" rel=\"noreferrer noopener\">How Do You Keep an AI Server Stable? GitOps for Local AI<\/a>: running the server from a Git repository: deploys, drift, pinning and safe AI agents<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/observability-for-local-ai\/\" target=\"_blank\" rel=\"noreferrer noopener\">What Is Your Local AI Server Actually Doing? Observability Basics<\/a>: making the server observable: traces, dashboards, usage and heat<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/multi-agent-stock-analysis-local-llm\/\" target=\"_blank\" rel=\"noreferrer noopener\">Can Local AI Agents Analyse Stocks? How We Tested and Scored Them<\/a>: a multi-agent stock-analysis experiment and how we scored the agents<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/telegram-ai-agent-local-llms\/\" target=\"_blank\" rel=\"noreferrer noopener\">Why Was Our Telegram AI Agent So Slow? The Hidden Prompt Cost<\/a>: a Telegram AI agent on local models, and why its fixed prompt became the budget<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/local-ai-business-workflows\/\" target=\"_blank\" rel=\"noreferrer noopener\">Can AI Automate Business Workflows Without Sending Data Out?<\/a>: document, email and knowledge-base workflows that keep data on the server<\/li>\n\n\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/private-ai-lab-lessons-and-whats-next\/\" target=\"_blank\" rel=\"noreferrer noopener\">Is Private AI Worth It? Lessons From Seven Weeks in Our Lab<\/a>: the lessons from seven weeks, the honest limits and what comes next<\/li>\n\n<\/ol>\n\n<\/details>\n\n<\/div><\/div>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:16px\"><em>This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.<\/em><\/p>\n\n\n\n<div class=\"wp-block-group has-background\" style=\"background-color:#f4f6f8;padding-top:28px;padding-right:32px;padding-bottom:28px;padding-left:32px\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-container-core-group-is-layout-40de7cf3 wp-block-group-is-layout-constrained\">\n\n<h3 class=\"wp-block-heading\">TL;DR<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li>Local or cloud? For sensitive, everyday work we chose local: data stays on hardware you own, the cost is fixed and you decide when anything changes. The cloud still fits public, non-sensitive work.<\/li>\n\n\n<li>The problem: cloud AI means sending your data to someone else, paying per token and accepting their changes. We wanted to know whether a small team could run its everyday AI on its own hardware instead.<\/li>\n\n\n<li>Our approach: one NVIDIA DGX Spark in our lab, seven weeks of realistic test workloads, and one rule in place of a data classifier: data never leaves the server.<\/li>\n\n\n<li>The result: it works, with two models resident in 37.8 GB and every call going through one gateway. The price is that you become the operator, and the lab server froze hard several times before we got there.<\/li>\n\n\n<li>The lesson: design around the memory you measure, not the model card, and turn every problem into a written rule that a check enforces.<\/li>\n\n<\/ul>\n\n<\/div><\/div>\n\n\n\n<div class=\"wp-block-group has-background\" style=\"background-color:#f4f6f8;padding-top:28px;padding-right:32px;padding-bottom:28px;padding-left:32px\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-container-core-group-is-layout-40de7cf3 wp-block-group-is-layout-constrained\">\n\n<details class=\"wp-block-details is-layout-flow wp-block-details-is-layout-flow\"><summary>Jargon buster: 15 terms used in this post, explained in plain English<\/summary>\n\n<ul class=\"wp-block-list\">\n\n<li><strong>Local LLM \/ local AI<\/strong>: An AI model that runs on your own hardware instead of a cloud service, so prompts and data stay on your server.<\/li>\n\n\n<li><strong>Model (weights)<\/strong>: The trained numbers that make up an AI model. Loading a model means copying its weights into memory.<\/li>\n\n\n<li><strong>Token<\/strong>: The small chunks of text a model reads and writes, roughly three-quarters of an English word each. Limits and costs are counted in tokens.<\/li>\n\n\n<li><strong>Prompt<\/strong>: Everything sent to a model in one request: the instructions, the conversation so far and any documents.<\/li>\n\n\n<li><strong>KV cache<\/strong>: Working memory a model keeps while it reads a prompt. It grows with the length of the context, so long contexts need a lot of memory.<\/li>\n\n\n<li><strong>Unified memory<\/strong>: A design where the CPU and GPU share one pool of memory instead of the GPU having its own. Flexible, but both compete for the same space.<\/li>\n\n\n<li><strong>GPU<\/strong>: Graphics processing unit: the chip that does the heavy maths for running AI models quickly.<\/li>\n\n\n<li><strong>Slot (parallel requests)<\/strong>: One request a model server can work on at the same time. More slots serve more people at once but reserve more memory.<\/li>\n\n\n<li><strong>Ollama<\/strong>: Free, open-source software that downloads and runs AI models on your own machine and answers requests over a local API.<\/li>\n\n\n<li><strong>LLM gateway<\/strong>: A single service every AI request must pass through, so access, budgets and logging are controlled in one place.<\/li>\n\n\n<li><strong>Workflow<\/strong>: A sequence of automated steps, for example: read an email, extract the details, draft a reply for a person to check.<\/li>\n\n\n<li><strong>OCR<\/strong>: Optical character recognition: turning scanned documents or images of text into machine-readable text.<\/li>\n\n\n<li><strong>AI agent<\/strong>: An AI model given tools (such as search, files or other programs) and allowed to take several steps on its own to complete a task.<\/li>\n\n\n<li><strong>Docker \/ container<\/strong>: A way of packaging an application with everything it needs so it runs the same anywhere; each packaged application runs in its own container.<\/li>\n\n\n<li><strong>systemd<\/strong>: The service manager on most Linux systems: it starts, stops and supervises background services.<\/li>\n\n<\/ul>\n\n<\/details>\n\n<\/div><\/div>\n\n\n\n<div class=\"wp-block-group has-background\" style=\"background-color:#f4f6f8;padding-top:28px;padding-right:32px;padding-bottom:28px;padding-left:32px\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-container-core-group-is-layout-40de7cf3 wp-block-group-is-layout-constrained\">\n\n<h3 class=\"wp-block-heading\">In this post<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li><a href=\"#the-problem-ai-is-useful-but-the-cloud-has-a-price\">The problem: AI is useful, but the cloud has a price<\/a><\/li>\n\n\n<li><a href=\"#what-we-were-aiming-for\">What we were aiming for<\/a><\/li>\n\n\n<li><a href=\"#how-we-approached-it\">How we approached it<\/a><\/li>\n\n\n<li><a href=\"#what-we-built\">What we built<\/a><\/li>\n\n\n<li><a href=\"#challenges-we-hit\">Challenges we hit<\/a><\/li>\n\n\n<li><a href=\"#the-result\">The result<\/a><\/li>\n\n\n<li><a href=\"#do-it-yourself-a-base-local-ai-server\">Do it yourself: a base local AI server<\/a><\/li>\n\n\n<li><a href=\"#key-takeaways\">Key takeaways<\/a><\/li>\n\n<\/ul>\n\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"the-problem-ai-is-useful-but-the-cloud-has-a-price\">The problem: AI is useful, but the cloud has a price<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Most businesses that try AI start with a cloud service, and that means three things. Your data leaves your control: requests, replies, inbound email, invoices and screenshots of error messages all go to someone else&#8217;s servers. You pay per token, so the bill grows with every workflow you add. And the provider decides when a model changes or disappears.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For a business with sensitive data, the first point alone can rule AI out. We wanted to know whether there was a practical alternative: could a small team run useful AI on its own hardware, with nothing sent to a cloud service? Data sheets couldn&#8217;t answer that, so we built one.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"what-we-were-aiming-for\">What we were aiming for<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We set four goals for the lab build:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li>data never leaves the server, enforced by the machine rather than by good intentions;<\/li>\n\n\n<li>a predictable cost, with no per-call bill;<\/li>\n\n\n<li>realistic work on one machine: workflows, document OCR, a chat assistant, coding tools and a multi-agent experiment;<\/li>\n\n\n<li>run it like production, so every change is safe, traceable and reversible.<\/li>\n\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"how-we-approached-it\">How we approached it<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">All test data was synthetic or public: sample document requests, public reference datasets, test invoices and sample PDFs, a synthetic mailbox, and a demo knowledge base built from open-source project documentation. We treated it as though it were real, because that is the point of the exercise, and ran the server under that load for seven weeks.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Our first design was a hybrid. A cloud model would be &quot;the brain&quot; and plan complex work, while local models would be &quot;the hands&quot; for bulk and sensitive processing. A three-tier sensitivity policy would keep personal data, credentials and confidential documents local, with a PII scanner and a keyword classifier inspecting every request.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We dropped the classifier, because a classifier has to be right about every single request. Instead we wrote one sentence into the lab&#8217;s operating rules:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Data never leaves the box; only the stock-analysis experiment may use a cloud model, and only a free one, on public market data.<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">A rule about where requests may go is easier to enforce than a judgement about what they contain. The gateway (<a href=\"https:\/\/catharsis.net.au\/blog\/llm-gateway-one-door-for-every-model\/\" target=\"_blank\" rel=\"noreferrer noopener\">Part 2<\/a>) enforced it in configuration, and a pre-commit check refused any change that would weaken it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For serving models, the plan was <strong>vLLM<\/strong>, a high-throughput inference server, with models in a 4-bit format. A larger 80B Qwen3-Coder variant would use speculative decoding and was expected to reach <strong>60 to 110 tokens per second<\/strong>. The memory table added up to &quot;~119 GB (fits, zero margin)&quot;.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"what-we-built\">What we built<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">The server<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The machine is an NVIDIA DGX Spark, a compact desktop AI system built on the GB10 Grace Blackwell Superchip with 128 GB of <strong>unified memory<\/strong> shared by the CPU and GPU. We had already <a href=\"https:\/\/catharsis.net.au\/blog\/nvidea-dgx-spark-benchmarking\/\" target=\"_blank\" rel=\"noreferrer noopener\">benchmarked two of them linked together<\/a>; this series is about running one as a private AI server. On a normal PC the graphics card has its own memory. Here the 20-core Arm CPU, the Blackwell GPU, the operating system, databases and every model draw from one pool. That is why a 30-billion-parameter model fits on a desktop at all, and it is behind most of the problems in this series. The operating system reports roughly 120 GB of the 128 GB as usable, and that measured figure is the one we budget against throughout. It runs NVIDIA DGX OS, a Linux distribution, with Docker, and every service has its own container.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The workloads<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">By the end of the test program the server ran about 30 containers. The biggest group was <strong>n8n<\/strong>, a workflow-automation tool that works roughly like a self-hosted Zapier, running a few dozen test workflows. One pulled the fields out of a sample document request, drafted a reply for a person to review and filed it. Others loaded public reference datasets on a schedule, routed email from a synthetic mailbox, ran a document OCR form and fed the demo knowledge base.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A <strong>Telegram agent<\/strong> built on Hermes Agent, an open-source personal AI agent, ran on local models, and only lab test accounts could talk to it. IDE coding plugins and a browser chat (Open WebUI) all called one OpenAI-compatible endpoint, LiteLLM, with a key for every client.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For OCR we used an open 3-billion-parameter vision-language model that turns PDFs and images into Markdown with the tables intact. It loads on demand (about 7 GB), handles one job at a time with a 50-page cap and unloads after 10 minutes idle. Two <strong>knowledge bases<\/strong> used retrieval-augmented generation (RAG), where the model searches a document store first and answers from what it found, with sources. Both sat in a Qdrant vector database.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The last big workload was TradingAgents, an open-source multi-agent framework where analysts, researchers, a trader, a risk team and a portfolio manager reach a decision on one ticker. We added a web GUI and a ledger that scores how well each call agreed with later price moves (Part 8). It tests agent behaviour on public market data and is not financial advice. Whisper (speech-to-text), Langfuse (tracing) and a monitoring stack ran alongside.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The models<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The primary model is a <strong>mixture-of-experts<\/strong> model: it uses only a few of its &quot;expert&quot; sub-networks for each token, so it answers at the speed of a much smaller model while all its weights still sit in memory. &quot;4-bit&quot; means <strong>quantised<\/strong>: each weight is stored in fewer bits, which shrinks a model to a third or a quarter of its size for a small loss of quality.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\">\n<thead>\n<tr>\n<th>Model<\/th>\n<th>Size on disk<\/th>\n<th>Role<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>qwen3-coder<\/code> (30B mixture-of-experts, 4-bit)<\/td>\n<td>18 GB<\/td>\n<td>Primary model: code, agents, stock analysis<\/td>\n<\/tr>\n<tr>\n<td><code>gemma4<\/code> (26B)<\/td>\n<td>17 GB<\/td>\n<td>Generalist, reads images, default for questions<\/td>\n<\/tr>\n<tr>\n<td><code>qwen3:0.6b<\/code><\/td>\n<td>0.5 GB<\/td>\n<td>Tiny classifier for the smart router<\/td>\n<\/tr>\n<tr>\n<td><code>qwen2.5-coder:1.5b<\/code><\/td>\n<td>1 GB<\/td>\n<td>IDE autocomplete<\/td>\n<\/tr>\n<tr>\n<td>a small specialist model (7B)<\/td>\n<td>4.7 GB<\/td>\n<td>Called by name only<\/td>\n<\/tr>\n<tr>\n<td><code>nemotron-3-nano<\/code> (30B)<\/td>\n<td>24 GB<\/td>\n<td>Only in its own sandboxed instance, opt-in per key<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<div class=\"wp-block-group has-pale-cyan-blue-background-color has-background\" style=\"padding-top:28px;padding-right:32px;padding-bottom:28px;padding-left:32px;margin-top:32px;margin-bottom:32px\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-container-core-group-is-layout-40de7cf3 wp-block-group-is-layout-constrained\">\n\n<p class=\"wp-block-paragraph\"><strong>Thinking about private AI for your organisation?<\/strong> We design, build and secure on-premises AI platforms for Australian organisations, with the guardrails this series describes built in from the start.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/catharsis.net.au\/local-ai\/?utm_source=blog&amp;utm_medium=article&amp;utm_campaign=local-ai-series&amp;utm_content=part-01-mid\" target=\"_blank\" rel=\"noreferrer noopener\">Explore Private AI<\/a><\/div>\n\n<\/div>\n\n<\/div><\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"challenges-we-hit\">Challenges we hit<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">The planned inference server didn&#8217;t work<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The day after deployment, the plan fell apart. The vLLM builds we could get crashed on the 4-bit mixture-of-experts kernels, and speculative decoding went with them. Ollama, the fallback, crashed on the GPU for exactly the two architectures we wanted, so the first deployment ran on the CPU at about <strong>15 tokens per second<\/strong>. The models now run fully on the GPU through Ollama, but vLLM never recovered here.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The dead ends left a mess. One vLLM service was shut down as broken. Another, serving a 35B model, kept about <strong>67 GB reserved<\/strong> until it was stopped, and for weeks it stayed first in the routing, so every non-coding request retried against a dead backend first. By the end, <strong>179 GB<\/strong> of downloaded weights sat on disk unserved. We deleted nothing (Part 6 explains why).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">A memory setting froze the machine<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><code>qwen3-coder<\/code> ran with four parallel slots and a 64k <strong>context window<\/strong>, the amount of text a model can consider at once, measured in tokens of roughly three-quarters of a word. We had written that 64k into the model&#8217;s own settings file to stop prompts being cut short, and it quietly overrode the server-wide limit. In Ollama each slot gets the full window and reserves <strong>KV cache<\/strong>, the working memory for its context, so memory grows with context times slots, and the model&#8217;s footprint grew to about <strong>33 GB, roughly 19 GB of weights plus roughly 14 GB of KV cache<\/strong>. With everything else on the same pool, the machine froze hard, with no kernel error, no NVIDIA driver (Xid) error and no OOM kill, and needed a power-cycle. There were five hard freezes in all (Part 3 is the full post-mortem). &quot;Fits, zero margin&quot; turned out to mean &quot;freezes&quot;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A hardware watchdog set to reboot after 60 seconds of a kernel hang didn&#8217;t fire, because something kept resetting it while everything else stalled, and we never proved what. We added an external heartbeat on a device that doesn&#8217;t share the server&#8217;s fate. Two days after the worst freeze, and again the next day, the server <strong>powered itself off<\/strong>, most likely from heat, though we didn&#8217;t prove it. After the restart, a two-second temperature recorder showed one ordinary stock-analysis test run taking the sensors from 48 \u00b0C to 95 \u00b0C in four minutes (Part 7).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The hand-built setup became unsafe to change<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The first build had grown by hand: about 20 containers, more than a dozen models and four compose stacks in four places. A container was recreated from a floating <code>:latest<\/code> tag, silently upgrading the model server a few days before the first freeze, and a lab backup job had quietly stopped. There was no history and no rollback, and a client could reach a model three ways (<a href=\"https:\/\/catharsis.net.au\/blog\/llm-gateway-one-door-for-every-model\/\" target=\"_blank\" rel=\"noreferrer noopener\">Part 2<\/a>).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We fixed it in five phases, the core in three days: stabilise, make a Git repository the single source of truth, write an operating contract, add observability, then close the side doors and give every client its own key. We removed nothing and only disabled things, recreated one container at a time and exposed nothing to the internet.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"the-result\">The result<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For private, everyday work such as drafting, extraction, OCR and internal questions and answers, local was the right call. For public data, or work that needs the quality of the strongest frontier models, the cloud still fits.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Today only <code>qwen3-coder<\/code> and <code>gemma4<\/code> stay loaded. Measured together at one request slot each, they take <strong>37.8 GB<\/strong> (20.3 GB and 17.5 GB), against about 33 GB for <code>qwen3-coder<\/code> alone before the fix. Both run from tags with a <strong>32,768-token<\/strong> context written in, although they support far more, and we keep at least 40 GB free. Everything else is cold: callable by name, loaded on demand, never picked automatically.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Everything except the gateway listens on the server itself (loopback), and the gateway and dashboards need authentication. No data leaves the server, and each call costs nothing. The Telegram agent answers in 3 to 7 seconds (Part 9). The operating rules fit on one page, such as &quot;never change the box by hand&quot;, &quot;no <code>:latest<\/code> tags&quot; and &quot;measure the memory budget, don&#8217;t derive it&quot;, and each exists because something went wrong once.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It still costs something. You become the operator. With one slot per model a second caller waits: over seven days of test traffic, a colliding caller waited a median of <strong>18 seconds<\/strong>, and 90 per cent of waits were under 58 seconds. Quality isn&#8217;t at the level of the strongest cloud models, although a 30B local model is very capable for code, routing, extraction and summarising. The stock-analysis experiment, whose data is public anyway, may use <strong>free<\/strong> cloud models for its two &quot;deep thinking&quot; calls per run, with a local fallback. Free cloud tiers may log or train on prompts, which is why nothing else goes near them. The stack is young on new hardware, so pin every image and soak upgrades in a sandbox first. If we started again, we would use two servers, not one, and set up monitoring that someone reads.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"do-it-yourself-a-base-local-ai-server\">Do it yourself: a base local AI server<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What you need:<\/strong> a Linux server with a GPU with at least 24 GB of VRAM, or at least 32 GB of unified memory, for a 30B 4-bit model. Install the NVIDIA driver first; <code>nvidia-smi<\/code> (or your GPU vendor&#8217;s equivalent) should list the GPU. You also need a terminal with <code>sudo<\/code> and about an hour.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n\n<li>\n<p><strong>Install Ollama<\/strong> with its official script, which also creates a systemd service.<\/p>\n<pre><code class=\"language-bash\">curl -fsSL https:\/\/ollama.com\/install.sh | sh\nsystemctl status ollama\n<\/code><\/pre>\n<p>Check: the service shows <code>active (running)<\/code> and <code>ollama --version<\/code> prints a version.<\/p>\n<\/li>\n\n\n<li>\n<p><strong>Keep Ollama on loopback.<\/strong> By default it listens only on the machine itself. Don&#8217;t set <code>OLLAMA_HOST<\/code> to listen on all interfaces; in Part 2 a gateway becomes the only door.<\/p>\n<pre><code class=\"language-bash\">ss -ltn | grep 11434\n<\/code><\/pre>\n<p>Check: the listening address is <code>127.0.0.1:11434<\/code>, not a wildcard address for all interfaces. From a second machine, <code>curl -m 5 http:\/\/&lt;server&gt;:11434<\/code> is refused or times out.<\/p>\n<\/li>\n\n\n<li>\n<p><strong>Set memory-safe defaults<\/strong> with a systemd override (<code>sudo systemctl edit ollama<\/code>), then <code>sudo systemctl restart ollama<\/code>.<\/p>\n<pre><code class=\"language-ini\">[Service]\nEnvironment=&quot;OLLAMA_NUM_PARALLEL=1&quot;\nEnvironment=&quot;OLLAMA_CONTEXT_LENGTH=32768&quot;\nEnvironment=&quot;OLLAMA_MAX_LOADED_MODELS=2&quot;\n<\/code><\/pre>\n<p>Check: <code>systemctl show ollama --property=Environment<\/code> lists all three. Don&#8217;t skip the context setting: recent Ollama versions pick a default from the memory they see, up to 256k tokens on a large machine. Note that <code>systemctl edit<\/code> reloads systemd; on a server already running GPU containers, read the GPU caveat in <a href=\"https:\/\/catharsis.net.au\/blog\/local-ai-server-freezes-unified-memory\/\" target=\"_blank\" rel=\"noreferrer noopener\">Part 3<\/a> first.<\/p>\n<\/li>\n\n\n<li>\n<p><strong>Pull one model and give it an explicit context.<\/strong> A model can carry its own context setting that overrides the server-wide one; ours did, at 65,536 tokens, because we had put it there. Build a tag with the limit written in.<\/p>\n<pre><code class=\"language-bash\">ollama pull qwen3-coder:30b\nprintf 'FROM qwen3-coder:30b\\nPARAMETER num_ctx 32768\\n' &gt; Modelfile\nollama create qwen3-coder-32k -f Modelfile\n<\/code><\/pre>\n<p>Check: <code>ollama list<\/code> shows both names at about 18\u201319 GB, and <code>ollama show qwen3-coder-32k<\/code> lists <code>num_ctx 32768<\/code>.<\/p>\n<\/li>\n\n\n<li>\n<p><strong>Send a first request<\/strong> to the local API.<\/p>\n<pre><code class=\"language-bash\">curl -s localhost:11434\/api\/generate \\\n  -d '{&quot;model&quot;:&quot;qwen3-coder-32k&quot;,&quot;prompt&quot;:&quot;Say hello in five words.&quot;,&quot;stream&quot;:false}'\n<\/code><\/pre>\n<p>Check: you get JSON with a <code>response<\/code> field. The first call is slower while the model loads.<\/p>\n<\/li>\n\n\n<li>\n<p><strong>Measure memory with the model loaded.<\/strong> Run <code>free -h<\/code> before and after the first request, then <code>ollama ps<\/code>.\nCheck: <code>ollama ps<\/code> shows <code>100% GPU<\/code> under PROCESSOR, a context of 32768 and a SIZE close to 20 GB. (The CONTEXT column is in recent Ollama versions; on older ones use <code>curl -s localhost:11434\/api\/ps<\/code>.) If it says CPU, the model isn&#8217;t on the GPU. Write the measured figure down; it is your real budget, not the file size.<\/p>\n<\/li>\n\n\n<li>\n<p><strong>Write the data rule down<\/strong> in a short operating-rules file kept in Git, starting with &quot;data never leaves the server&quot; and &quot;every model call goes through the gateway&quot;. Add your memory margin.\nCheck: <code>git log<\/code> shows the file, and each later change to it is a reviewed commit.<\/p>\n<\/li>\n\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"key-takeaways\">Key takeaways<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li>A compact desktop AI system like the DGX Spark can carry a small team&#8217;s everyday AI work, as long as you design around its shared memory rather than the model card.<\/li>\n\n\n<li>&quot;Data never leaves the server&quot; works when the gateway&#8217;s configuration enforces it, not people&#8217;s good intentions.<\/li>\n\n\n<li>A plan with zero memory margin is a plan to freeze. Measure what each model reserves.<\/li>\n\n\n<li>Local AI swaps a per-token bill for an operations job. Budget for the operating work as well as the hardware.<\/li>\n\n\n<li>Turn every problem into a written rule that a check enforces, such as &quot;no <code>:latest<\/code> tags&quot; refused at commit.<\/li>\n\n<\/ul>\n\n\n\n<div class=\"wp-block-group has-light-green-cyan-background-color has-background\" style=\"padding-top:28px;padding-right:32px;padding-bottom:28px;padding-left:32px;margin-top:40px;margin-bottom:32px\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-container-core-group-is-layout-40de7cf3 wp-block-group-is-layout-constrained\">\n\n<h2 class=\"wp-block-heading\">Planning private AI?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Catharsis Cyber Security is a Sydney cyber security company. We design, build and secure private AI platforms like the one in this lab, and help you decide where local or cloud AI fits.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/catharsis.net.au\/contact-us\/?type=contact&amp;service=local-ai&amp;utm_source=blog&amp;utm_medium=article&amp;utm_campaign=local-ai-series&amp;utm_content=part-01-contact\" target=\"_blank\" rel=\"noreferrer noopener\">Talk to us about AI<\/a><\/div>\n\n\n<div class=\"wp-block-button is-style-outline is-style-outline--1\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/catharsis.net.au\/secure-ai\/?utm_source=blog&amp;utm_medium=article&amp;utm_campaign=local-ai-series&amp;utm_content=part-01-end\" target=\"_blank\" rel=\"noreferrer noopener\">Explore Secure AI<\/a><\/div>\n\n<\/div>\n\n<\/div><\/div>\n\n\n\n<div class=\"wp-block-group has-background\" style=\"background-color:#f4f6f8;padding-top:28px;padding-right:32px;padding-bottom:28px;padding-left:32px\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-container-core-group-is-layout-40de7cf3 wp-block-group-is-layout-constrained\">\n\n<p class=\"wp-block-paragraph\"><strong>Keep reading the series<\/strong><\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n\n<div class=\"wp-block-button is-style-outline is-style-outline--2\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/catharsis.net.au\/blog\/running-local-ai-in-production-series-guide\/\" target=\"_blank\" rel=\"noreferrer noopener\">Series guide<\/a><\/div>\n\n\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/catharsis.net.au\/blog\/llm-gateway-one-door-for-every-model\/\" target=\"_blank\" rel=\"noreferrer noopener\">Next: Part 2 \u2192<\/a><\/div>\n\n<\/div>\n\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>We built a private AI server in our lab and ran realistic test workloads on it for seven weeks. This post covers why we went local, what ran on the server, what that choice cost, and how to set up the same base server yourself.<\/p>\n","protected":false},"author":2,"featured_media":1088,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[69],"tags":[83,78,81,75,82,80,76,77,79],"class_list":["post-1089","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-ai-lab","tag-data-sovereignty","tag-litellm","tag-local-llm","tag-n8n","tag-ollama","tag-on-premises-ai","tag-private-ai","tag-unified-memory"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11) - Catharsis | Blog<\/title>\n<meta name=\"description\" content=\"Local AI or cloud AI? Why we built a private AI server in our lab, what it cost, when the cloud still fits, and how to set up the base server yourself.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11) - Catharsis | Blog\" \/>\n<meta property=\"og:description\" content=\"Local AI or cloud AI? Why we built a private AI server in our lab, what it cost, when the cloud still fits, and how to set up the base server yourself.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/\" \/>\n<meta property=\"og:site_name\" content=\"Catharsis | Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-16T23:30:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-11-03T09:53:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2026\/11\/local-ai-series-part-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"720\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Sajal Jain\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sajal Jain\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"18 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/\"},\"author\":{\"name\":\"Sajal Jain\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#\\\/schema\\\/person\\\/f0826efb92a7aaf8d46b24715990e099\"},\"headline\":\"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11)\",\"datePublished\":\"2026-08-16T23:30:00+00:00\",\"dateModified\":\"2026-11-03T09:53:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/\"},\"wordCount\":3399,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/local-ai-series-part-01.jpg\",\"keywords\":[\"AI lab\",\"data sovereignty\",\"LiteLLM\",\"local LLM\",\"n8n\",\"Ollama\",\"on-premises AI\",\"private AI\",\"unified memory\"],\"articleSection\":[\"Artificial Intelligence\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/\",\"url\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/\",\"name\":\"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11) - Catharsis | Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/local-ai-series-part-01.jpg\",\"datePublished\":\"2026-08-16T23:30:00+00:00\",\"dateModified\":\"2026-11-03T09:53:10+00:00\",\"description\":\"Local AI or cloud AI? Why we built a private AI server in our lab, what it cost, when the cloud still fits, and how to set up the base server yourself.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#primaryimage\",\"url\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/local-ai-series-part-01.jpg\",\"contentUrl\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/local-ai-series-part-01.jpg\",\"width\":1280,\"height\":720,\"caption\":\"A technician working on a server rack\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/why-run-ai-on-your-own-hardware\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/\",\"name\":\"Catharsis | Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#organization\",\"name\":\"Catharsis | Blog\",\"url\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/01\\\/catharsis_au.jpg\",\"contentUrl\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/01\\\/catharsis_au.jpg\",\"width\":215,\"height\":60,\"caption\":\"Catharsis | Blog\"},\"image\":{\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/#\\\/schema\\\/person\\\/f0826efb92a7aaf8d46b24715990e099\",\"name\":\"Sajal Jain\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/01\\\/Sajal-copy-96x96.jpg\",\"url\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/01\\\/Sajal-copy-96x96.jpg\",\"contentUrl\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/01\\\/Sajal-copy-96x96.jpg\",\"caption\":\"Sajal Jain\"},\"description\":\"Cyber Security Enthusiast | OSCP | CEH | CHFI | CNSS | Master of IT Security &amp; Master of IT Management\",\"url\":\"https:\\\/\\\/catharsis.net.au\\\/blog\\\/author\\\/sajal-jain\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11) - Catharsis | Blog","description":"Local AI or cloud AI? Why we built a private AI server in our lab, what it cost, when the cloud still fits, and how to set up the base server yourself.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/","og_locale":"en_US","og_type":"article","og_title":"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11) - Catharsis | Blog","og_description":"Local AI or cloud AI? Why we built a private AI server in our lab, what it cost, when the cloud still fits, and how to set up the base server yourself.","og_url":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/","og_site_name":"Catharsis | Blog","article_published_time":"2026-08-16T23:30:00+00:00","article_modified_time":"2026-11-03T09:53:10+00:00","og_image":[{"width":1280,"height":720,"url":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2026\/11\/local-ai-series-part-01.jpg","type":"image\/jpeg"}],"author":"Sajal Jain","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Sajal Jain","Est. reading time":"18 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#article","isPartOf":{"@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/"},"author":{"name":"Sajal Jain","@id":"https:\/\/catharsis.net.au\/blog\/#\/schema\/person\/f0826efb92a7aaf8d46b24715990e099"},"headline":"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11)","datePublished":"2026-08-16T23:30:00+00:00","dateModified":"2026-11-03T09:53:10+00:00","mainEntityOfPage":{"@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/"},"wordCount":3399,"commentCount":0,"publisher":{"@id":"https:\/\/catharsis.net.au\/blog\/#organization"},"image":{"@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#primaryimage"},"thumbnailUrl":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2026\/11\/local-ai-series-part-01.jpg","keywords":["AI lab","data sovereignty","LiteLLM","local LLM","n8n","Ollama","on-premises AI","private AI","unified memory"],"articleSection":["Artificial Intelligence"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/","url":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/","name":"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11) - Catharsis | Blog","isPartOf":{"@id":"https:\/\/catharsis.net.au\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#primaryimage"},"image":{"@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#primaryimage"},"thumbnailUrl":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2026\/11\/local-ai-series-part-01.jpg","datePublished":"2026-08-16T23:30:00+00:00","dateModified":"2026-11-03T09:53:10+00:00","description":"Local AI or cloud AI? Why we built a private AI server in our lab, what it cost, when the cloud still fits, and how to set up the base server yourself.","breadcrumb":{"@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#primaryimage","url":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2026\/11\/local-ai-series-part-01.jpg","contentUrl":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2026\/11\/local-ai-series-part-01.jpg","width":1280,"height":720,"caption":"A technician working on a server rack"},{"@type":"BreadcrumbList","@id":"https:\/\/catharsis.net.au\/blog\/why-run-ai-on-your-own-hardware\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/catharsis.net.au\/blog\/"},{"@type":"ListItem","position":2,"name":"Local AI or Cloud AI? Why We Built a Private AI Server (Part 1 of 11)"}]},{"@type":"WebSite","@id":"https:\/\/catharsis.net.au\/blog\/#website","url":"https:\/\/catharsis.net.au\/blog\/","name":"Catharsis | Blog","description":"","publisher":{"@id":"https:\/\/catharsis.net.au\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/catharsis.net.au\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/catharsis.net.au\/blog\/#organization","name":"Catharsis | Blog","url":"https:\/\/catharsis.net.au\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/catharsis.net.au\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2019\/01\/catharsis_au.jpg","contentUrl":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2019\/01\/catharsis_au.jpg","width":215,"height":60,"caption":"Catharsis | Blog"},"image":{"@id":"https:\/\/catharsis.net.au\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/catharsis.net.au\/blog\/#\/schema\/person\/f0826efb92a7aaf8d46b24715990e099","name":"Sajal Jain","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2019\/01\/Sajal-copy-96x96.jpg","url":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2019\/01\/Sajal-copy-96x96.jpg","contentUrl":"https:\/\/catharsis.net.au\/blog\/wp-content\/uploads\/2019\/01\/Sajal-copy-96x96.jpg","caption":"Sajal Jain"},"description":"Cyber Security Enthusiast | OSCP | CEH | CHFI | CNSS | Master of IT Security &amp; Master of IT Management","url":"https:\/\/catharsis.net.au\/blog\/author\/sajal-jain\/"}]}},"_links":{"self":[{"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/posts\/1089","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/comments?post=1089"}],"version-history":[{"count":4,"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/posts\/1089\/revisions"}],"predecessor-version":[{"id":1216,"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/posts\/1089\/revisions\/1216"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/media\/1088"}],"wp:attachment":[{"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/media?parent=1089"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/categories?post=1089"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/catharsis.net.au\/blog\/wp-json\/wp\/v2\/tags?post=1089"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}