Can AI Automate Business Workflows Without Sending Data Out? (Part 10 of 11)

Chatbots get the attention, but most of what businesses ask of private AI is quiet work: reading requests, turning PDFs into tables, answering questions from documentation. Here is how we wired those workloads in our lab, how they broke, and how to build your own.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

The problem: can quiet AI work happen without the data leaving?

Businesses rarely ask for "a chatbot". They want help reading emails and pulling out what the sender wants, turning PDFs into a spreadsheet, or answering questions from documentation without making things up. That work touches invoices, enquiries and internal documents, which is exactly the data you don’t want leaving your hands.

So in our lab we built test versions of those jobs on sample data. n8n, an open-source workflow automation tool, did the plumbing, and local models did the reading, extracting and answering. Two risks showed up quickly. Automations can stop working without any error, which means work silently not getting done. And a convenient retry setting can quietly send a document to a cloud provider, which is a privacy failure nobody sees.

What we were aiming for

• Every model call traceable to a named client, so a broken workflow or runaway job is easy to find.
• Documents and requests never leaving the server, enforced in code rather than by habit.
• Heavy, rarely used models that don’t sit in memory all day.
• Workflows we can prove complete end to end, especially after upgrades.

How we approached it

The simplest option was to let every tool call the model server, Ollama, directly. That is how most tutorials start, and early in the lab it loaded models nobody had planned for, with no record of who asked. Cloud APIs fail the privacy goal.

We chose to put one gateway, LiteLLM (Part 2), in front of everything. It holds the keys, the input budget for each model, the queue and the trace. Everything except the gateway listens on the server itself (loopback), and the gateway and dashboards need authentication. Every workflow, chat window, agent and lab device is a client of that gateway with its own key. The server is a compact desktop AI server with roughly 120 GB of unified memory shared by the CPU and GPU, so memory, not disk, is the budget.

What we built

Workflows with their own key

n8n ran a few dozen active test workflows, plus drafts kept for reference. Document request workflows use a local 12-billion-parameter model to pull fields out of a sample request, draft a reply for a person to review, and file it. Mailbox triage workflows read a synthetic mailbox and sort each message into a category.

n8n has its own gateway key and never uses the admin key, and its old direct route to Ollama closed when Ollama moved to loopback. Usage and failures are reported per caller, and revoking one client leaves the others alone.

OCR that loads only when needed

Document OCR turns scanned PDFs and images into Markdown with tables. The model is an open 3-billion-parameter vision-language model served behind a small API. A tester uploads a file through an n8n form and gets Markdown back. We tested it on sample invoices, test PDFs and public documents.

A 6–7 GB model idle most of the day shouldn’t stay resident, so the API loads it on the first job, handles one job at a time, caps a document at 50 pages and unloads after 10 minutes idle. That is part of why the memory budget keeps at least 40 GB free.

Knowledge bases as models on the gateway

A general model answers questions about documentation confidently and sometimes wrongly. So we built two knowledge bases from open-source project documentation using RAG, retrieval-augmented generation: the system first finds the most relevant passages in trusted documents and puts them in the prompt, so the model answers from them. The search uses embeddings, which turn each passage into a vector so that passages with similar meaning sit close together. Qdrant, a vector database, holds one collection per knowledge base and finds the nearest passages quickly.

Each knowledge base is a small worker container. It re-ingests its documentation on a schedule, retrieves and reranks passages, and asks gemma4 for an answer with citations, falling back to qwen3-coder, another local model.

The decision we would copy is registering both workers as models on the gateway, each under its own name. The workers already spoke the OpenAI chat API, so every client got them for free. The lab’s chat UI lists them, n8n and the Telegram agent (Part 9) call them with their own keys. They are marked routable: false, so the smart router (Part 5) never sends a general question to a documentation set. The workers listen on loopback only, and their own model calls go back through the gateway on a key that allows only the two local models.

One chat window and every device on the same door

Open WebUI, the lab’s ChatGPT-style interface, lost its direct connections to Ollama and the workers. Its only backend is the gateway, on its own key, so every chat lands in the same trace as the workflows. Each lab device gets its own named key, so a lost device means revoking one key. Coding assistants in the editor point at the gateway too, with autocomplete on a 1.5B model, because a big model behind autocomplete gets a request on almost every keystroke and pushes everyone else’s model out of memory.

No fallback to the cloud

LiteLLM can retry a failed or slow request on another model. Give a document-reading model a short timeout and a cloud fallback, and whenever the GPU slot is busy the gateway quietly re-sends the document to the cloud.

A fallback belongs to the model, not the caller, so limiting which keys may use cloud models doesn’t help. The rule is structural. Local models fall back only to each other, and a pre-commit check refuses any gateway config with a local-to-cloud fallback, a default fallback naming a cloud model, or a fallback naming a model that doesn’t exist, since a typo would otherwise switch the check off. It runs on every commit, whoever wrote the config. The one written exception is the stock-analysis test in Part 8, whose two deep reasoning calls may use free OpenRouter models on their own key, with a local fallback and every call logged. Free tiers may log or train on prompts, so that step only ever sees public market data.

Challenges we hit

Forms that never finished after an upgrade

A tester reported the stock-analysis form "is doing analysis for 20 mins and no output". The analyses had finished, in 325 and 415 seconds, but the browser never left the form page and the executions sat in waiting forever. A recent n8n upgrade expected the Form Trigger’s path as a top-level parameter, and our forms still had it under options. The URL still answered from the old registration, but the completion step never fired. A throwaway form-and-wait workflow proved it, completing in 8 seconds once path moved. We fixed both forms through n8n’s API and watched a test request finish in a real browser.

Tool calls that vanished

The same day, agent jobs failed with no errors, because the gateway used Ollama’s generate endpoint, which has no native tool calling; switching chat models to ollama_chat/ fixed it (Part 8). Autocomplete stays on the generate endpoint, which fill-in-the-middle needs.

Blank answers for every streaming client

The first live test of the knowledge bases returned nothing. The workers ignored stream: true and sent one complete JSON response, while the gateway expected server-sent events and passed on an empty chunk and [DONE]. Every streaming client saw a blank answer while non-streaming calls worked. We fixed the worker to return OpenAI-style chunks of about 240 characters when asked to stream.

Budgets worked out instead of measured

Two smaller models hit their 6,000-token budget with test prompts of up to 10,832 tokens, so we gave them 32k-context versions and measured the cost. One took 3.22 GB more memory and the other only 0.80 GB, because it uses sliding-window attention. The embedding model for the gateway’s compaction, nomic-embed-text, had been given 4,096 tokens by arithmetic. Two inputs identical for 2,400 tokens and different only at the end produced byte-identical vectors, because it clamps at 2,048 and truncates silently. The budget is now 2,048.

OCR quirks

The faster serving engine, SGLang, didn’t work on this server, so OCR runs on plain Transformers in bfloat16, because fp16 broke the masking operations. Without repetition limits (no_repeat_ngram_size), the output repeated one phrase over and over. A "remove unused images" clean-up in a container UI deleted the OCR base image, so that UI is now view-only. And when the OCR container lost GPU access (Part 3), the restore script missed it because it only knew about Ollama. It now finds every GPU container.

The result

Every model call in the lab is now traceable to a named client, and no document path leads off the server.

What it still costs is latency, since a second caller queues behind the first on the one-slot GPU. Quality has a ceiling: a 3B OCR model and 12B–30B extraction models are good for first drafts, with a person checking anything before it is used. Upgrades break things quietly, so someone has to own re-validation. Now and then a knowledge-base answer comes back with sources but no text, and asking again fixes it. Next on our list are scheduled smoke tests, better OCR fidelity on a larger sample set, and a real fix for that blank answer.

Do it yourself: a document workflow that stays on the server

What you need: a Linux server with a GPU, Docker, Ollama and LiteLLM set up as in Part 1 and Part 2, plus self-hosted n8n, Qdrant, an OCR model you can serve over HTTP, and a reverse proxy with TLS and authentication. Allow a day.

1. Run n8n and Qdrant in Docker Compose, pinned by image digest, with restart: unless-stopped, ports published on loopback only, and a Qdrant API key even on loopback, since other containers can reach it. Testers and approvers need to reach n8n’s forms, so put n8n behind an authenticated reverse proxy with TLS, and tell n8n its public address:

   services:
     n8n:
       image: n8nio/n8n@sha256:<digest>
       restart: unless-stopped
       ports: ["127.0.0.1:5678:5678"]
       environment:
         WEBHOOK_URL: https://<your-n8n-hostname>/
     qdrant:
       image: qdrant/qdrant@sha256:<digest>
       restart: unless-stopped
       ports: ["127.0.0.1:6333:6333"]
       env_file: [qdrant.env]   # mode 600; sets the Qdrant service API key
   

   qdrant.env sets QDRANT__SERVICE__API_KEY to a long random value; keep the same value in QDRANT_API_KEY for the commands below. Check: curl -fsS 127.0.0.1:5678/healthz/readiness returns HTTP 200, and curl -s 127.0.0.1:6333/collections -H "Authorization: Bearer $QDRANT_API_KEY" answers. From another machine, only the proxy’s HTTPS address answers, and only after you log in.

2. Give n8n its own gateway key with /key/generate, allowed only the models it needs. In n8n, create an OpenAI credential whose base URL is the gateway’s /v1 address.

   Check: a test chat node returns a reply, and the call appears under n8n’s key in the gateway’s logs.

3. Serve OCR behind a small API that loads on demand. Load the model on the first request, run one job at a time behind a lock, cap the pages, and free it after a few idle minutes. If the model needs trust_remote_code, pin the model revision. In outline:

   lock, model, last_used = asyncio.Lock(), None, 0.0
   
   @app.post("/ocr")
   async def ocr(file: UploadFile):
       global model, last_used
       async with lock:
           model = model or load_model()          # first job pays the load time
           pages = to_images(await file.read())[:MAX_PAGES]
           text = "\n\n".join(run_ocr(model, p) for p in pages)
           last_used = time.time()
       return {"markdown": text}
   
   async def unload_when_idle():                  # started as a background task
       global model
       while True:
           await asyncio.sleep(60)
           async with lock:
               if model and time.time() - last_used > IDLE_SECONDS:
                   model = free(model)            # returns None and empties the GPU cache
   

   Check: posting a sample PDF returns Markdown, and the model’s memory is released after the idle period.

4. Build the workflow: a Form Trigger for the upload, an HTTP Request node to the OCR API, then a model call through the gateway that extracts the fields you need as JSON and drafts a reply.

   Check: a sample invoice produces valid JSON with every field filled or explicitly empty.

5. Make a person approve the draft. Add a Wait node that resumes "On Form Submitted", send the approver its resume-form URL (for example by email from the workflow), and only file or send after it.

   Check: the execution sits in waiting until someone approves, then completes.

6. Create a knowledge-base collection sized to your embedding model. For nomic-embed-text that is 768 dimensions.

   curl -X PUT 127.0.0.1:6333/collections/docs -H "Authorization: Bearer $QDRANT_API_KEY" \
     -H 'Content-Type: application/json' \
     -d '{"vectors": {"size": 768, "distance": "Cosine"}}'
   

   Check: curl -s 127.0.0.1:6333/collections/docs -H "Authorization: Bearer $QDRANT_API_KEY" shows status green.

7. Chunk, embed and load the documents. Split them into chunks well under the embedding model’s measured limit, embed through the gateway’s /v1/embeddings, and store each chunk with its source. nomic-embed-text expects a search_document: prefix on chunks and search_query: on questions.

   Check: at your maximum chunk size, two inputs that differ only in their last sentence give different vectors (proving nothing is cut), and the collection’s point count matches your chunk count.

8. Serve the knowledge base as a model. A small worker with an OpenAI-compatible chat endpoint that retrieves passages, asks a local model through the gateway, and streams when asked. In outline:

   @app.post("/v1/chat/completions")
   async def chat(req: dict):
       question = req["messages"][-1]["content"]
       hits = qdrant.query_points("docs", query=embed("search_query: " + question), limit=6).points
       answer = await gateway_chat(LOCAL_MODEL, cited_prompt(question, hits))
       if req.get("stream"):
           return StreamingResponse(as_sse_chunks(answer), media_type="text/event-stream")
       return completion_json(answer)
   

   Register it on the gateway under its own name, and keep it out of automatic routing.

   Check: ask it the same question streamed and unstreamed, and both return text with sources.

9. Block cloud fallbacks in a pre-commit hook that fails when any local model’s fallback names a cloud model or a model that doesn’t exist.

   Check: add a fake local-to-cloud fallback and confirm the commit is refused.

10. Re-validate after every upgrade. Enable n8n’s public API, then re-activate each active workflow with POST /api/v1/workflows/<id>/activate and an X-N8N-API-KEY header; it reports validation errors. Submit one test form end to end.

    Check: the form reaches its completion page, and a request with tools returns tool_calls.

Key takeaways

• Send workflows, chat, agents and devices through one gateway, each with its own key.
• Expose knowledge bases as models, so every client you already have can use them.
• Load heavy, rarely used models on demand and unload them when idle, because memory is the budget.
• Re-validate after every upgrade, and test the real shape of the request: streamed, with tools, through the gateway. An endpoint that answers is not a workflow that completes.
• Enforce "no local-to-cloud fallback" in code. A fallback belongs to the model, so it applies to every caller.