Can Local AI Agents Analyse Stocks? How We Tested and Scored Them (Part 8 of 11)

We ran an open-source team of AI analysts, debaters and managers on our lab AI server, with public market data as the test workload. Getting a run to finish took fixes to tool calling, token counting and market data. Then we made the agents keep score on themselves, and the first score was not flattering.

This series documents a private AI server we built and stress-tested in our lab. Workloads and data are synthetic or generalised; the problems, measurements and fixes are real.

General information only, not financial advice. This lab experiment used public market data to test AI agents; it doesn’t consider anyone’s objectives, financial situation or needs, and Catharsis doesn’t hold an Australian Financial Services Licence.

The problem: an agent team that failed silently

TradingAgents is an open-source project that splits stock analysis across a team of AI agents, like a research desk: analysts gather data with tools, researchers debate, and managers decide. A tool call is the model asking the program to run a function, such as "fetch indicators for this ticker". A run makes about 30 "quick" calls, many using tools, and 2 "deep" calls for the final decisions. We chose it as a stress test for the lab server, a compact desktop AI server with roughly 120 GB of unified memory shared by the CPU and GPU, because long, tool-heavy, number-heavy conversations are what break a local setup.

It broke. Runs died with a recursion error while the gateway logged every call as a success. Token estimates were badly wrong. Some analyses ran on missing or zero-price data and still produced confident reports.

Even working runs raised a harder question. Of 31 runs, 77% were Buy, and one test ticker had been analysed seven times over three weeks with "Buy" every time. A Buy from a system that nearly always says Buy tells you little. The same risks apply to any agent a business deploys: a run that looks successful in the logs but did nothing, and a confident report built on missing data.

What we were aiming for

• Runs that finish on local models, through the gateway.
• Loud failures: bad data, loops and missing sources stop the run with a clear message.
• An honest score for the agents beside a naive baseline, under rules that can’t flatter them.
• Cloud models only for public data, with a local fallback and a record of every call.

How we approached it

We moved TradingAgents off direct Ollama access and onto the gateway from Part 2, with its own key, so every call is counted, budgeted and traced. Rather than fork the framework, we patch it at import time.

Quick, tool-using calls stay local. The two deep calls may use free cloud models, because their inputs are only public market data and the agents’ debate. Free tiers may log or train on prompts, so the lab never sends real or private data to them.

The scoring ledger is an evaluation of the agents, not a trading record, and its rules came first so the numbers couldn’t be bent afterwards.

What we built

Each piece below answers a failure described under the challenges further down; we list the finished design first so you can copy it.

Keeping tool results the agent still needs

Agents lost track of what they had already fetched and asked again, endlessly. So every message is now kept, every tool call stays paired with its result, big outputs are cut down in place, and an "already called" index survives even a poor summary. A loop breaker answers an identical tool call once from memory. After 16 tool-calling turns, an analyst’s next call goes out without tools, so all it can do is write its report. The 16 was measured, not picked: the worst healthy analyst on record used 13.

Checking market data before the agents start

Some symbols came back with no prices at all, and the agents analysed them anyway. Now the service counts valid closing prices before any model is called. With none, the job ends in seconds. With fewer than 200 trading days, every agent is told how much history exists. A bar with no closing price is dropped, never forward-filled, because a forward-filled close is a price that never happened.

A web app with history that lasts

Analyses used to start from an n8n form, where a running job looked like a hung one and history vanished on every deploy. A separate web app now keeps history in Postgres, streams a live timeline, and shows an explicit queue: one running, three waiting, and a fifth refused with the running job’s name. Each run links to its trace (Part 7).

A ledger that keeps score

The ledger compares each past call with the later price move in the test window:

• The reference is the close on the run date, the last price the model could have seen. Using the same day’s open would be look-ahead.
• Results are checked at +1, +5 and +20 trading days, counted from the price series.
• A Hold is never scored: there’s no defensible threshold for "correctly did nothing".
• Pending and missing data are named exclusions, never 0%.
• Every figure sits beside "always Buy" over the same rows, plus an equal-weighted basket of everything analysed.

Cells are coloured by whether the call agreed with the later move, not by whether the price rose.

Fenced cloud models for the deep calls

The two deep calls may use free OpenRouter models (two Nemotron variants and a Qwen model), granted to this workload’s key alone, inside four fences:

• The fallback is local. Each cloud model falls back to the local qwen3-coder through the gateway. OpenRouter’s allow_fallbacks: false only stops it switching to another provider of the same model; the model fallback is the gateway’s. Rate-limit errors aren’t retried.
• The prompt is compacted to the local budget before it leaves, so a fallback always fits.
• The OpenRouter key had a $0 credit limit, and a pre-commit check refused any non-free model ID and any local-to-cloud fallback.
• Every call is logged as cloud, cloud-FAILED or LOCAL-fallback, with alerts at 90% of the daily free allowance or when half the calls fall back.

The deep model is chosen when a run starts, from the ledger’s quality order once each candidate has 20 scored calls. It falls back to local when the free allowance runs low, and moves off the local GPU when the server is hot or the queue is long; the reason is logged on the run’s timeline.

Challenges we hit

Tool calls that never came back

Runs died after about 3.5 minutes with GraphRecursionError: Recursion limit of 100 reached. The gateway mapped the models with the ollama/ prefix, which uses Ollama’s /api/generate endpoint, and that endpoint has no native tool calling. The gateway logged 50 successful LLM calls and zero errors. The fix was one prefix: ollama_chat/, which uses /api/chat. A full analysis then finished in 300 seconds.

Token estimates that were 2.79 times too low

One night, in the early hours, a run’s prompts climbed from 50k to 65,007 tokens and the server hard-froze: the context window setting that let a 65k prompt fit had reserved too much memory (Part 3). The gateway gained an admission check that refuses oversized prompts (Part 4), and TradingAgents got a client-side trimmer that condensed the middle of its transcript (75,221 tokens down to 9,350 in one case).

Then no run could finish, and our diagnosis was wrong twice. We blamed the tool definitions; measured the gateway’s way, each was under 1,200 tokens. The real gap was digits. Market data is prices, volumes and dates, and these tokenisers spend about one token per digit. On an actual failed transcript, the trimmer’s characters-divided-by-four estimate said 22,729 tokens where the gateway counted 63,437. That’s why halving the budget had done nothing.

Condensing caused endless loops

With counting fixed, runs still hit the recursion limit. An analyst calls nine tools in one turn. The next turn is over budget, so the condenser replaces those nine results with 600 tokens of prose. Without the answers it just received, the analyst calls the same tools again, and again. For a tool-calling agent, summarising the middle of the conversation removes exactly what it needs.

Market data that was quietly wrong

The price source’s rate limiting was self-inflicted: the cache filename contained both window dates, so each day’s first run re-downloaded every history. A cache keyed on the symbol fixed it.

Macro data had never worked. The macro tool uses FRED, the St. Louis Fed’s public data service, which needs a free API key; without one, every analysis ran without interest rates or inflation, and the error just read "unavailable". Worse, a gold symbol was analysed from zeros. The price source doesn’t cover it, and the pipeline produced a confident report anyway. Even "above zero" wasn’t a strict enough check, because that series held a close of 9.5e-12.

Thin history and a 23-minute loop

A test ticker with only 71 trading days of history came back Hold, Buy, Buy, (failed), Sell over five runs. In the failed run, the analyst wanted indicators needing 200 days and called the same tool for 23 minutes, over 19 turns and 356,384 tokens, until the gateway refused it.

The result

With tool pairs kept, a run finished with 56 calls and zero refusals. Later the main Ollama instance moved to one slot with a 32,768-token context and a 26,000-token gateway budget, the gateway’s own compaction kept tool pairs, and we retired the client-side trimmer. The proof run took 263 seconds, with a largest call of 21,131 tokens. With a FRED key, the macro tool returned real series.

The first ledger evaluation, taken the day the web app shipped:

The agents didn’t beat the naive baseline. The whole gap is one call: the only Sell was followed by a rise. On 40 runs the next day it was 50% against 55% at one day (20 scored) and 27% against 36% at five days (11 scored). Ten or twenty calls prove nothing, the calls aren’t independent and nothing is risk-adjusted; the page says so on every view.

A head-to-head on three test tickers changed only the deep model. The local qwen3-coder once suggested putting most of a portfolio into one stock, exactly the output nobody should act on. One Nemotron model gave conditional calls with a median deep call of 11 s (the other took 61 s). About one call in four hit free-tier limits and ran locally, and no run failed. It’s a signal, not a controlled test.

It costs GPU time and heat: one run took the server from 48 °C to 95 °C in four minutes, and other callers queue behind it. One upstream fault remains: the framework discards an analyst’s finished report when the same message carries a tool call. Next we want enough scored calls for the ledger to pick the model, and a second market-data source.

Do it yourself: TradingAgents on local models through a gateway

What you need: a Linux server with Ollama and a tool-capable model with a fixed context (we used a 32k tag of qwen3-coder, built as in Part 3), LiteLLM set up as in Part 2, a recent Python, a free FRED API key and about two hours.

1. Install TradingAgents. Clone its GitHub repository, create a virtual environment (python3 -m venv .venv), install it as its README describes, and run its command-line tool once with the defaults. Config options change between versions, so read default_config.py in yours. Check: the CLI starts and lists the analysts you can choose.

2. Map two gateway models with the chat prefix in LiteLLM’s config.yaml. analyst-quick serves the tool-using analysts and debaters; analyst-deep serves the two managers’ final decisions. Both can point at the same local model to start:

   model_list:
     - model_name: analyst-quick
       litellm_params:
         model: ollama_chat/qwen3-coder-32k
         api_base: http://127.0.0.1:11434
       model_info:
         max_input_tokens: 26000
     - model_name: analyst-deep
       litellm_params:
         model: ollama_chat/qwen3-coder-32k
         api_base: http://127.0.0.1:11434
       model_info:
         max_input_tokens: 26000
   

   Check: a request to analyst-quick with a tools array returns tool_calls, not plain text.

3. Give the workload its own virtual key, limited to its models:

   curl -s "$GATEWAY/key/generate" -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H 'Content-Type: application/json' -d '{"models": ["analyst-quick", "analyst-deep"]}'
   

   Check: the key works for analyst-quick and is refused for other models.

4. Point TradingAgents at the gateway. In its config, set llm_provider to the OpenAI-compatible option, backend_url to the gateway’s /v1 address, quick_think_llm to analyst-quick and deep_think_llm to analyst-deep, and pass the virtual key as OPENAI_API_KEY. Check: one analysis completes, and the gateway’s spend log shows its calls under that key.

5. Count tokens the way your gateway does. Use the same counter your gateway’s admission check uses, never characters divided by four. Check: on a price table, the real count is well above len(text)//4.

6. Validate data before any model call. Recent yfinance versions return a two-level column index even for one ticker, hence the squeeze:

   closes = df["Close"].squeeze("columns").dropna()
   valid = closes[closes > 1e-6]
   if valid.empty:
       raise SystemExit("no valid closing prices for this symbol")
   history_note = f"Only {len(valid)} trading days of history." if len(valid) < 200 else ""
   

   Check: an unsupported symbol fails in seconds, with no model call.

7. Add the FRED key as FRED_API_KEY, or the variable your TradingAgents version documents, and treat "unavailable" from any data tool as an error, not a result. Check: a macro call returns a real series; without the key, the run fails loudly.

8. Break tool loops. Answer identical tool calls from a per-run cache, and after a fixed number of tool turns (start at 16, then measure your own runs) call the model without tools. Keep LangGraph’s recursion limit as a backstop. Check: a thin-history ticker ends with a report, not a recursion error.

9. Keep a ledger and a baseline:

   CREATE TABLE decisions (run_id text PRIMARY KEY, ticker text, run_date date,
                           decision text, ref_close numeric);
   

   Nightly, look up the close +1, +5 and +20 trading days after run_date, using the same price source as the run so reference closes match. Score each Buy or Sell, skip Holds, and compute "always Buy" on the same rows. Check: the report shows the agents beside the baseline, with exclusions named.

Key takeaways

• Check the endpoint, not only the model. A tool-calling agent on an endpoint without native tool support fails silently.
• Never estimate tokens from character counts on numeric data. Here the estimate was 2.79 times too optimistic.
• Don’t summarise away tool results an agent is still using. Cut them down in place and keep each call paired with its result.
• Make missing data loud. Missing macro data and a series of zero prices both produced confident reports.
• Score your agents against a naive baseline, and publish the result even when it’s bad.
• Use free cloud tiers for public data only, with a local fallback and a log line for every call.

Disclaimer: A lab software experiment on public market data. General information only, not financial product advice; it doesn’t recommend any security or strategy.