Skip to content
Barry Wheeler, ND

· Building & Tinkering

I Benchmarked 78 AI Models and Almost Picked the Wrong Winner.

The model with the perfect score couldn't do the job. That's the short version.

I had a handoff document from two days ago telling me to benchmark every model on my infrastructure and route daily traffic to the best one. Primary, thinking, fast — three roles, pick the winner for each. The handoff listed 78 models across Ollama on two boxes plus Amazon Bedrock. It assured me the winner would be obvious by the end. It wasn't.

What I actually ended up with: a top-scoring local model I can't use as primary, a second-best that's now primary, and a Bedrock provider I had to unlock by generating an AWS IAM credential nobody mentioned. All in — about 62 minutes of real runtime, plus a couple hours of me debugging my own scripts. Which is always where the time actually goes.

The setup

Two machines connected by a 10GbE direct link. Machine B — the inference box — has an RTX PRO 6000 Blackwell with 96GB of VRAM and runs 37 Ollama models, many of them large (some of the 235B test variants weigh 142GB each). Machine A is my workstation, an RTX 4090 with 24GB. It runs 16 smaller Ollama models plus PersonaPlex, the full-duplex voice model that eats ~20GB of VRAM whenever it's loaded. On top of that, OpenClaw — the personal AI gateway that routes everything — has access to Amazon Bedrock via a curated subset they call "Mantle" (39 models) and the general Bedrock catalog (125 foundation models if you count inference profiles).

The handoff's "78 models" turned out to be aspirational. After I ran the discovery phase the real testable inventory was 56, and after Phase 1 only 19 were actually healthy enough to bother benchmarking. The rest either timed out, returned empty strings, or didn't load at all.

About those empty strings.

The PING OK test lied to me

Phase 1 sends every model the same prompt: Reply with exactly: PING OK. Cap the output at 10 tokens, measure latency, done. Fast, cheap, ruthless. If a model can't echo two words back, it's out.

Twenty-two models returned EMPTY. I stared at that list for a while before the pattern clicked. Every Qwen3. Every DeepSeek-R1. The GLM-Z1 reasoning model. Kwangsuklee's Opus-distilled reasoning 27B. All of them — the thinking family.

They weren't broken. They were thinking. Thinking models open with a long hidden <think>…</think> block before producing the user-facing answer. My 10-token budget was getting consumed entirely by silent deliberation. By the time Ollama hit num_predict the model had generated exactly zero characters of output. A PING check that rules out thinking models is useless — those are half the models I actually care about.

Fix: add /no_think to the prompt suffix for Qwen-family models, run a recovery pass. Suddenly qwen3:32b is responding in 12-14 seconds at 55 tokens per second. qwen3.5:27b does 54. The "dead" models were alive the whole time.

Phase 2: where speed actually lives

Once I had a working list, I measured sustained throughput on a real clinical prompt — three safety considerations for starting a 55-year-old on testosterone cypionate. Asked for 512 tokens of output, temperature 0.1. Measured the model's own eval_duration, not wall time, so the number reflects generation speed rather than cold-load penalty.

Bedrock wrecked the locals. Not close. Here's the top of the speed table:

  • qwen.qwen3-coder-30b-a3b (Bedrock) — 227 tok/s
  • qwen.qwen3-32b-v1:0 (Bedrock) — 226
  • us.amazon.nova-lite-v1:0 (Bedrock) — 153
  • RogerBen/qwen3.5-35b-opus-distill (Ollama, Machine B) — 141
  • qwen.qwen3-235b-a22b-2507 (Bedrock) — 129
  • zai.glm-4.7-flash (Bedrock) — 127
  • moonshotai.kimi-k2.5 (Bedrock) — 125
  • anthropic.claude-haiku-4-5 (Bedrock) — 102
  • anthropic.claude-sonnet-4-6 (Bedrock) — 38

Machine A was an also-ran at 12 tok/s for every model I tested there, bottlenecked by PersonaPlex hogging most of the VRAM. I wasn't going to shut down my voice model to benchmark, so Machine A was always going to lose on this dimension.

The surprise for me: Bedrock's cold latency is brutal (the first call to Claude Sonnet 4.6 takes 22 seconds to wake up) but once the provider is warm, the tokens-per-second numbers are better than anything I can run locally on a single GPU. AWS is clearly batching requests across tenants and smoothing throughput in ways a single-inference-request setup just can't match.

Phase 3: quality on five OpenClaw-critical tests

Speed is easy to measure and nearly useless on its own. The real question is whether a model can actually do agentic work — tool calling, instruction following with precise constraints, clinical reasoning, code generation, multi-step inference. I gave each top-candidate model the same five tests and scored each out of 10 based on specific heuristics (did the code have the expected function signature, did the clinical response name the right labs, did the tool-call response produce valid JSON, and so on).

Two models tied at a perfect 50/50:

  • RogerBen/qwen3.5-35b-opus-distill — a community-distilled Qwen trained on Claude Opus outputs. 141 tok/s. Local. Free.
  • qwen2.5-coder:32b — also local, 58 tok/s.

The rest of the field clustered at 44/50. Claude Sonnet 4.6, Claude Haiku 4.5, Nova Lite, GLM 4.7 Flash, Qwen3-32B on Bedrock — all tied. Same test, same prompts, same scoring. The gap wasn't small. RogerBen's opus-distill was clearly the winner — fastest local model that also aced every test I threw at it.

I moved to apply the config change. openclaw config set agents.defaults.model to the new winner, restart the gateway, run a smoke test. That's when everything fell apart.

The trap

Here's the first thing I saw when I ran the smoke test against the new default: 

LLM request failed: provider rejected the request schema or tool payload.
rawError: 400 {"error":"registry.ollama.ai/RogerBen/qwen3.5-35b-opus-distill:latest does not support tools"}

OpenClaw's agent pipeline always sends a tool schema with every inference request — that's how the whole skill system works. The model needs to be able to receive a list of available functions and decide whether to call one. Ollama advertises tool support on a per-model basis via the capabilities API, and RogerBen's opus-distill was compiled without it. The benchmark didn't catch this because my quality tests went directly to Ollama's /api/chat endpoint, bypassing the tool schema layer entirely.

So: the highest-scoring model in my benchmark cannot be the default for my agent gateway. Not without rebuilding the Ollama model file to add tool capabilities, which — maybe — but not today.

I rolled the change back. Went with the tied winner, qwen2.5-coder:32b. Ran the smoke test. Got PRIMARY OK. Moved on.

This is the part of benchmarking that isn't in any protocol doc. Raw output quality and production fitness aren't the same thing. A model that crushes five isolated tests can still flunk the integration layer. I should've put "must respond to a request with a tool schema attached" at the top of my Phase 1 connectivity check. Next time.

Bedrock was supposed to be the easy part

I had already tested 17 Bedrock models by this point, so I knew they worked. I'd sent the requests directly via the AWS CLI — aws bedrock-runtime converse — using the credentials on ~/.aws/credentials. Every one of them returned valid clinical text. GLM-4.7 Flash at 127 tok/s, Haiku 4.5 at 102, Nova Lite at 153. All working.

But calling Bedrock through the OpenClaw CLI was a different story. The configured "Mantle" subset routes through an AWS-hosted proxy with an API key that auto-rotates every 12 hours. That worked. The direct amazon-bedrock provider — the one that would let me use Claude Haiku 4.5 directly — returned a cryptic error: 

Error: No API key found for amazon-bedrock.
Use /login or set an API key environment variable.

I ran the login command. It told me it needed an interactive TTY. This is a background agent session. No TTY. Dead end.

Or — was it?

I pulled the OpenClaw amazon-bedrock plugin source and grep'd it for the string "API key." The answer was in one line of a file I didn't know existed: 

const explicitToken = env.AWS_BEARER_TOKEN_BEDROCK?.trim();

OpenClaw checks an env var called AWS_BEARER_TOKEN_BEDROCK before doing anything else. If it's set, it uses it directly. No login needed. The login flow is only there as the convenient UX.

The other thing I didn't know existed until that moment: AWS IAM has a dedicated service for generating long-lived Bedrock Bearer tokens. It's called service-specific-credentials and you create one with a single command: 

aws iam create-service-specific-credential \
  --service-name bedrock.amazonaws.com \
  --user-name revive-admin

The API returns a Bearer secret that's tied to your IAM user's permissions and doesn't expire until you revoke it. I dropped it into a systemd drop-in file for the OpenClaw gateway service, restarted, and ran the smoke test against amazon-bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0. It returned HAIKU OK in 18 seconds. provider: amazon-bedrock — direct, not Mantle, not a fallback. Worked on first try after the env var was in place.

Three aliases added: haiku-4-5, sonnet-4-6, nova-lite. All three verified end-to-end.

The actual clinical test

After the routing was live I threw a real question at Claude Sonnet 4.6 through the new provider path. A 58-year-old on TRT, total T 920, free T 24, hematocrit 54, PSA went from 1.2 a year ago to 2.8 now, ALT at 85. Pick the two most urgent labs to follow up on.

Sonnet flagged the PSA velocity — 133% year-over-year rise on testosterone meets the AUA threshold of 0.75 ng/mL/year for urology referral, regardless of the absolute value — and flagged the hematocrit at 54 as exceeding the 52 threshold for dose reduction or phlebotomy. Three sentences. Numbers right, guidelines named, the word "velocity" used correctly. This is the response I wanted.

I'd trust that as a second opinion in a way I wouldn't trust a random local model. It's also $0.014 for the request. I've seen residents write worse notes.

Lessons I'd like to remember

Don't trust "78 models available." Audit what's actually reachable before you plan phases against it. My real list was 56 on paper, 19 in practice.

Test the production code path in Phase 1, not just the provider endpoint. If your gateway sends tool schemas, your Phase 1 has to send tool schemas. Otherwise you'll find out about the incompatibility after the benchmark has already blessed the wrong model.

Quality and production fitness are orthogonal. Don't conflate them.

Cloud inference is measurably faster than a single local GPU once you get past the cold-start. This was the surprise I hadn't braced for — I had a mental model where local always wins on latency because the data doesn't leave the box. Turns out network round-trip is cheap and AWS batches fast. For steady-state throughput, the cloud wins.

When a CLI refuses to help you non-interactively, go read the source. Half the time the non-interactive path is one env var away.

The config, for posterity

Final aliases on the gateway, as of tonight:

  • primary (default) — ollama/qwen2.5-coder:32b. 50/50 quality, 58 tok/s, supports tools, free, local.
  • reasoning — ollama/qwen3.5:27b with thinking enabled.
  • thinking — amazon-bedrock-mantle/anthropic.claude-opus-4-7.
  • fast — amazon-bedrock-mantle/zai.glm-4.7-flash, 127 tok/s, 44/50 quality.
  • haiku-4-5 — amazon-bedrock/us.anthropic.claude-haiku-4-5, 102 tok/s, 44/50, direct Bedrock.
  • sonnet-4-6 — amazon-bedrock/us.anthropic.claude-sonnet-4-6, 38 tok/s, 44/50, direct Bedrock.
  • nova-lite — amazon-bedrock/us.amazon.nova-lite-v1:0, 153 tok/s, 44/50, direct Bedrock.
  • opus-distill — ollama/RogerBen/qwen3.5-35b-opus-distill:latest. Kept around for direct-prompt work where tool-calling isn't needed. Still the fastest local model on my rig.

Full report is at ~/bench/FINAL_REPORT.md, all 56 raw quality responses are preserved, and the whole thing is reproducible if I want to rerun it next quarter. Which I probably will. Bedrock ships new models every few weeks now.

The thing I keep coming back to: I almost shipped opus-distill as the default. It had the best numbers. It would've looked good in the changelog. And it would have broken every skill invocation in a way that took an hour to diagnose.

Good benchmarks aren't about finding the winner. They're about finding the trap.

References