Anatomy of a Broken Benchmark Runner: How Seven AI Models Fixed (or Didn't Fix) run-code.sh

Introduction

Introduction

There is a question quietly buried inside every task handed to a language model, and it rarely gets asked out loud. Do you trust one model? Do you go looking for the best model? Do you run several models side by side and keep whatever each of them happens to catch? Or — one turn further still — do you hand that whole pile of partial answers to yet another model and ask it to combine them into one? Each answer sounds reasonable on its own. Each is also, on its own, incomplete.

The goal here is to actually run that experiment rather than assume how it comes out. A single broken bash script, with seven distinct bugs of varying severity, is the test case, and the research runs in two rounds.

Round one: one model, unaided. Four models — Sonnet 5, HY3, Qwen3-Max, DeepSeek-V4-Flash — each fix the script working alone, with no knowledge of what the other three are doing. That's the first framing: one model, fixing only what it personally notices.

Best of the four. Once all four independent attempts exist side by side, the next question follows on its own: if you had to ship just one of these four scripts, which is best? Table 1 works that out below — though "best" here will turn out to mean "most complete," not "complete." Even the strongest single script among the four leaves real bugs unresolved.

Round two: each model selects the best from all four. That incompleteness is why a second round exists at all. Seven more models — ling-3.0-flash, Mistral-Medium-3.5, Nemotron-3-Super-120B, Qwen3-Max, DeepSeek-V4-Flash, Gemini Pro, dots-studio-3-note — are each shown the same four fixed scripts from round one and given the same job: pick out whichever fix, from whichever of the four sources, is genuinely the best solution to each bug, and assemble those choices into a single script.

Who actually combined best. All seven believe they've picked the best pieces. That's exactly the open question this round is built to test: does the model whose combination is objectively the strongest also turn out to be the one whose combination reads as the strongest — or does a different one of the seven, working from the exact same four inputs, put together something better than the obvious pick? Table 2 is where that gets settled, not here.

The more important finding, though, sits underneath both tables rather than at the top of either. A script that merely works is not evidence that a model found everything wrong with it; a script that also explains its lineage — which fix came from where, and why it was kept — is a different, rarer kind of evidence. That distinction, competence versus honest stewardship of other people's work, is what these two tables are actually built to expose, and it's where the closing thought of this piece ends up.

Background

run-code.sh is a bash harness that calls codegen_test.py against a list of LLMs, collects a JSON result file per model, and prints a summary leaderboard. The original file had seven real bugs, ranging from cosmetic (stale filenames in comments) to fatal (a summary table that always shows zero, no matter what actually happened). Four models (Sonnet 5, HY3, Qwen3-Max, DeepSeek-V4-Flash) were each asked to fix the original independently. Then seven more models (ling-3.0-flash, Mistral-Medium-3.5, Nemotron-3-Super-120B, Qwen3-Max, DeepSeek-V4-Flash, Gemini Pro, dots-studio-3-note) were asked to look at all four fixes and merge the best of each into one final script.

To judge who actually fixed what — not just who claimed to — every script was:

1. Diffed line-by-line against the original.

2. Syntax-checked with bash -n.

3. Run against a stub codegen_test.py that emits a realistic mix of statuses (PASS, TEST_FAIL, BAD_JSON, SYNTAX_ERROR, RATE_LIMITED) so the summary table's actual output could be inspected, not just read.

4. Run against a real, malformed model list to confirm whether the known-bad data actually still throws errors or not.

For that last step, the article originally used the script's fictional OPENROUTER_FREE preset as the running example. That's been replaced below with the script's OpenRouter preset instead, populated with model IDs that are genuinely live on OpenRouter's free tier as of August 2026 — meta-llama/llama-3.3-70b-instruct:free, qwen/qwen3-coder:free, openai/gpt-oss-120b:free, openai/gpt-oss-20b:free, and nvidia/nemotron-3-ultra-550b-a55b:free — confirmed directly against OpenRouter's own blog and its live model catalog. The bug pattern is identical either way; grounding it in a real, checkable provider just makes the illustration something a reader can go verify themselves instead of trusting a fictional gateway.

Below is what each bug looked like, why it mattered, and then the two results tables.

Bug #1 — Corrupted data rows in OPENROUTER_FREE

The original array had three broken lines mixed in with the working ones:

bash

OPENROUTER_FREE=(
    "meta-llama/llama-3.3-70b-instruct:free"          # <- no "|RPM|RPD" suffix at all
    "qwen/qwen3-coder:free|0|0"
    "openai/gpt-oss-120b:free|0|0"
    "openai/gpt-oss-20b:free|0|"                       # <- RPD field left empty
    "nvidia/nemotron-3-ultra-550b-a55b:free|0|"        # <- RPD field left empty
)

Further down, the loop parses each entry:

bash

IFS='|' read -r MODEL RPM RPD <<< "$ENTRY"
...
if [ "$RPD" -gt 0 ] && ...        # RPD is "" here for the broken rows

Bash's [ "" -gt 0 ] doesn't crash the script (there's no set -e), but it does print bash: [: : integer expression expected to stderr for every broken row, and the on-screen limit display comes out blank instead of 0 RPM / 0 RPD. It's not fatal, but it's exactly the kind of noise that makes people distrust a tool's output — especially awkward here, since OpenRouter's actual rate limiting isn't even per-model in the first place: it's a flat 20 requests/minute, 50/day (or 1,000/day with a $10 top-up) applied across the whole account, which is why every real entry in this list should legitimately read |0|0 and let SLEEP_OVERRIDE set the pace instead.

How I verified it: running each script against this OpenRouter-style list and grepping stderr for integer expression expected.

Bug #2 — No defensive parsing of RPM/RPD

This is the general version of Bug #1: the loop trusts that every |RPM|RPD field is a clean non-negative integer, with no fallback for typos, missing fields, or a user hand-editing the list — and OpenRouter's free-model roster is exactly the kind of list that gets hand-edited often, since the lineup rotates as providers add and pull models. A one-line guard fixes both the specific typos above and any future ones a maintainer introduces next month:

bash

case "$RPM" in ''|*[!0-9]*) RPM=0 ;; esac
case "$RPD" in ''|*[!0-9]*) RPD=0 ;; esac

or, more simply:

bash

RPM="${RPM:-0}"
RPD="${RPD:-0}"

(Note: ${VAR:-0} also fires on an empty string, not just an unset variable — which is exactly what read produces for a missing field. So this simpler one-liner actually neutralizes Bug #1 too, without touching the data at all.)

Bug #3 — No pre-flight check that codegen_test.py exists

The script sets a default:

bash

SCRIPT="${SCRIPT:-codegen_test.py}"

...but never checks the file is actually there before looping over 6–30 models and shelling out to python3 "$SCRIPT" ... for each one. If the file is missing, you get bash's own cryptic error on the first iteration:

python3: can't open file 'codegen_test.py': [Errno 2] No such file or directory

...repeated once per model, with no hint about what to do. A two-line guard before the loop turns that into a single, actionable message:

bash

if [ ! -f "$SCRIPT" ]; then
    echo "Missing script file: $SCRIPT"
    echo "  Put codegen_test.py next to run-code.sh, or set SCRIPT=/path/to/codegen_test.py"
    exit 1
fi

Bug #4 — The summary table reads status keys that don't exist (the critical bug)

This is the one that actually breaks the tool's purpose. codegen_test.py writes real status values like PASS, TEST_FAIL, BAD_JSON, SYNTAX_ERROR, RATE_LIMITED. The original summary script instead does:

python

c = collections.Counter(r["status"] for r in recs)
hall_rate = c["HALLUCINATION"] / n * 100
rows.append((hall_rate, -c["OK"], model, c, n, avg, min(times), max(times)))
...
print(f"{model:<34} {c['OK']:>3} {c['HALLUCINATION']:>5} {other:>5} ...")

Counter returns 0 for a missing key instead of raising an error, so this never crashes — it just silently produces a useless table. I confirmed this by actually running it:

модель                              OK  HALL  иное  галл.%     min     avg     max
------------------------------------------------------------------------------------------
test-model-a                         0     0     3      0%    2.2s    2.8s    3.1s
test-model-b                         0     0     3      0%    2.9s    3.4s    3.9s
test-model-c                         0     0     3      0%    1.1s    1.2s    1.3s

Every model shows OK=0, HALL=0, 100% dumped into "other" — regardless of whether the model actually passed every test or failed every test. This is the whole point of the script (compare models by code-generation quality), and it was completely broken.

The correct fix has to (a) know the real status vocabulary and (b) bucket it sensibly:

python

OK_STATUS     = {"PASS", "OK"}
HALL_STATUS   = {"TEST_FAIL", "HALLUCINATION", "WRONG"}
BROKEN_STATUS = {"BAD_JSON", "SYNTAX_ERROR", "MALFORMED", "PARSE_ERROR", "EXEC_ERROR"}
INFRA_STATUS  = {"RATE_LIMITED", "TRUNCATED", "TIMEOUT", "ERROR", "HTTP_ERROR", "CONNECTION_ERROR", "API_ERROR"}

Bug #5 — No distinction between "totally crashed" and "some attempts failed"

bash

if python3 "$SCRIPT" ... ; then
    :
else
    echo ">>> ERROR on model: $MODEL"
    FAILED+=("$MODEL")
fi

codegen_test.py exits non-zero if even one attempt out of N failed — which is completely normal (a model getting 8/10 right is a good result, not a crash). The original code can't tell the difference between "this model got 8/10" and "this model's API key was wrong and nothing ran." I confirmed by testing: with a mix of pass/fail statuses, every single model got reported under "errors," even ones that mostly passed. That's actively misleading for a benchmark tool — a reader skimming the "errors" list would assume those models are broken.

The fix needs to check whether a results file was actually written before calling it a hard failure:

bash

if [ "$rc" -ne 0 ] && [ ! -s "$JSON_OUT" ]; then
    FAILED+=("$MODEL")          # nothing was recorded — genuine crash
elif [ "$rc" -ne 0 ]; then
    DEGRADED+=("$MODEL")        # results exist, some attempts just failed
fi

Bug #6 — Fragile summary parser

python

with open(path) as f:
    recs = json.load(f)
...
model = recs[0]["model"]

If any single JSON file is truncated, empty, or missing a key (e.g. because the run was killed mid-write), this throws an uncaught exception and kills the entire summary for every model, not just the broken one. The fix is to wrap the risky parts in try/except and use .get() with a fallback instead of bare indexing.

Bug #7 — Stale documentation

The script was clearly renamed from run.sh to run-code.sh at some point, but ~11 comment lines and one error-hint message were never updated:

bash

#   ./run.sh                      # free-tier Gemini, 3 attempts
...
echo "  HOST=https://your-host/v1 PRESET=openrouter ./run.sh"

Copy-pasting these into a terminal simply fails with "no such file." Minor, but it's exactly the kind of thing that makes a script look unmaintained.

Table 1 — Individual "Fixer" Models

Reading this table: no model got everything. Sonnet 5 has the best code hygiene (docs, defensive try/except) but missed both bugs that actually determine whether the tool's output means anything (#4 and #5) — I verified this isn't a nitpick, its summary table is genuinely useless when I ran it. HY3 built the best-designed summary logic but never re-tested it against the actual data it was reading, so it left the one data bug (#1) live. Qwen3-Max is the only one that fixed 4 of 7 outright with zero regressions, including catching that a general defensive guard (#2) is a better fix than patching three specific typos (#1).

Winner: Qwen3-Max, with HY3 a close second.

Table 2 — "Combiner" Models

Each of these was shown all four fixes above and asked to merge the best of each into one final script. All seven were syntax-checked, run through the same fake-status stress test, and re-run against the real, broken OpenRouter-preset data.

Reading this table: every one of the seven combiners produced a functionally correct script — confirmed by actually running each of them, not just reading them. That's the good news: the core bugs (#1–#5) are basically solved territory at this point; none of the seven regressed on them.

What separates them is discipline about the full source material, not just the two "obviously important" fixes. Three combiners — ling-3.0-flash, Mistral-Medium-3.5, and Nemotron-3-Super-120B — merged HY3's and Qwen's work but silently dropped Sonnet 5's documentation cleanup (all three still have 12 leftover ./run.sh references). dots-studio-3-note falls into the same group: functionally solid, same doc-cleanup gap, no attribution. None of these four explain why they kept what they kept.

Qwen3-Max is the only one that produced a genuine audit trail — a table mapping each fix to the model that produced it and the line number it lives on. That is the actual deliverable of a "combine the best of everything" task: not just working code, but traceability for why each piece survived the merge. Gemini Pro is a close second — equally complete technically (it's the only one besides Qwen and DeepSeek to also fix the doc issue), just without the attribution.

Key Takeaways

• No single model, however capable, catches every bug — each one has its own blind spot, and the blind spots don't overlap the same way twice.

• Being strong in one area doesn't mean a model even attempts an unrelated one: the model with the best-designed summary logic never checked its own data for corruption, and vice versa.

• The overall top-ranked model in Table 1 still missed a bug that a lower-ranked model caught — rank measures how many bugs a model found, not which specific ones it found.

• Running several models in parallel and keeping everyone's catches finds more than any one model alone — but only if whatever merges them keeps the minority contributor's fix instead of quietly dropping it.

• A script that runs without errors isn't proof a model found everything wrong with it; several of the worst bugs here only surfaced once the scripts were actually executed against real data.

• Attribution isn't a nicety in a "combine the best" task — it's the only way anyone downstream can confirm that a weaker source's one good catch survived the merge.

Recommendations

• For an isolated bug-fix task: Qwen3-Max had the best diagnosis-to-fix ratio and defaults to defensive coding (guards over data patches) even when I didn't test for it.

• For a "merge/reconcile several outputs" task: Qwen3-Max again, with Gemini Pro as a strong alternative if you don't need the fixes explained, just working code.

• Avoid ling-3.0-flash, Mistral-Medium-3.5, Nemotron-3-Super-120B, and dots-studio-3-note for unsupervised merge tasks specifically — they all quietly discarded a minority contributor's fix without flagging it, which is the exact failure mode you don't want from something whose job is "look at everything and combine the best of it."

Closing thought

Return to the manuscript. A single careful writer will always produce something coherent — Sonnet 5's script reads cleanly, its comments are tidy, its intentions are legible line by line. But coherence is not the same as correctness, and a lone editor's blind spot stays blind no matter how many times they reread their own draft; that's precisely what happened to the summary table nobody thought to run. A committee of editors catches more, as the four fixers collectively did — between them, every one of the seven bugs was found by someone.

Look closely at who found what, though, and the story gets sharper than "the committee wins." Qwen3-Max fixed four of seven bugs outright, topped both tables, and by any reasonable measure was the strongest model in this test. It still never thought to check that codegen_test.py existed before looping over it. HY3 — ranked behind it in Table 1 — did, and was the only one of the four who did. Finishing first did not make Qwen3-Max the model that caught everything; it just made it the model that caught the most. Every model here corrected something real. No model corrected everything. And the bug the eventual winner missed was sitting, unclaimed, in the very script it outranked.

A committee is only as good as whoever holds the pen at the final pass, deciding which mark to keep and, crucially, willing to say so on the page. Six of the seven combiners kept the right marks and said nothing about where they came from. One kept the right marks and showed its work. That is the entire difference between a model that merges answers and a model that can be trusted to merge them unsupervised — and it's exactly because no single model, however capable, is complete on its own that showing your sources isn't a nicety. It's the only way anyone downstream can tell that a weaker model's one good catch didn't get thrown out along with its worse mistakes.