Benchmarks

Updated August 8, 2026 · Crinkle 0.1.0-alpha.17 · raw scorecards downloadable below

Method

• Model: qwen3.6-35B-A3B (32k context), running locally. The same model serves every arm and, in the Crinkle arm, every agent role.
• Tasks: a standard suite of 17 tasks (greenfield builds, modifications of working fixture apps, symptom-only bug fixes, a refactor, an ambiguity judgment task) and a hard suite of 5 tasks (interacting criteria, cross-file feature work with regression sweeps, multi-bug hunts, a monolith-to-modules refactor, a deterministic state machine). Every task runs 3 times per arm.
• Arms: single-shot (one prompt containing the goal, fixture file contents, and the same file-output format Crinkle uses; one response, files written, graded) · naive loop (the same, plus up to 5 retries fed the toolchain's and browser's error output; the "weekend script" competitor) · Crinkle (the full autonomous team: planning, specialists, per-criterion browser verification, adversarial input, repair).
• Grading: identical external checks for every arm: Playwright drives the built app in a real browser and asserts on behavior (clicks, inputs, reloads, DOM state). A run passes only if every check passes. A false completion is a run that declared success while failing its checks.
• Harness: ships in the repository (tests/benchmark.mjs, tests/baseline-benchmark.mjs, --hard for the hard suite), so the whole experiment is rerunnable.

Hard suite: 5 tasks × 3 runs

Where the bare model failed: it dropped features from multi-feature specs, broke working behavior during the refactor, leaked NaN under interacting hostile input, implemented the scheduling state machine loosely, and fixed one of three planted bugs while declaring victory. Crinkle's two 0/3 tasks ended at 23/27 checks each, and it never claimed success on them.

Standard suite: 17 tasks × 3 runs

Latest scale test: Nemotron 550B

The obvious objection to the tables above: maybe a larger model simply doesn't need the harness. So we ran the same 5 hard tasks, same fixtures, same external Playwright checks, against nvidia/nemotron-3-ultra-550b-a55b (a frontier-scale open model roughly 15× the size of the main suite's model, hosted on NVIDIA NIM), working on its own in both baseline arms.

The August 8 rerun reinforces the central result while making the cost visible:

• More capability does not make "done" trustworthy. The best bare-model arm passed 5 of 15 runs, but still made 8 false completion claims. Crinkle made none: when proof was incomplete, it did not say the project was complete.
• The small model inside Crinkle still beat the big model without it. 8/15 runs and 87% of checks for Crinkle with a 35B local model, versus at best 5/15 and 72% for the bare 550B. On these tasks, verification and repair were worth more than a 15× larger model.
• The remaining failures are concrete, not near-random misses. The latest failures concentrated in hostile numeric-input handling on the invoice task and scheduling, persistence and review-state behavior on flashcards. CRM modification, CRM bug repair and the monolith-to-modules refactor passed all three repetitions.
• Crinkle lifts the big model too. The same 550B model rose from a best baseline of 5 to 9 passing runs, from 88 to 113 passing checks, and from 8 false completions to zero. The harness is not only a small-model amplifier.
• The false "done" also shows on the easy suite. On the 17-task standard suite, bare Nemotron still falsely declared completion 4 times in 51 single-shot runs and 2 times in 51 retry-loop runs. Model scale changes how often the code is right; it does not change whether the completion claim can be trusted.

Fairness notes: the latest Crinkle arm used 587 provider calls, 2.36M tokens, 7.03 hours and about $1.81 of measured NIM credit. The best baseline used 16 calls, 0.11M tokens, 0.73 hours and about $0.12. That is roughly 37× the calls, 21× the tokens, 10× the time and 15× the measured cost. None of the 15 Crinkle runs falsely completed, but all 15 reached their time budget; 9 artifacts passed externally at timeout. Crinkle currently buys substantially better software and safer completion behavior with substantially more compute. Reducing that premium and teaching successful runs to conclude promptly are active work.

The immediately preceding Nemotron run passed 10/15 artifacts and 115/123 checks using 705 calls and 2.61M tokens. The latest run used 17% fewer calls and 10% fewer tokens, while passing one fewer artifact and two fewer checks. With only three repetitions per task, that small quality movement should be treated as run-to-run variation, not a proven regression; the efficiency reduction is directly measured.

Claude Sonnet 5 pilot: a frontier model changes what Crinkle is for

We also ran the hard suite with Claude Sonnet 5. The baseline arms below are the FULL three-repetition comparison (15 runs each); the Crinkle arm is a five-run pilot (one run per task) on the current build, so its rate is directional until repeated. An earlier five-run slice on the previous build passed 4 of 5 with 37/41 checks; both slices are downloadable below.

Read this table honestly and it says two different things. On raw capability, bare Sonnet 5 with a simple retry loop is genuinely strong: 13 of 15 hard runs in 17 minutes. A frontier model does not need Crinkle to write most of this code. What it does not deliver is a completion claim you can trust: even the strongest bare arm falsely declared "done" on 2 of 15 runs, and single-shot on 5 of 15. The Crinkle pilot passed every check with zero false completions and recovered the last-mile artifacts the bare arms missed. On frontier models, Crinkle's value is verification, evidence, and honest completion, at a real compute premium (about 9× the tokens and 7× the wall time of the loop arm).

Two more disclosures: only one of the five Crinkle pilot runs cleanly concluded (four passed every external check at timeout), and the reported provider cost is $0 because this run used a Claude Code subscription connection; that accounting does not mean the compute was free.

Honest caveats

• Time and tokens are the trade. Crinkle spends orders of magnitude more compute than a single prompt; that is the price of verification, and on simple tasks a capable model alone is often sufficient. The product claim is about complex work and about never shipping an unverified "done".
• Runs work until the clock, not until the finish line. In these runs Crinkle typically used its full time budget even after its software passed external checks; it keeps verifying and polishing. Convergence tuning is ongoing; the wall-time numbers above overstate the cost of reaching a correct artifact.
• Limited models and repetitions. Different models will shift every column, and the Claude result currently has only one run per task. The harness is in the repo so the claims can be rerun and challenged.

Raw scorecards

Every number above traces to these files, exactly as the harness wrote them (per-run checks, end states, criteria verification detail, per-agent call counts):

• crinkle-standard.json · crinkle-hard.json
• baseline-single-standard.json · baseline-loop-standard.json
• baseline-single-hard.json · baseline-loop-hard.json
• baseline-single-hard-nemotron.json · baseline-loop-hard-nemotron.json
• crinkle-hard-nemotron.json · baseline-single-standard-nemotron.json · baseline-loop-standard-nemotron.json
• latest Crinkle + Nemotron hard suite · latest Nemotron single-shot · latest Nemotron loop
• preceding Crinkle + Nemotron hard suite (historical comparison)
• Nemotron baselines, first session (single) · (loop)
• Claude baselines, 15 runs (single) · (loop) · 1-run recheck (single) · (loop)
• Crinkle + Claude pilot (current build) · earlier slice (previous build)