Benchmarks
Run August 2026 · Crinkle 0.1.0-alpha.17 · raw scorecards downloadable below
The claim being tested: the same model produces more software that meets its requirements, and stops falsely claiming success, when it works inside Crinkle. One mid-size open model, three arms, graded by external browser automation that clicks the built apps. No model judges any model.
Method
- Model: qwen3.6-35B-A3B (32k context), running locally. The same model serves every arm and, in the Crinkle arm, every agent role.
- Tasks: a standard suite of 17 tasks (greenfield builds, modifications of working fixture apps, symptom-only bug fixes, a refactor, an ambiguity judgment task) and a hard suite of 5 tasks (interacting criteria, cross-file feature work with regression sweeps, multi-bug hunts, a monolith-to-modules refactor, a deterministic state machine). Every task runs 3 times per arm.
- Arms: single-shot (one prompt containing the goal, fixture file contents, and the same file-output format Crinkle uses; one response, files written, graded) · naive loop (the same, plus up to 5 retries fed the toolchain's and browser's error output; the "weekend script" competitor) · Crinkle (the full autonomous team: planning, specialists, per-criterion browser verification, adversarial input, repair).
- Grading: identical external checks for every arm: Playwright drives the built app in a real browser and asserts on behavior (clicks, inputs, reloads, DOM state). A run passes only if every check passes. A false completion is a run that declared success while failing its checks.
- Harness: ships in the repository (
tests/benchmark.mjs,tests/baseline-benchmark.mjs,--hardfor the hard suite), so the whole experiment is rerunnable.
Hard suite: 5 tasks × 3 runs
| Arm | Runs passed | Checks passed | False completions | Wall time | Tokens |
|---|---|---|---|---|---|
| Single-shot | 1/15 (7%) | 68/123 (55%) | 10/15 (67%) | 19 min | ~0.1M |
| Naive loop | 2/15 (13%) | 79/123 (64%) | 9/15 (60%) | 21 min | ~0.1M |
| Crinkle | 8/15 (53%) | 107/123 (87%) | 0 | 6.9 h | 3.95M |
Where the bare model failed: it dropped features from multi-feature specs, broke working behavior during the refactor, leaked NaN under interacting hostile input, implemented the scheduling state machine loosely, and fixed one of three planted bugs while declaring victory. Crinkle's two 0/3 tasks ended at 23/27 checks each, and it never claimed success on them.
Standard suite: 17 tasks × 3 runs
| Arm | Runs passed | Checks passed | False completions | Wall time | Tokens |
|---|---|---|---|---|---|
| Single-shot | 42/51 (82%) | 152/168 (90%) | 8/51 | 20 min | 0.12M |
| Naive loop | 46/51 (90%) | 161/168 (96%) | 5/51 | 22 min | 0.14M |
| Crinkle | 51/51 (100%) | 168/168 (100%) | 0 | 9.5 h | 5.67M |
Honest caveats
- Time and tokens are the trade. Crinkle spends orders of magnitude more compute than a single prompt; that is the price of verification, and on simple tasks a capable model alone is often sufficient. The product claim is about complex work and about never shipping an unverified "done".
- Runs work until the clock, not until the finish line. In these runs Crinkle typically used its full time budget even after its software passed external checks; it keeps verifying and polishing. Convergence tuning is ongoing; the wall-time numbers above overstate the cost of reaching a correct artifact.
- One model, one machine. These numbers are for the named local model. Different models will shift every column; the harness is in the repo so you can measure yours.
Raw scorecards
Every number above traces to these files, exactly as the harness wrote them (per-run checks, end states, criteria verification detail, per-agent call counts):