v0.1 · Preliminary
cue bench
How coding agents do on real tasks from cue's own development, graded against each task's acceptance criteria. Higher scores are better.
Results
| # | |||||||
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 · high | 74.7 | 95% | $6.98 | 24 min | 78 | 1 |
| 2 | Claude Opus 5.5 · medium | 72.3 | 95% | $4.87 | 18 min | 63 | 1 |
| 3 | Claude Sonnet 5.5 · high | 71.5 | 94% | $2.65 | 22 min | 60 | 1 |
| 4 | Claude Haiku 5.5 · xhigh | 70.7 | 92% | $2.99 | 27 min | 128 | 2 |
| 5 | Claude Haiku 5.5 · max | 70.2 | 93% | $3.75 | 32 min | 147 | 1 |
| 6 | Claude Haiku 5.5 · high | 69.2 | 92% | $1.70 | 27 min | 83 | 1 |
| 7 | DeepSeek V4.1 Flash (1017) · default† | 67.5 | 92% | $0.12 | 22 min | 78 | 1 |
| 8 | DeepSeek V4.1 Flash (1017) · max | 66.8 | 92% | $0.13 | 30 min | 86 | 1 |
| 9 | GPT-6.1 Sol · high | 65.0 | 91% | $1.66 | 19 min | 64 | 1 |
| 10 | GPT-6 Luna · max | 64.2 | 90% | $0.33 | 31 min | 175 | 1 |
| 11 | GLM 5.3 Flash · default† | 61.8 | 92% | $0.86 | 44 min | 125 | 1 |
| 12 | DeepSeek V4.1 Flash (1017) · high | 57.3 | 89% | $0.11 | 21 min | 74 | 1 |
Results are subject to variance; small differences in scores may not be statistically meaningful.
Hidden tests: the share of hidden test cases that passed. Costs are as reported by each tool; GPT costs are API list-price equivalents.
† No reasoning effort set; provider default via OpenCode Go.
Score per task
Mean score of each configuration on each task, 0 to 100.
| Task | Opus 5.5medium | Opus 5.5high | Sonnet 5.5high | Haiku 5.5high | Haiku 5.5xhigh | Haiku 5.5max | GPT-6.1 Solhigh | GPT-6 Lunamax | DeepSeek V4.1 Flashdefault | DeepSeek V4.1 Flashhigh | DeepSeek V4.1 Flashmax | GLM 5.3 Flashdefault |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SPaused alerts on missing data | 89.0 | 93.0 | 90.0 | 89.0 | 94.0 | 88.0 | 92.0 | 90.0 | 93.0 | 87.0 | 95.0 | 93.0 |
| SFalse security error at startup | 91.0 | 91.0 | 91.0 | 91.0 | 89.5 | 89.0 | 55.0 | 72.0 | 91.0 | 52.0 | 89.0 | 48.0 |
| MMetered write path* requires decisions not stated in the ticket | 58.0 | 52.0 | 48.0 | 47.0 | 43.5 | 45.0 | 55.0 | 45.0 | 42.0 | 35.0 | 45.0 | 55.0 |
| MAlert rule engine* requires decisions not stated in the ticket | 58.0 | 62.0 | 58.0 | 45.0 | 51.5 | 55.0 | 45.0 | 48.0 | 50.0 | 38.0 | 40.0 | 50.0 |
| MSwappable market-data provider | 93.0 | 95.0 | 92.0 | 93.0 | 92.0 | 94.0 | 91.0 | 92.0 | 91.0 | 94.0 | 94.0 | 92.0 |
| LWatchlists and read-only agent tools* requires decisions not stated in the ticket | 45.0 | 55.0 | 50.0 | 50.0 | 53.5 | 50.0 | 52.0 | 38.0 | 38.0 | 38.0 | 38.0 | 33.0 |
* Requires decisions not stated in the ticket. The hidden tests of these tasks encode behaviour decided during the original review, so no model passes all of them.
Methodology
Each task is a real ticket from cue's v0.9 development. The agent gets the ticket and the repository as it was before the change, and works until it stops. Claude models run in Claude Code and GPT models in Codex CLI, both headless; DeepSeek and GLM run in OpenCode.
Score. A grader (Claude Opus 5.5) judges each result blind against the ticket's acceptance criteria, code quality and scope, on a scale of 0 to 100. The headline number is the mean over tasks; each task weighs the same, and repeated runs of a task are averaged first.
Hidden tests. The tests the original change added, run after the agent finishes. The agent never sees them.
Cost, time and turns. Per task, as reported by each tool. Codex CLI reports tokens but no cost, so GPT costs are its token counts priced at OpenAI's standard API list prices as of 2026-10-23. Time leaves out waits for a shared test machine.
- SmallPaused alerts on missing dataKeep an alert paused while candles are missing, instead of letting it resume as active before the gap is read.
- SmallFalse security error at startupStop the command-line tool from reporting a security error while the service is still publishing its runtime file.
- MediumMetered write path*Charge an agent's chart edit under the connection mode and signed-in account that hold when it is committed, and keep a resent request deduplicated after old usage records are pruned.
- MediumAlert rule engine*Build the alert rule language and a deterministic evaluator that runs rules on closed bars, with a breakout-and-retest example, simulation and plain-English rule descriptions.
- MediumSwappable market-data providerMove the exchange's market-data API behind a cue-owned provider interface, so the data source can change without touching the rest of the app.
- LargeWatchlists and read-only agent tools*Add freely named watchlists with groups, plus read-only agent tools for watchlists, candles by time range and a chart's drawings.
Changelog
- cue bench v0.1: 6 tasks from cue's v0.9 development, 9 model configurations.
- Added GPT-6.1 Sol (high) and GPT-6 Luna (max), run in Codex CLI.