What Terminal-Bench-Science’s 30% result measures

Terminal-Bench-Science 0.1 is a Stanford-led evaluation of AI agents on scientific research workflows. Steven Dillmann, writing for the benchmark team, reports that 920 proposed workflows became 70 accepted tasks after domain, implementation, and difficulty review. The team’s stated principle is: “Scientists, not model developers or data vendors, set the bar for scientific capability in AI.”

What the 30% result covers

“Of 920 proposals, 464 were approved for implementation and 386 pull requests were opened, but only 70 tasks made it into Terminal-Bench-Science 0.1.” Each configuration ran three independent trials per task across all 70 tasks. The accepted set spans scientific data analysis, statistical inference, simulation, optimization, theorem proving, image reconstruction, signal processing, inverse problems, sensor calibration, model fitting, classification, and scientific machine learning. The leaderboard records the model together with its agent harness, because the harness is part of the evaluated configuration.

“The strongest model evaluated, Claude Opus 5, achieves a 30% resolution rate on Terminal-Bench-Science 0.1.” The suite is calibrated to sit more than 10 percentage points below Terminal-Bench 3.0 for every model evaluated on both. Because reviewers excluded workflows that frontier systems already solved at high rates, 30% describes performance on a deliberately difficult accepted set. It should not be read as a success rate for arbitrary laboratory work or as evidence that one model can replace a scientist.

Cost and tokens change the comparison

“GPT-5.6 Sol matches Claude Fable 5's performance at less than a third of the cost ($4.2k vs $14.2k).” Claude Opus 5 reaches the highest resolution at $7.0k. On tokens, Claude Fable 5 matches GPT-5.6 Sol’s performance while using about a quarter fewer tokens (6.4B versus 8.4B). Only Kimi K3 and Claude Opus 5 appear on both the cost-resolution and token-resolution Pareto frontiers.

The highest resolution, lowest evaluation cost, and lowest token use do not select the same configuration. A team choosing an evaluation candidate therefore needs a quality threshold and a budget, rather than a single overall winner.

How this score differs from the coding-agent chart

The aicharts homepage now includes Terminal-Bench-Science 0.1 as the scientific-workflow member of its five-role benchmark portfolio. Its scores remain separate from the interactive Artificial Analysis coding-agent chart, which plots Coding Agent Index v1.5, DeepSWE v1.1, Terminal-Bench 4, and SWE-Atlas-QnA against API cost, active time, or total token use. Those coding observations come from a checked Artificial Analysis coding-agents snapshot retrieved Oct 5, 2026, 2:10 PM UTC.

For orientation, the snapshot’s highest stored Terminal-Bench 4 score is 66.2 for Sonnet 5.5 on Claude Code at the max setting. That value belongs to a software-engineering terminal benchmark; it cannot be compared numerically with the 30% Science resolution rate. The shared lesson is methodological: keep the task set, model, harness, quality measure, cost, and token use attached to every comparison.

Limits

• Resolution rates, costs, and token totals belong to the named 0.1 release, models, harnesses, and three-trial protocol on the announcement. They can change in a later release.
• Terminal-Bench-Science 0.1 is a homepage benchmark family, not a field in the checked Artificial Analysis coding-agent snapshot. A stored Coding Agent Index Terminal-Bench 4 component is a separate evaluator cohort.
• Reported evaluation costs are totals across all 70 tasks. They are not a production invoice, a subscription price, or a per-query quote.
• The suite is living and versioned; later releases can add, retire, or recalibrate tasks.
• Resolution varies by scientific domain, so the aggregate is not a domain-specific capability claim.