AI model and agent benchmark analysis

Sourced analysis of AI model and agent benchmarks: what each evaluation measures, what the results show, and where the evidence stops. The first collection focuses on coding agents.

Explore the coding-agent chart

Articles

16 sourced analysis articles

Seven small ivory forms, from a sphere to a pyramid, sit on separate charcoal plinths along one shelf, each with a blank tag hanging from a thin brass rail.

Introducing AI Charts

AI Charts plots published AI benchmark results, and each benchmark snapshot records its source, named version, and retrieval date.

Sep 24, 20265 min read

A tall matte ivory block stands near the right end of a long low charcoal table, with one thin brass line running left along the table’s edge.

Opus 5.5 tops the coding-agent chart at 66.0 and $13.04 a task

Claude Code · Opus 5.5 (max) scores 66.0 on AA Index at $13.04 per task, first of 20 configurations and the costliest row. The next frontier point down gives up 3.8 points for 95% of the cost.

Sep 28, 20269 min read

Five matte ivory spheres of decreasing size rest one per step on a rising charcoal staircase, with a brass line along the top step.

Claude Opus 5.5 leads the Intelligence Index at 57.6 for $5.98

Claude Opus 5.5 at max effort scores 57.6 on the Intelligence Index at $5.98 per task, first of 97 configurations. The four highest points on the cost frontier are all its effort levels.

Sep 25, 20269 min read

A matte ivory sphere rests on a charcoal ledge above two separate dark pools, each holding its own reflection of it.

GPT-6 Sol scores 56.7 on the coding-agent chart at $2.99 a task

Codex · GPT-6 Sol (max) scores 56.7 on the coding-agent AA Index at $2.99 per task, seventh of 20 configurations, and GPT-6 Sol (max) scores 47.5 on the Intelligence Index at $1.06 per task. Each row sits on its own chart’s cost frontier.

Sep 24, 20269 min read

A large matte ivory stone rests across two separate charcoal plinths of different heights, each edged by its own short brass line.

What Grok 4.7’s 56 on the coding-agent chart measures

On the September 25, 2026 snapshots, Grok Build · Grok 4.7 (xhigh) scores 56 on the coding-agent AA Index and Grok 4.7 (xhigh) scores 46 on the Intelligence Index. Different harnesses, task sets, and costs sit behind the two numbers, and cheaper configurations score higher on both charts.

Sep 23, 20269 min read

A small ivory sphere rests on a low charcoal step beside three tall dark pillars, on a staircase traced by one thin brass line.

What MiMo-V2.6-Pro’s 46 at $0.13 per task measures

On the September 22, 2026 Intelligence Index snapshot, Xiaomi’s open-weights flagship scores within a point of GPT-5.6 Sol and Grok 4.7 at less than a tenth of their cost per task. Its cybersecurity lead is on one kind of task, measured by Xiaomi.

Sep 22, 202611 min read

A dark ring with four pale dots sits at the center, with modular pedestals and one blue cube under a glass dome around it.

What Fan et al.’s harness-component ablations measure

Nine researchers held one coding-agent loop fixed and toggled planning, tools, and context management across 176 settings. Each component helped only under named conditions.

Sep 21, 202611 min read

A pale fiber core passes through a square, a circle, and a triangle frame, then flares into a wide fan.

What HarnessTax’s same-model cost gap measures

UC Berkeley and Arena researchers ran 21 model and harness pairs on two public suites. Success stayed close across harnesses; cost did not.

Sep 17, 20268 min read

A locked dark glass cabinet packed with black cubes sits beside a small pale-blue cube in a spotlight.

What Real-SWE’s 38.8% on private enterprise code measures

Specific Labs licensed private production codebases and scored eight model-and-harness pairs over 640 rollouts. The leading 38.8% is an aggregate that per-task results reorder.

Sep 14, 202612 min read

A thick charcoal band carries a pale inner path that curves and meets a thinner side path, then continues as one quieter channel.

What Devin Fusion’s 39% saving measures

Cognition’s 39% is one comparison: an Astra-led Fusion pair against Codex on the Artificial Analysis index, at a lower score. The same post’s other reported savings run from 11% to 46%.

Sep 11, 20267 min read

A dark circular well sits under a brass-marked measuring beam on a tall stand, with four small brass weights on a round tray.

What Terminal-Bench-Science’s 30% result measures

Scientists accepted 70 of 920 proposed workflows. The leading configuration resolved 30 percent; cost and token frontiers show why that rate is incomplete.

Aug 31, 20264 min read

A small pale cube with a brass pin sits on a short track of circular stations beside a large dark block.

GPT-5.6 Luna made one daily news page cost about $0.10

French-Owen reports about $0.10 per run versus about $1 with earlier Sonnet-class models, from one person’s experiment.

Aug 28, 20266 min read

A pale path splits: one branch enters a lit arched doorway with an open brass gate, and the other disappears behind a folded black screen.

Why a coding-agent high score still needs a holdout

Dan Luu’s FRE loop won a public regex suite and then failed a holdout. A high coding-agent score still needs cases the optimizer could not see.

Aug 26, 20267 min read

A dark rounded core sits inside a gold lattice frame with four open loops.

Open models closed SemiAnalysis composites, not this table

SemiAnalysis’s faster catch-up describes era composites. Closed configurations still lead AA Index in this coding-agent snapshot.

Aug 26, 20267 min read

Mixed blocks, cylinders, and a thin brass frame sit on a dark tray; one cube glows cool blue.

Highest AA Index and lowest cost pick different coding agents

The checked snapshot keeps a configuration on the frontier only when nothing cheaper scores at least as well on AA Index.

Aug 22, 20265 min read

A closed black box and an open assembled machine return matching white geometric outputs.

MirrorCode scores complete-program reimplementation

Epoch AI and METR hide the original source and grade a replacement on held-out tests under project-scale budgets.

Aug 4, 20265 min read

Method

Sources

Each note starts with primary or first-party evidence. Material claims link directly to those sources.

Changing results

Leaderboard values are paired with their observation date and named configuration. They can change after publication.

Limits

Methodology limits and interpretation are kept near the results they qualify. Benchmark performance is not treated as a general production claim.