AI model and agent benchmark analysis

Sourced analysis of AI model and agent benchmarks: what each evaluation measures, what the results show, and where the evidence stops. The first collection focuses on coding agents.

Explore the coding-agent chart

Articles

Seven small ivory forms, from a sphere to a pyramid, sit on separate charcoal plinths along one shelf, each with a blank tag hanging from a thin brass rail.

Introducing aicharts

aicharts plots published AI benchmark scores against cost and tokens per task, and a local collector measures your own agents' token use.

3 min read

A matte ivory column stands behind a sheer dark veil near the left end of a long low charcoal shelf; two taller dark pillars stand together at the right end, with one brass line along the shelf’s front edge.

Gemini 4 Argon at 63.8: every higher row costs at least 2.2x

Antigravity CLI · Gemini 4 Argon (default) scores 63.8 on AA Index at $5.84 per task, third of 31 configurations. Every row above it costs at least 2.2x as much.

11 min read

Five ivory discs rest on a short charcoal rail; the fourth is the largest, and the fifth is smaller than the third.

GPT-6.1 Sol’s best Codex row is 62.9 at $1.04 a task, not max

Codex · GPT-6.1 Sol (xhigh) scores 62.9 on AA Index at $1.04 per task, fourth of 31 configurations. Max scores 60.1 at $1.55.

10 min read

A tall matte ivory block stands near the right end of a long low charcoal table, with one thin brass line running left along the table’s edge.

Claude Code · Opus 5.5 scores 66.0 on the coding-agent chart

Claude Code · Opus 5.5 (max) scores 66.0 on AA Index at $13.04 per task, second of 31 configurations. The next frontier point down gives up 2.2 points for 45% of the cost.

8 min read

Five shallow ivory terraces rise along one low charcoal rail; a matte ivory wedge sits on the highest terrace, with a short brass line on that terrace’s leading edge.

Sonnet 5.5 is first on the coding-agent chart at 68.4, $14.19

Claude Code · Sonnet 5.5 (max) scores 68.4 on AA Index at $14.19 per task, first of 31 configurations and the costliest row. The next frontier point down gives up 2.4 points for 92% of the cost.

8 min read

Five matte ivory spheres of decreasing size rest one per step on a rising charcoal staircase, with a brass line along the top step.

Claude Opus 5.5 leads the Intelligence Index at 57.6 for $5.98

Claude Opus 5.5 at max effort scores 57.6 on the Intelligence Index at $5.98 per task, first of 108 configurations.

8 min read

A matte ivory sphere rests on a charcoal ledge above two separate dark pools, each holding its own reflection of it.

GPT-6 Sol scores 56.7 on the coding-agent chart at $2.99 a task

Codex · GPT-6 Sol (max) scores 56.7 on the coding-agent AA Index at $2.99 per task, seventh of 20 configurations, and GPT-6 Sol (max) scores 47.5 on the Intelligence Index at $1.06 per task. Each row sits on its own chart’s cost frontier.

10 min read

A large matte ivory stone rests across two separate charcoal plinths of different heights, each edged by its own short brass line.

Grok 4.7 reaches 56 on AA Index inside Grok Build

In the snapshots shown below, Grok Build · Grok 4.7 (xhigh) scores 56 on the coding-agent AA Index and Grok 4.7 (xhigh) scores 46 on the Intelligence Index. Different harnesses, task sets, and costs sit behind the two numbers, and cheaper configurations score higher on both charts.

10 min read

A small ivory sphere rests on a low charcoal step beside three tall dark pillars, on a staircase traced by one thin brass line.

MiMo-V2.6-Pro pairs a 46.3 score with $0.133 per task

Artificial Analysis records an Intelligence Index score of 46.3 at $0.133 per task for MiMo-V2.6-Pro in the snapshot retrieved Oct 10, 2026, 5:51 AM UTC.

10 min read

A dark ring with four pale dots sits at the center, with modular pedestals and one blue cube under a glass dome around it.

Context management helps most when the window is tight

Nine researchers held one coding-agent loop fixed and toggled planning, tools, and context management across 176 settings. Each component helped only under named conditions.

11 min read

A pale fiber core passes through a square, a circle, and a triangle frame, then flares into a wide fan.

HarnessTax finds cost gaps at similar success rates

UC Berkeley and Arena researchers ran 21 model and harness pairs on two public suites. Success stayed close across harnesses; cost did not.

8 min read

A locked dark glass cabinet packed with black cubes sits beside a small pale-blue cube in a spotlight.

Real-SWE’s task results reorder its aggregate rankings

Specific Labs licensed private production codebases and scored eight model-and-harness pairs over 640 rollouts. The leading 38.8% is an aggregate that per-task results reorder.

11 min read

A thick charcoal band carries a pale inner path that curves and meets a thinner side path, then continues as one quieter channel.

What Devin Fusion’s 39% saving measures

Cognition’s 39% is one comparison: an Astra-led Fusion pair against Codex on the Artificial Analysis index, at a lower score. The same post’s other reported savings run from 11% to 46%.

7 min read

A dark circular well sits under a brass-marked measuring beam on a tall stand, with four small brass weights on a round tray.

What Terminal-Bench-Science’s 30% result measures

Scientists accepted 70 of 920 proposed workflows. The leading configuration resolved 30 percent; cost and token frontiers show why that rate is incomplete.

4 min read

A small pale cube with a brass pin sits on a short track of circular stations beside a large dark block.

GPT-5.6 Luna made one daily news page cost about $0.10

French-Owen reports about $0.10 per run versus about $1 with earlier Sonnet-class models, from one person’s experiment.

6 min read

A pale path splits: one branch enters a lit arched doorway with an open brass gate, and the other disappears behind a folded black screen.

Why a coding-agent high score still needs a holdout

Dan Luu’s FRE loop won a public regex suite and then failed a holdout. A high coding-agent score still needs cases the optimizer could not see.

7 min read

A dark rounded core sits inside a gold lattice frame with four open loops.

Open models closed SemiAnalysis composites, not this table

SemiAnalysis’s faster catch-up describes era composites. Closed configurations still lead AA Index in this coding-agent snapshot.

7 min read

Mixed blocks, cylinders, and a thin brass frame sit on a dark tray; one cube glows cool blue.

Highest AA Index and lowest cost pick different coding agents

The cost frontier shows which configurations offer a higher AA Index only at a higher mean task cost.

5 min read

A closed black box and an open assembled machine return matching white geometric outputs.

MirrorCode scores complete-program reimplementation

Epoch AI and METR hide the original source and grade a replacement on held-out tests under project-scale budgets.

5 min read

Method

Sources

Each note starts with primary or first-party evidence. Material claims link directly to those sources.

Changing results

Leaderboard values are paired with their observation date and named configuration. They can change after publication.

Limits

Methodology limits and interpretation are kept near the results they qualify. Benchmark performance is not treated as a general production claim.