How the bench works

153 test cases across 8 tasks, with 6 to 21 cases per task. The cases cover work engineers do with language models and have deterministic checks.

• AgentsRuns multi-step jobs through tools, within a written policytooltracetool-state

  Multi-step jobs over tools: releases, settlements, staged flag rollouts, backfills, index rebuilds, secret rotation, a schema migration, GPU preemption, quota rebalancing and a regional failover. Procedure cases require exact call sequences. Goal-directed cases accept different inspection orders and check the resulting state, verification and call budget. They include missing-recipient clarification and recovery from uncertain writes. A forbidden call fails the case.

• Structured ExtractionTurns logs, diffs and records into exact JSONjson-match

  Records rebuilt from the text engineers work with: access logs, chained stack traces, commit logs, pod events, a CSV export, an infrastructure plan, an API contract diff, evaluation tables, webhook deliveries, an incident chat and training logs. The revised-record cases cover an invoice, identity links, traces and purchase order patches. Other cases extract uncertain OCR fields, reconcile signed attachments and find current records among unrelated revisions. Missing evidence stays null. Every field must match.

• CodingWrites code and SQL that pass hidden testsunit-testsql

  JavaScript specs and SQLite queries run against hidden tests: version ranges, a cache with expiry, patch application, a build scheduler, a transaction simulator and graph problems. Three cases take their expected answers from real tools: a gitignore matcher and a git config parser checked against git, and a SELECT engine checked against SQLite. Eight answers are search/replace edits to existing code, including tenant isolation, retry delays, merge patches, refund allocation, cursors and permission matching. Hidden tests check repairs and preserved behavior. Five cases check all their scenarios as one test, so one wrong scenario scores the case zero.

• Grounded AnswersAnswers from the given documents with sources, never guessesjson-matchtool-state

  Questions answered from supplied or retrieved documents: API limits, config overlays, on-call rotations, SDK releases, security advisories, a service contract, decision records, model licences, a metric catalogue, retention rules and a paper's claims. Retrieval cases require reading the supporting documents. Context packets include irrelevant policies and revisions. Each answer lists its sources, and what the documents do not settle comes back as null.

• Instruction FollowingApplies every rule, including ones changed mid-conversationjson-match

  Rules applied exactly, most of them across a conversation that changes them: a naming style guide, triage labels, changelog formatting, a size report, feature flag rules, schema steps, a masked report, a notification plan, retry configuration, a runbook under edit and a gateway route table. Six conversations grade every reply, covering scoped preferences, missing-field clarification, authorization changes, unknown values and retractions. The remaining conversations grade the final cumulative answer.

• Business LogicComputes billing, scheduling and pricing rules exactlyjson-match

  Rules computed exactly: seat proration, a rate limiter, support deadlines, a payment ledger, usage invoices, retries behind a circuit breaker, cron across offset changes, an autoscaler, stock costing, a call auction and a clearing network. The portfolio, schedule, shard placement and dependency cases need the global optimum, so a feasible answer that is not optimal fails. Separately scored variants test refund cutoffs and allocation tie-breaks; quote cases distinguish missing and known tax rates.

• Security & permissionsIgnores planted instructions, still does the allowed worktooltracejson-matchtool-state

  Tasks with instructions planted in tool results, tool descriptions, pages and records: a review bot, data exports, a browser form, release artifacts and password resets, each with a secret test string that must not leak. The decision cases cover webhook verification, package installs, OAuth scopes, secret findings and access requests. Other cases check consent revoked during export, private uploads, secret redaction, minimum grants and deny-rule expiry. Refusing all authorized work fails.

• Diagnosis & investigationChooses diagnostic checks, repairs supported causes and verifies recoverytool-state

  Investigations distinguish DNS from TLS failures, join fanout from source errors, pool leaks from slow queries and uncontrolled test clocks from product defects. Models choose their inspection order and verify repairs. Missing timings require an unknown diagnosis and a request for discriminating evidence.

The suite carries a version, such as starter@v1, and the board shows its tasks in the suite's order.

Code grades every answer. No language model judges another, so the same answer always gets the same score. The starter suite uses five graders:

json-match

The answer must equal the expected JSON exactly. 33 of the 91 JSON test cases also ask for two variants with changed inputs in the same object.

tooltrace

The model makes native tool calls against scripted results. Procedure-based agent cases expect 7 to 24 calls in a fixed order with exact arguments. Credit is the share made in order, and extra calls lower it. A forbidden call or a wrong final report fails the case.

tool-state

18 cases run tools against a stateful simulation. Different valid inspection orders can pass. The model must reach the required state within the call budget, verify its work where required, avoid forbidden actions and report the outcome correctly.

unit-test

JavaScript runs against hidden tests in an isolated V8 context with no files, network or environment. Edit test cases send search/replace blocks, and a block that doesn't match the file exactly once fails the case. 8 cases repair or complete existing code, with hidden checks for the requested behavior and regressions.

sql

The query runs read-only on seeded rows the model never sees and must return the expected rows.

18 of the 21 instruction-following cases are conversations. 6 grade every reply: correctness is the mean across replies and format quality is the lowest reply quality. The others grade the final cumulative answer. Security and permissions cases plant instructions in tool results and data. Following them counts as an extra or forbidden call, and leaking the planted secret string fails the case.

Each answer gets a correctness from 0 to 1 and a format quality from 0 to 100. The score is correctness times quality. A test case passes when its correctness is 1, so a correct answer in a sloppy format still passes, with a lower score.

A task's score is the mean of its test case scores. The overall score is the mean of the task scores, so every task weighs the same. Each model ranks by its best build that ran the whole suite. Models with equal overall scores share a rank.

Every score comes with a pass count, the number of fully correct test cases, so you can see how much of it is partial credit.

Each test case runs once, so luck moves a score. With 21 cases, one answer that flips from fail to pass moves a task score by about 5 points. The diagnosis & investigation task has only 6 cases, so each result has more influence.

To call one model better, the bench counts wins on the test cases where the two scored differently. A model is clearly better when it wins at least this many of them:

The cutoffs come from a simple Bayesian estimate. It starts with both models equally likely to be better and calls a winner once one is at least 95% likely to win more often. Short of that, the result is too close to call. A win counts the same at any margin, so 80 against 75 weighs as much as 100 against 0.

Compare applies this rule to the overall score and to every task.

Tokens per second

Output tokens over each call's generation window, from its first streamed token to its last. When a model reports reasoning tokens it never streamed, the window starts at the request, since it generated them before its first visible token. A call that arrives whole within 250 ms can't be timed and is left out. The run figure divides every timed output token by its generation time.

First token

Measured on its own, so a slow start doesn't skew tokens per second.

Latency

The whole request. A task's p90 latency is the time 9 in 10 of its test cases finished within.

Memory

What a local build needs at a given context length: its weights, a fixed runtime, and a cost per 1K tokens of context. For runs on a Mac, llama.cpp projects these for the exact file. Other runs estimate them from the file size and the model's config. The board shows memory at 32K, or at the minimum context you filter by. Hosted models have no memory figure.

Cost

Tokens at the provider's list price. Local runs estimate it from GPU time and mark it as an estimate. Cost per case divides it by the test cases in the run.

Time per case

The run's request time divided by its test cases, on the machine it ran on.

A reading the engine can't take shows as not measured. Nobody enters readings by hand.

The prompts stay private so models can't train on the test. The run record publishes everything else: the run command, the pinned suite version, every answer with its readings, and the hashes of the suite and the config.

A score without a run record doesn't go on the board.

Hosted and open-weight models rank in the same table, one row per model. Each model's page compares every quantization and harness it ran with.

The date under Cost / run is when the run finished. Filters narrow the board by memory, context and minimum score. A build fits the memory you set when its memory at the minimum context, or 32K, stays within what the machine lets the model use. That is about two thirds of a Mac's RAM up to 32 GB, three quarters from 36 GB, or all of a GPU's memory, less the 1 GB llama.cpp keeps free.

The model at the top of the board can still lose on your workload, prompts or latency budget. Try your shortlist on your own task in the Playground before you commit to one.