What gets tested
The bench tests the work engineers actually ship. Each task is a set of hardened cases with a known-correct outcome — not trivia, and not vibes. The task catalog the official suite draws from:
- Tool Calling — Right tool, right args, right sequence?
- Structured Output — Valid JSON matching the schema, correct fields?
- RAG / Retrieval QA — Grounded, correct answer from context — no hallucination?
- Context Recall — Finds the needle — the one fact buried in a long context?
- Coding — Working code that passes hidden unit tests?
- Reasoning & Math — Multi-step math and logic with the exact right answer?
- Instruction Following — Obeys format, length, and content constraints to the letter?
- Classification — Correct label from a fixed set?
- Summarization — Compact summary that keeps the facts and invents none?
Categories are declared by the suite itself, versioned as suite@vN — the board renders whatever the suite declares, in its declared order.
How a case is graded
Every case is scored 0–100 by a grader matched to the task: exact-match, JSON-schema validation, tool-trace checks, unit tests, or a rubric. Every launch task grades deterministically — no LLM judge sits between a model and its score, so the full suite reproduces bit-for-bit.
A case passes when it is fully correct; scores also credit partial passes. A task’s score is the mean over its cases, and the overall figure is the mean across tasks. The pass counts behind every score are shown next to it — rigor is a visible number, not a claim.
Speed, memory & cost
Tokens/second and latency are measured on the scored generation window — time to first token is kept apart from steady-state throughput, so warmup can’t inflate the headline. Memory is the peak footprint polled throughout the run (VRAM, unified, or host RAM, whichever the deployment uses), and cost comes from provider-reported usage. A reading the engine couldn’t attribute is shown as “not measured” — never a hand-typed number.
Private suite, public receipts
The prompts never leave the suite — publishing them would let models train on the test. What is public is everything needed to check the work: the exact run command, the pinned suite version, per-test results with the model’s own responses and readings, and the hashes — suite, config, per-case, and pinned weights — that stand in for the tests.
Every score links to its full run record. If a score can’t show you its config, its hashes, and its per-test breakdown, it doesn’t belong on the board.
One board, honest comparisons
Models rank by their best build, with every quantization and harness one expansion away — the build you pull matters, so the board shows what quantization costs. Hosted frontier models run the same suite and sit inline as reference points, marked hosted. Filters narrow by the constraints that decide real deployments: the memory you have, the context you need, the score floor you’ll accept.
What this can’t tell you
A benchmark is a proxy. A model that tops the board can still lose on your workload, your prompts, your latency budget. That’s what the playground is for — point it at your own endpoint and race the shortlist on your own task before you commit.