226.5tok/s
Mean per test case, 225 tok/s over the run
3.9Ktokens
Answer plus reasoning
3Ktokens
78% of the output
1.8M
1.2M in, 596.7K out
1h 5m
25.5s per caseRequest time, summed
$0.42
$0.0028 per caseAt the provider's list price
Every test case
One square per test case. Select a square to read the answer.
61Agents12 of 21 passed
90Structured Extraction19 of 21 passed
86Coding18 of 21 passed
76Grounded Answers16 of 21 passed
94Instruction Following19 of 21 passed
67Business Logic14 of 21 passed
67Security & permissions14 of 21 passed
17Diagnosis & investigation1 of 6 passed
Speed and output per test case
One bar per test case. Hover to inspect a case, select it to read the answer.
Peak 580 tok/s
ReasoningAnswer
Test by test
Every answer with its reasoning and the checks it had to pass.
Reproduce this run
The prompts stay private so models can't train on them. The hashes pin the exact suite and config this run used.
OpenRouterHosted
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"anthropic/claude-haiku-5.5","provider":{"order":["amazon-bedrock"],"allow_fallbacks":false},"service_tier":"default","messages":[{"role":"user","content":"Hello"}]}'The request the benchmark sent, pinned to the provider that served it.
Configuration
Where the suite ran, and the token budget each task got.
- Hosted
- OpenRouter
- Amazon Bedrock
- Provider default
- 1M
- 0
- v1.0.0
- 2026-10-07 20:54 UTC
{
"provider": {
"order": [
"amazon-bedrock"
],
"allow_fallbacks": false
},
"service_tier": "default"
}| Task | Reasoning | Max output per call |
|---|---|---|
| Agents | 4.1Khigh | 12.3K |
| Structured Extraction | 4.1Khigh | 12.3K |
| Coding | 4.1Khigh | 12.3K |
| Grounded Answers | 4.1Khigh | 12.3K |
| Instruction Following | 4.1Khigh | 12.3K |
| Business Logic | 4.1Khigh | 12.3K |
| Security & permissions | 4.1Khigh | 12.3K |
| Diagnosis & investigation | 4.1Khigh | 12.3K |