90.5tok/s
Mean per test case, 90 tok/s over the run
10.3Ktokens
Answer plus reasoning
6.5Ktokens
63% of the output
14.5 GB
llama.cpp's figure for this run's 128K context and 4 slots
2.7M
1.2M in, 1.6M out
4h 55m
1m 56s per caseRequest time, summed
$2.46
$0.016 per caseEstimated from GPU time
Every test case
One square per test case. Select a square to read the answer.
9Agents1 of 21 passed
5Structured Extraction1 of 21 passed
5Coding1 of 21 passed
13Grounded Answers3 of 21 passed
28Instruction Following4 of 21 passed
0Business Logic0 of 21 passed
8Security & permissions1 of 21 passed
0Diagnosis & investigation0 of 6 passed
Speed and output per test case
One bar per test case. Hover to inspect a case, select it to read the answer.
Peak 97 tok/s
ReasoningAnswer
Test by test
Every answer with its reasoning and the checks it had to pass.
Reproduce this run
The prompts stay private so models can't train on them. The hashes pin the exact suite and config this run used.
llama.cpp
llama-server -hf JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q8_0 --port 9931 -c 131072Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on port 9931.
Configuration
Where the suite ran, and the token budget each task got.
- Local
- llama.cpp
- Q8_0
- Not recorded
- 128K
- Apple M5 Pro
- Apple M5 Pro, 18 cores
- 48 GB, shared with the GPU
- macOS 27.0.1, arm64
- 0
- v1.0.0
- 2026-10-09 18:37 UTC
| Task | Reasoning | Max output per call |
|---|---|---|
| Agents | 4.1Khigh | 12.3K |
| Structured Extraction | 4.1Khigh | 12.3K |
| Coding | 4.1Khigh | 12.3K |
| Grounded Answers | 4.1Khigh | 12.3K |
| Instruction Following | 4.1Khigh | 12.3K |
| Business Logic | 4.1Khigh | 12.3K |
| Security & permissions | 4.1Khigh | 12.3K |
| Diagnosis & investigation | 4.1Khigh | 12.3K |