82.2tok/s
Mean per test case, 82 tok/s over the run
9.2Ktokens
Answer plus reasoning
8.6Ktokens
93% of the output
26.9 GB
llama.cpp's figure for this run's 128K context and 4 slots
2.5M
1.1M in, 1.4M out
4h 49m
1m 53s per caseRequest time, summed
Not measured
No price recorded
Every test case
One square per test case. Select a square to read the answer.
12Agents1 of 21 passed
10Structured Extraction2 of 21 passed
1Coding0 of 21 passed
5Grounded Answers1 of 21 passed
16Instruction Following0 of 21 passed
5Business Logic1 of 21 passed
10Security & permissions1 of 21 passed
0Diagnosis & investigation0 of 6 passed
Speed and output per test case
One bar per test case. Hover to inspect a case, select it to read the answer.
Peak 84 tok/s
ReasoningAnswer
Test by test
Every answer with its reasoning and the checks it had to pass.
Reproduce this run
The prompts stay private so models can't train on them. The hashes pin the exact suite and config this run used.
llama.cpp
llama-server -hf LiquidAI/LFM2.5-24B-A2B-GGUF:Q8_0 --port 9931 -c 131072 -ngl -1 --temp 0.6 --top-k 50 --reasoning on --repeat-penalty 1.05Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on port 9931.
Configuration
Where the suite ran, and the token budget each task got.
- Local
- llama.cpp
- Q8_0
- -ngl -1 --temp 0.6 --top-k 50 --reasoning on --repeat-penalty 1.05
- 128K
- Apple M5 Pro
- Apple M5 Pro, 18 cores
- 48 GB, shared with the GPU
- macOS 27.0.1, arm64
- Not set
- v1.0.0
- 2026-10-14 05:55 UTC
| Task | Reasoning | Max output per call |
|---|---|---|
| Agents | 4.1Khigh | 12.3K |
| Structured Extraction | 4.1Khigh | 12.3K |
| Coding | 4.1Khigh | 12.3K |
| Grounded Answers | 4.1Khigh | 12.3K |
| Instruction Following | 4.1Khigh | 12.3K |
| Business Logic | 4.1Khigh | 12.3K |
| Security & permissions | 4.1Khigh | 12.3K |
| Diagnosis & investigation | 4.1Khigh | 12.3K |