117.6tok/s
Mean per test case, 117 tok/s over the run
10.5Ktokens
Answer plus reasoning
9.8Ktokens
93% of the output
16.7 GB
llama.cpp's figure for this run's 128K context and 4 slots
2.9M
1.3M in, 1.6M out
3h 52m
1m 31s per caseRequest time, summed
Not measured
No price recorded
Every test case
One square per test case. Select a square to read the answer.
8Agents0 of 21 passed
0Structured Extraction0 of 21 passed
1Coding0 of 21 passed
10Grounded Answers2 of 21 passed
15Instruction Following1 of 21 passed
0Business Logic0 of 21 passed
11Security & permissions2 of 21 passed
0Diagnosis & investigation0 of 6 passed
Speed and output per test case
One bar per test case. Hover to inspect a case, select it to read the answer.
Peak 121 tok/s
ReasoningAnswer
Test by test
Every answer with its reasoning and the checks it had to pass.
Reproduce this run
The prompts stay private so models can't train on them. The hashes pin the exact suite and config this run used.
llama.cpp
llama-server -hf LiquidAI/LFM2.5-24B-A2B-GGUF:Q4_K_M --port 9931 -c 131072 -ngl -1 --temp 0.1 --top-k 50 --reasoning on --repeat-penalty 1.05Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on port 9931.
Configuration
Where the suite ran, and the token budget each task got.
- Local
- llama.cpp
- Q4_K_M
- -ngl -1 --temp 0.1 --top-k 50 --reasoning on --repeat-penalty 1.05
- 128K
- Apple M5 Pro
- Apple M5 Pro, 18 cores
- 48 GB, shared with the GPU
- macOS 27.0.1, arm64
- Not set
- v1.0.0
- 2026-10-13 21:27 UTC
| Task | Reasoning | Max output per call |
|---|---|---|
| Agents | 4.1Khigh | 12.3K |
| Structured Extraction | 4.1Khigh | 12.3K |
| Coding | 4.1Khigh | 12.3K |
| Grounded Answers | 4.1Khigh | 12.3K |
| Instruction Following | 4.1Khigh | 12.3K |
| Business Logic | 4.1Khigh | 12.3K |
| Security & permissions | 4.1Khigh | 12.3K |
| Diagnosis & investigation | 4.1Khigh | 12.3K |