6
5 of 153 test cases fully passed
111.6tok/s
Mean per test case, 104 tok/s over the run
13.5Ktokens
Answer plus reasoning
9Ktokens
66% of the output
5.8 GB
llama.cpp's figure for this run's 256K context and 4 slots
3.2M
1.1M in, 2.1M out
5h 33m
2m 11s per caseSum of request times
Not measured
No price recorded
Every test case
One square per test case. Select a square to read the answer.
13Agents2 of 21 passed
0Structured Extraction0 of 21 passed
5Coding1 of 21 passed
0Grounded Answers0 of 21 passed
14Instruction Following0 of 21 passed
0Business Logic0 of 21 passed
16Security & permissions2 of 21 passed
0Diagnosis & investigation0 of 6 passed
Speed and output per test case
One bar per test case. Hover a bar to see its numbers. Select it to read the answer.
Peak 124 tok/s
ReasoningAnswer
Test by test
Every answer with its reasoning and the checks it had to pass.
Reproduce this run
The prompts stay private so models can't train on them. The hashes pin the exact suite and config this run used.
llama.cpp
llama-server -hf LiquidAI/LFM3-1.7B-GGUF:Q8_0 --port 9931 -c 262144 -ngl all --temp 0.1 --top-k 50 --reasoning on --repeat-penalty 1.1Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on port 9931.
Configuration
Where the suite ran, and the token budget each task got.
- Local
- llama.cpp
- Q8_0
- -ngl all --temp 0.1 --top-k 50 --reasoning on --repeat-penalty 1.1
- 256K
- Apple M5 Pro
- Apple M5 Pro, 18 cores
- 48 GB, shared with the GPU
- macOS 27.2, arm64
- Not set
- v1.0.0
- 2026-10-28 18:27 UTC
| Task | Reasoning | Max output per call |
|---|---|---|
| Agents | 4.1K | 12.3K |
| Structured Extraction | 4.1K | 12.3K |
| Coding | 4.1K | 12.3K |
| Grounded Answers | 4.1K | 12.3K |
| Instruction Following | 4.1K | 12.3K |
| Business Logic | 4.1K | 12.3K |
| Security & permissions | 4.1K | 12.3K |
| Diagnosis & investigation | 4.1K | 12.3K |