run
This run
- suite
- starter@v1
- ran at
- 2026-07-30 05:46 UTC
record id
config
Deployment configuration
- harness
- llama.cpp
- quantization
- Q4_K_XL
- context
- 262.1K
- temperature
- 0
- multi-token prediction
- —
- engine
- v1.0.0
- provider
- llama.cpp
category call budgetreasoningtotal
Tool Calling4.1K/ high8.2K/ call
Structured Output2.0K/ medium4.1K/ call
RAG / Retrieval QA2.0K/ medium4.1K/ call
Context Recall2.0K/ medium4.1K/ call
Coding4.1K/ high8.2K/ call
Reasoning & Math4.1K/ high8.2K/ call
Instruction Following2.0K/ medium4.1K/ call
Classification2.0K/ medium4.1K/ call
Summarization2.0K/ medium4.1K/ call
run this buildllama.cpp
llama-server -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL -c 262144Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on :8080.
measured
How it was measured
- latency
- 4508689 ms
- cost
- $1.0308est.
- memory samples
- 4559
tok/s and latency measure the scored generation window; per-test figures above are the readings for each case. Memory (VRAM, unified RAM, or host RAM) is the peak footprint polled throughout the run — measured via process rss across 4559 samples, or blank when no probe could attribute it — never a hand-typed number.
recipe
prompts → hashes
Reproduce this run
The private prompts never leave the suite. What’s public is the command, the pinned suite, and the hashes that stand in for the tests — enough to re-run the exact config against your own copy of the suite.
command
npm run engine -- run --provider llama.cpp --model unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL --suite suites/starter.suite.json --name 'Gemma 4 12B' --category 'tool-calling,reasoning'- suite hash
- config hash
- suite pinned to
- starter@v1