Underdog Saluki 27B run record

Underdog Saluki 27B

ConwayResearch/Underdog-Saluki-27B-1.0

starter@v1Localllama.cppIQ2_XS256K context2026-10-14 21:33 UTC

Overall score

50 of 153 test cases fully passed

Throughput

22.6tok/s

Mean per test case, 23 tok/s over the run

Output per test

6.9Ktokens

Answer plus reasoning

Reasoning per test

5.8Ktokens

84% of the output

Memory reserved

25.4 GB

llama.cpp's figure for this run's 256K context and 4 slots

Total tokens

2.3M

1.3M in, 1.1M out

Run time

13h 24m

5m 15s per caseRequest time, summed

Cost

Not measured

No price recorded

Every test case

153 test cases in 8 tasks

One square per test case. Select a square to read the answer.

PassedPartial creditFailed

40Agents6 of 21 passed

6 of 21 passed

43Structured Extraction9 of 21 passed

9 of 21 passed

16Coding1 of 21 passed

1 of 21 passed

43Grounded Answers9 of 21 passed

9 of 21 passed

46Instruction Following9 of 21 passed

9 of 21 passed

33Business Logic7 of 21 passed

7 of 21 passed

43Security & permissions9 of 21 passed

9 of 21 passed

0Diagnosis & investigation0 of 6 passed

0 of 6 passed

Speed and output per test case

One bar per test case. Hover to inspect a case, select it to read the answer.

ThroughputPeak 23 tok/s

Output tokensReasoningAnswer

Test by test

Every answer with its reasoning and the checks it had to pass.

Reproduce this run

The prompts stay private so models can't train on them. The hashes pin the exact suite and config this run used.

Serve the modelllama.cpp

llama-server -hf ConwayResearch/Underdog-Saluki-27B-1.0:IQ2_XS --port 9931 -c 262144 -ngl -1 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on

Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on port 9931.

Record id
Suite hash, starter@v1
Config hash

Configuration

Where the suite ran, and the token budget each task got.

Deployment: Local
Engine: llama.cpp
Quantization: IQ2_XS
Server flags: -ngl -1 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on
Context window: 256K
GPU: Apple M5 Pro
CPU: Apple M5 Pro, 18 cores
Memory: 48 GB, shared with the GPU
System: macOS 27.0.1, arm64
Temperature: Not set
Bench version: v1.0.0
Ran at: 2026-10-14 21:33 UTC

Token budget per task
Task	Reasoning	Max output per call
Agents	4.1Khigh	12.3K
Structured Extraction	4.1Khigh	12.3K
Coding	4.1Khigh	12.3K
Grounded Answers	4.1Khigh	12.3K
Instruction Following	4.1Khigh	12.3K
Business Logic	4.1Khigh	12.3K
Security & permissions	4.1Khigh	12.3K
Diagnosis & investigation	4.1Khigh	12.3K

Underdog Saluki 27B Compare with another model

Every test case

Speed and output per test case

Test by test

40AgentsRuns multi-step jobs through tools, within a written policy6 of 21 passedp90 latency 19m 25s

43Structured ExtractionTurns logs, diffs and records into exact JSON9 of 21 passedp90 latency 3m 44s

16CodingWrites code and SQL that pass hidden tests1 of 21 passedp90 latency 9m 19s

43Grounded AnswersAnswers from the given documents with sources, never guesses9 of 21 passedp90 latency 4m 5s

46Instruction FollowingApplies every rule, including ones changed mid-conversation9 of 21 passedp90 latency 8m 59s

33Business LogicComputes billing, scheduling and pricing rules exactly7 of 21 passedp90 latency 4m 10s

43Security & permissionsIgnores planted instructions, still does the allowed work9 of 21 passedp90 latency 5m 57s

0Diagnosis & investigationChooses diagnostic checks, repairs supported causes and verifies recovery0 of 6 passedp90 latency 13m 33s

Reproduce this run

Configuration