[mle]/DaBench
Leaderboard

Underdog Saluki 27B

ConwayResearch/Underdog-Saluki-27B-1.0

LocalIQ2_XS10.2 GB at 32K23 tok/s on M5 Pro256K context
Run recordCompare
Overall score
33

50 of 153 test cases fully passed

Scores by task

Scores give partial credit. Passed counts fully correct answers only. Select a task to read its test cases.

  1. 40Agents6 of 21 passedp90 latency 19m 25s
  2. 43Structured Extraction9 of 21 passedp90 latency 3m 44s
  3. 16Coding1 of 21 passedp90 latency 9m 19s
  4. 43Grounded Answers9 of 21 passedp90 latency 4m 5s
  5. 46Instruction Following9 of 21 passedp90 latency 8m 59s
  6. 33Business Logic7 of 21 passedp90 latency 4m 10s
  7. 43Security & permissions9 of 21 passedp90 latency 5m 57s
  8. 0Diagnosis & investigation0 of 6 passedp90 latency 13m 33s

What it built

What it wrote for showcase prompts, running live in your browser.

Run it yourself

The exact build behind this score, and the command that serves it.

Deployment
Local
Engine
llama.cpp
Quantization
IQ2_XS
Server flags
-ngl -1 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on
Context window
256K
Memory
10.2 GB at 32K
Machine
Apple M5 Pro, 48 GB
Throughput
23 tok/s
Cost of a suite run
Not measured
Cost per case
Not measured
Time per case
5m 15s
Measured
2026-10-15
Run this buildllama.cpp
llama-server -hf ConwayResearch/Underdog-Saluki-27B-1.0:IQ2_XS --port 9931 -c 262144 -ngl -1 --temp 0.6 --top-p 0.95 --top-k 20 --reasoning on

Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on port 9931.

Memory it needs

7.3 GB of weights, 746 MB of runtime, and 69 MB per 1K tokens of context.

By context length
ContextMemorySmallest MacSmallest GPU
8K8.6 GB16 GB12 GB
32K10.2 GB16 GB12 GB
128K16.8 GB32 GB24 GB
256K25.4 GB36 GB32 GB

llama.cpp's projection for this file. One slot, a 16-bit KV cache, no vision projector. A Mac fits when the build stays within the share of RAM macOS lets the GPU use, about two thirds up to 32 GB and three quarters from 36 GB. A GPU keeps 1 GB free.

One email when a new model lands on the board or in the showcase. Unsubscribe anytime.

LeaderboardCompare Underdog Saluki 27B with another model