[mle]/DaBench
Leaderboard

Mellum2.1 Thinking

JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF

LocalQ8_012.9 GB at 32K90 tok/s on M5 Pro128K context
Run recordCompare
Overall score
9

11 of 153 test cases fully passed

Scores by task

Scores give partial credit. Passed counts fully correct answers only. Select a task to read its test cases.

  1. 9Agents1 of 21 passedp90 latency 3m 7s
  2. 5Structured Extraction1 of 21 passedp90 latency 2m 24s
  3. 5Coding1 of 21 passedp90 latency 2m 24s
  4. 13Grounded Answers3 of 21 passedp90 latency 2m 24s
  5. 28Instruction Following4 of 21 passedp90 latency 6m 12s
  6. 0Business Logic0 of 21 passedp90 latency 2m 17s
  7. 8Security & permissions1 of 21 passedp90 latency 2m 18s
  8. 0Diagnosis & investigation0 of 6 passedp90 latency 2m 27s

Run it yourself

The exact build behind this score, and the command that serves it.

Deployment
Local
Engine
llama.cpp
Quantization
Q8_0
Server flags
Not recorded
Context window
128K
Memory
12.9 GB at 32K
Machine
Apple M5 Pro, 48 GB
Throughput
90 tok/s
Cost of a suite run
$2.46est.
Cost per case
$0.016est.
Time per case
1m 56s
Measured
2026-10-09
Run this buildllama.cpp
llama-server -hf JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q8_0 --port 9931 -c 131072

Pulls the GGUF from Hugging Face and serves an OpenAI-compatible endpoint on port 9931.

Memory it needs

12.0 GB of weights, 307 MB of runtime, and 18 MB per 1K tokens of context.

By context length
ContextMemorySmallest MacSmallest GPU
8K12.5 GB24 GB16 GB
32K12.9 GB24 GB16 GB
128K14.6 GB24 GB16 GB

llama.cpp's projection for this file. One slot, a 16-bit KV cache, no vision projector. A Mac fits when the build stays within the share of RAM macOS lets the GPU use, about two thirds up to 32 GB and three quarters from 36 GB. A GPU keeps 1 GB free.

One email when a new model lands on the board or in the showcase. Unsubscribe anytime.

LeaderboardCompare Mellum2.1 Thinking with another model