Mellum2.1 Thinking vs MiMo-V2.6-Pro
Task by task
A score turns orange only when that model wins clearly more of the task's test cases. Under each score, the average time per test case on the machine it ran on. Mellum2.1 Thinking spends 1.2× as many thinking tokens per test case.
TaskAMellum2.1 ThinkingBMiMo-V2.6-Pro
Overall
91m 56s
482m 13s
Agents
92m 3s
474m 48s
Structured Extraction
51m 36s
601m 26s
Coding
52m 11s
242m 58s
Grounded Answers
131m 24s
621m 9s
Instruction Following
283m 10s
591m 23s
Business Logic
01m 39s
571m 52s
Security & permissions
81m 40s
711m 41s
Diagnosis & investigation
01m 15s
02m 54s
What you'd run
The build behind each score. Orange marks the faster, smaller, cheaper or longer-context side.
SpecAMellum2.1 ThinkingBMiMo-V2.6-Pro
Speed
90 tok/s
55 tok/s
Machine
Apple M5 Pro, 48 GB
Hosted
Memory at 32K
12.9 GB
—
Context
128K
1.1M (better)
Cost per run
$2.46 est.
$1.80 (better)
Cost per case
$0.016 est.
$0.012 (better)
Time per case
1m 56s (better)
2m 13s
Quantization
Q8_0
BF16
Harness
llama.cpp
openrouter
The race
Each model's running score as the test cases go by. The dashed line is a perfect run.
Every test case
Each square is one test case, colored by the model that scored higher.
Agents
Structured Extraction
Coding
Grounded Answers
Instruction Following
Business Logic
Security & permissions
Diagnosis & investigation