[mle]/DaBench

LFM2.5 24B A2B vs Qwen3.8 Flash

Side A
Model page
Side B
Model page
Overall
7
vs
Overall
32

Qwen3.8 Flash is clearly better. It wins 56 of the 62 test cases where they differ.

Learn to build evals like this in the AI Engineering Academy

Task by task

A score turns orange only when that model wins clearly more of the task's test cases. Under each score, the average time per test case on the machine it ran on. LFM2.5 24B A2B spends 2.7× as many thinking tokens per test case.

Thinking tokensAnswer tokens
TaskAB
Overall
71m 53s
9.2K, 93% thinking4K, 81% thinking
3258.5s
Agents
123m 49s
18.5K, 98% thinking4.2K, 76% thinking
491m 25s
Structured Extraction
1054.9s
4.4K, 94% thinking3K, 90% thinking
3336.9s
Coding
11m 15s
6.1K, 68% thinking5.8K, 61% thinking
471m 21s
Grounded Answers
557.8s
4.7K, 96% thinking3.1K, 90% thinking
2641.4s
Instruction Following
162m 44s
13.4K, 97% thinking5K, 89% thinking
371m 4s
Business Logic
51m 2s
5K, 83% thinking3.9K, 91% thinking
1746.8s
Security & permissions
101m 52s
9K, 97% thinking3.2K, 83% thinking
5054.1s
Diagnosis & investigation
04m 12s
19.9K, 98% thinking3.3K, 73% thinking
058.4s

What you'd run

The build behind each score. Orange marks the faster, smaller, cheaper or longer-context side.

SpecAB
Speed
82 tok/s
77 tok/s
Machine
Apple M5 Pro, 48 GB
Hosted
Memory at 32K
25.0 GB
—
Context
128K
1M (better)
Cost per run
Not measured
$0.46
Cost per case
Not measured
$0.003
Time per case
1m 53s
58.5s (better)
Quantization
Q8_0
—
Harness
llama.cpp
openrouter

The race

No lead changes

Each model's running score as the test cases go by. The dashed line is a perfect run.

LFM2.5 24B A2BQwen3.8 Flash
025507510050100150Perfect run732

Every test case

153 test cases

Each square is one test case, colored by the model that scored higher.

6 won by LFM2.5 24B A2B56 won by Qwen3.8 Flash91 tied