Head to head

Any two models from the board, on the same tasks and graders.

Side A

Profile

Hostedopenai45 tok/sopenai

Side B

Profile

Hostedqwen64 tok/sopenrouter

Overall

Wins 4 tasks, 20 test cases

Overall

Wins 2 tasks, 37 test cases

Qwen3.8 Flash leads by 6 points, though Qwen3.8 27B wins 4 of 9 tasks.

Learn to build evals like this in the AI Engineering Academy

Task by task

Qwen3.8 27B wins 4 of 9

Average score on each task, best build on each side. The higher score is highlighted. Hover a score for the test cases behind it.

TaskAQwen3.8 27BBQwen3.8 FlashLead

Overall

Tool Calling

Right tool, right args, right sequence?

+14

Structured Output

Valid JSON matching the schema, correct fields?

RAG / Retrieval QA

A correct answer grounded in the given context, with nothing made up?

+18

Context Recall

Finds the one fact buried in a long context?

Coding

Working code that passes hidden unit tests?

+68

Reasoning & Math

Multi-step math and logic with the exact right answer?

Even

Instruction Following

Obeys format, length, and content constraints to the letter?

Even

Classification

Correct label from a fixed set?

Even

Summarization

Compact summary that keeps the facts and invents none?

The race

2 lead changes

Score piled up over the 238 test cases both models ran, as a share of the suite's maximum. The dashed line is a perfect run.

Qwen3.8 27BQwen3.8 Flash

Every test case

238 test cases

One square per test case, in suite order. A square takes the color of the model that scored higher on it.

Tool Calling17

Structured Output21

RAG / Retrieval QA72

Context Recall21

Coding020

Reasoning & Math11

Instruction Following11

Classification22

Summarization42

20 won by Qwen3.8 27B37 won by Qwen3.8 Flash181 tied

Where it breaks

238 shared test cases

On the impossible tier, Qwen3.8 27B passes 29% of test cases and Qwen3.8 Flash passes 35%.

TierAQwen3.8 27BBQwen3.8 FlashLead

Baseline

36 test cases

86%

89%

+3 pts

Hard

69 test cases

57%

Even

Impossible

133 test cases

29%

35%

+6 pts

Tiers come from the suite. Baseline is fair, hard is adversarial, and impossible is built so that nothing solves it. Each number is the share of test cases the model fully passed.

Thinking spend

Qwen3.8 27B spends 1.6× as many thinking tokens per test case.

Qwen3.8 27B

3.4Ktokens

Per test case, 98% of it thinking

Qwen3.8 Flash

3Ktokens

Per test case, 70% of it thinking

TaskAQwen3.8 27BBQwen3.8 Flash

Tool Calling

4.8K88% thinking

6.7K66% thinking

Structured Output

2.6K99% thinking

1.9K67% thinking

RAG / Retrieval QA

2.5K100% thinking

2.4K66% thinking

Context Recall

50398% thinking

55595% thinking

Coding

6.2K100% thinking

5.2K71% thinking

Reasoning & Math

5.4K100% thinking

4.3K73% thinking

Instruction Following

2.9K100% thinking

2.3K69% thinking

Classification

2.7K100% thinking

1.8K72% thinking

Summarization

1.9K99% thinking

1.4K70% thinking

Average output tokens per test case, and how much of it was thinking.

What you'd run

The build behind each score. The better value in each row is highlighted.

SpecAQwen3.8 27BBQwen3.8 Flash

Speed

45 tok/s

64 tok/s

Memory

—

Context

—

Cost per run

$0.5705

Quantization

—

Harness

openai

openrouter

Faster, smaller, cheaper and more context count as better. Quantization and harness only describe the builds.