Spark X2.6 1.7B vs Claude Haiku 5.5, head to head

Side A

Side B

Overall

Claude Haiku 5.5 is clearly better. It wins 113 of the 114 test cases where they differ.

Learn to build evals like this in the AI Engineering Academy

Task by task

A score turns orange only when that model wins clearly more of the task's test cases. The time under each score is the average per test case, on the machine it ran on. Spark X2.6 1.7B spends 2.4× as many thinking tokens per test case.

Thinking tokensAnswer tokens

TaskASpark X2.6 1.7BBClaude Haiku 5.5Spark X2.6 1.7BClaude Haiku 5.5

Overall

41m 43s

10.8K3.9K

7025.5s

Agents

Runs multi-step jobs through tools, within a written policy

02m 10s

13.6K3.7K

6139.5s

Structured Extraction

Turns logs, diffs and records into exact JSON

01m 19s

7.3K2.6K

9013.5s

Coding

Writes code and SQL that pass hidden tests

01m 51s

11.6K6.8K

8630.6s

Grounded Answers

Answers from the given documents with sources, never guesses

056.4s

6.1K2.2K

7613.9s

Instruction Following

Applies every rule, including ones changed mid-conversation

242m 53s

18.7K4.3K

9431.5s

Business Logic

Computes billing, scheduling and pricing rules exactly

01m 22s

8.9K5.5K

6726.0s

Security & permissions

Ignores planted instructions, still does the allowed work

1059.4s

6.5K2.7K

6723.4s

Diagnosis & investigation

Chooses diagnostic checks, repairs supported causes and verifies recovery

03m 16s

21.5K2.3K

1725.5s

What you'd run

The build behind each score. Orange marks the faster, smaller, cheaper or longer-context side.

SpecASpark X2.6 1.7BBClaude Haiku 5.5

Speed

106 tok/s

225 tok/s

Machine

Apple M5 Pro, 48 GB

Hosted

Memory at 32K

2.9 GB

—

Context

Cost per run

Not measured

$0.42

Cost per case

Not measured

$0.0028

Time per case

1m 43s

25.5s

Quantization

Q8_0

—

Harness

llama.cpp

openrouter

The race

No lead changes

Each model's running score as the test cases go by. The dashed line is a perfect run.

Spark X2.6 1.7BClaude Haiku 5.5

Every test case

153 test cases

Each square is one test case, colored by the model that scored higher.

Agents013

Structured Extraction019

Coding018

Grounded Answers016

Instruction Following018

Business Logic014

Security & permissions114

Diagnosis & investigation01

1 won by Spark X2.6 1.7B113 won by Claude Haiku 5.539 tied