Claude Haiku 5.5 vs LFM3 1.7B, head to head

Side A

Side B

Overall

Claude Haiku 5.5 is clearly better. It wins 110 of the 110 test cases where they differ.

Learn to build evals like this in the AI Engineering Academy

Task by task

A score turns orange only when that model wins clearly more of the task's test cases. The time under each score is the average per test case, on the machine it ran on. LFM3 1.7B spends 2.9× as many thinking tokens per test case.

Thinking tokensAnswer tokens

TaskAClaude Haiku 5.5BLFM3 1.7BClaude Haiku 5.5LFM3 1.7B

Overall

7025.5s

3.9K13.5K

62m 11s

Agents

Runs multi-step jobs through tools, within a written policy

6139.5s

3.7K18.4K

132m 40s

Structured Extraction

Turns logs, diffs and records into exact JSON

9013.5s

2.6K10.1K

01m 25s

Coding

Writes code and SQL that pass hidden tests

8630.6s

6.8K9.5K

51m 21s

Grounded Answers

Answers from the given documents with sources, never guesses

7613.9s

2.2K9.8K

01m 54s

Instruction Following

Applies every rule, including ones changed mid-conversation

9431.5s

4.3K23.1K

144m 39s

Business Logic

Computes billing, scheduling and pricing rules exactly

6726.0s

5.5K11.5K

01m 37s

Security & permissions

Ignores planted instructions, still does the allowed work

6723.4s

2.7K13K

161m 48s

Diagnosis & investigation

Chooses diagnostic checks, repairs supported causes and verifies recovery

1725.5s

2.3K11.6K

01m 36s

What you'd run

The build behind each score. Orange marks the faster, smaller, cheaper or longer-context side.

SpecAClaude Haiku 5.5BLFM3 1.7B

Speed

225 tok/s

104 tok/s

Machine

Hosted

Apple M5 Pro, 48 GB

Memory at 32K

—

2.8 GB

Context

256K

Cost per run

$0.42

Not measured

Cost per case

$0.0028

Not measured

Time per case

25.5s

2m 11s

Quantization

—

Q8_0

Harness

openrouter

llama.cpp

The race

No lead changes

Each model's running score as the test cases go by. The dashed line is a perfect run.

Claude Haiku 5.5LFM3 1.7B

Every test case

153 test cases

Each square is one test case, colored by the model that scored higher.

Agents110

Structured Extraction190

Coding170

Grounded Answers160

Instruction Following200

Business Logic140

Security & permissions120

Diagnosis & investigation10

110 won by Claude Haiku 5.50 won by LFM3 1.7B43 tied