LFM3 1.7B vs Claude Haiku 5.5, head to head

Side A

Side B

Overall

Claude Haiku 5.5 is clearly better. It wins 110 of the 110 test cases where they differ.

Learn to build evals like this in the AI Engineering Academy

Task by task

A score turns orange only when that model wins clearly more of the task's test cases. The time under each score is the average per test case, on the machine it ran on. LFM3 1.7B spends 2.9× as many thinking tokens per test case.

Thinking tokensAnswer tokens

TaskALFM3 1.7BBClaude Haiku 5.5LFM3 1.7BClaude Haiku 5.5

Overall

62m 11s

13.5K3.9K

7025.5s

Agents

Runs multi-step jobs through tools, within a written policy

132m 40s

18.4K3.7K

6139.5s

Structured Extraction

Turns logs, diffs and records into exact JSON

01m 25s

10.1K2.6K

9013.5s

Coding

Writes code and SQL that pass hidden tests

51m 21s

9.5K6.8K

8630.6s

Grounded Answers

Answers from the given documents with sources, never guesses

01m 54s

9.8K2.2K

7613.9s

Instruction Following

Applies every rule, including ones changed mid-conversation

144m 39s

23.1K4.3K

9431.5s

Business Logic

Computes billing, scheduling and pricing rules exactly

01m 37s

11.5K5.5K

6726.0s

Security & permissions

Ignores planted instructions, still does the allowed work

161m 48s

13K2.7K

6723.4s

Diagnosis & investigation

Chooses diagnostic checks, repairs supported causes and verifies recovery

01m 36s

11.6K2.3K

1725.5s

What you'd run

The build behind each score. Orange marks the faster, smaller, cheaper or longer-context side.

SpecALFM3 1.7BBClaude Haiku 5.5

Speed

104 tok/s

225 tok/s

Machine

Apple M5 Pro, 48 GB

Hosted

Memory at 32K

2.8 GB

—

Context

256K

Cost per run

Not measured

$0.42

Cost per case

Not measured

$0.0028

Time per case

2m 11s

25.5s

Quantization

Q8_0

—

Harness

llama.cpp

openrouter

The race

No lead changes

Each model's running score as the test cases go by. The dashed line is a perfect run.

LFM3 1.7BClaude Haiku 5.5

Every test case

153 test cases

Each square is one test case, colored by the model that scored higher.

Agents011

Structured Extraction019

Coding017

Grounded Answers016

Instruction Following020

Business Logic014

Security & permissions012

Diagnosis & investigation01

0 won by LFM3 1.7B110 won by Claude Haiku 5.543 tied