Mistral Large 4 run record

Mistral Large 4

mistralai/mistral-large-4-0

starter@v1HostedOpenRouterserved by Mistral512K context2026-10-06 15:26 UTC

Overall score

43 of 153 test cases fully passed

Throughput

71.9tok/s

Mean per test case, 56 tok/s over the run

Output per test

6.2Ktokens

Answer plus reasoning

Reasoning per test

4.4Ktokens

72% of the output

Total tokens

1M in, 946.5K out

Run time

4h 54m

1m 56s per caseRequest time, summed

Cost

$2.67

$0.017 per caseAt the provider's list price

Every test case

153 test cases in 8 tasks

One square per test case. Select a square to read the answer.

PassedPartial creditFailed

11Agents1 of 21 passed

1 of 21 passed

15Structured Extraction4 of 21 passed

4 of 21 passed

33Coding5 of 21 passed

5 of 21 passed

38Grounded Answers8 of 21 passed

8 of 21 passed

69Instruction Following13 of 21 passed

13 of 21 passed

24Business Logic5 of 21 passed

5 of 21 passed

33Security & permissions7 of 21 passed

7 of 21 passed

0Diagnosis & investigation0 of 6 passed

0 of 6 passed

Speed and output per test case

One bar per test case. Hover to inspect a case, select it to read the answer.

ThroughputPeak 221 tok/s

Output tokensReasoningAnswer

Test by test

Every answer with its reasoning and the checks it had to pass.

Reproduce this run

The prompts stay private so models can't train on them. The hashes pin the exact suite and config this run used.

Serve the modelOpenRouterHosted

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"mistralai/mistral-large-4-0","provider":{"order":["mistral"],"allow_fallbacks":false},"service_tier":"default","messages":[{"role":"user","content":"Hello"}]}'

The request the benchmark sent, pinned to the provider that served it.

Record id
Suite hash, starter@v1
Config hash

Configuration

Where the suite ran, and the token budget each task got.

Deployment: Hosted
API: OpenRouter
Serving provider: Mistral
Precision: Provider default
Context window: 512K
Temperature: 0
Bench version: v1.0.0
Ran at: 2026-10-06 15:26 UTC

Provider parameters

{
  "provider": {
    "order": [
      "mistral"
    ],
    "allow_fallbacks": false
  },
  "service_tier": "default"
}

Token budget per task
Task	Reasoning	Max output per call
Agents	4.1Khigh	12.3K
Structured Extraction	4.1Khigh	12.3K
Coding	4.1Khigh	12.3K
Grounded Answers	4.1Khigh	12.3K
Instruction Following	4.1Khigh	12.3K
Business Logic	4.1Khigh	12.3K
Security & permissions	4.1Khigh	12.3K
Diagnosis & investigation	4.1Khigh	12.3K

Mistral Large 4 Compare with another model

Every test case

Speed and output per test case

Test by test

11AgentsRuns multi-step jobs through tools, within a written policy1 of 21 passedp90 latency 48.5s

15Structured ExtractionTurns logs, diffs and records into exact JSON4 of 21 passedp90 latency 6.14s

33CodingWrites code and SQL that pass hidden tests5 of 21 passedp90 latency 4m 8s

38Grounded AnswersAnswers from the given documents with sources, never guesses8 of 21 passedp90 latency 4m 20s

69Instruction FollowingApplies every rule, including ones changed mid-conversation13 of 21 passedp90 latency 5m 35s

24Business LogicComputes billing, scheduling and pricing rules exactly5 of 21 passedp90 latency 3m 59s

33Security & permissionsIgnores planted instructions, still does the allowed work7 of 21 passedp90 latency 3m 21s

0Diagnosis & investigationChooses diagnostic checks, repairs supported causes and verifies recovery0 of 6 passedp90 latency 3m 27s

Reproduce this run

Configuration