run
This run
- suite
- starter@v1
- ran at
- 2026-09-10 13:29 UTC
record id
config
Deployment configuration
- harness
- openrouter
- quantization
- —
- context
- 1.0M
- temperature
- 0
- multi-token prediction
- —
- engine
- v1.0.0
- provider
- openrouter
category call budgetreasoningtotal
Tool Calling4.1K/ high8.2K/ call
Structured Output2.0K/ medium4.1K/ call
RAG / Retrieval QA2.0K/ medium4.1K/ call
Context Recall2.0K/ medium4.1K/ call
Coding4.1K/ high8.2K/ call
Reasoning & Math4.1K/ high8.2K/ call
Instruction Following2.0K/ medium4.1K/ call
Classification2.0K/ medium4.1K/ call
Summarization2.0K/ medium4.1K/ call
run this buildOpenRouterhosted
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"deepseek/deepseek-v4.1-flash","messages":[{"role":"user","content":"Hello"}]}'Hosted API — not self-hostable. This is the call the benchmark made; try it live in the Playground.
measured
How it was measured
- latency
- 3578616 ms
- cost
- $0.7017
- memory samples
- —
tok/s and latency measure the scored generation window; per-test figures above are the readings for each case. Memory (VRAM, unified RAM, or host RAM) is the peak footprint polled throughout the run, or blank when no probe could attribute it — never a hand-typed number.
recipe
prompts → hashes
Reproduce this run
The private prompts never leave the suite. What’s public is the command, the pinned suite, and the hashes that stand in for the tests — enough to re-run the exact config against your own copy of the suite.
command
npm run engine -- run --provider openrouter --model deepseek/deepseek-v4.1-flash --suite suites/starter.suite.json --name 'DeepSeek V4.1 Flash'- suite hash
- config hash
- suite pinned to
- starter@v1