Open beta Zabio is in open beta: anyone can sign up and use it in production. Models, prices and features can still change while we learn from the first customers. Tell us what breaks at hello@zabio.ai.
Menu
Start free

Model from DeepSeek

deepseek-flash benchmarks

On Math (hardest), thinking high gets 98.2% right (range 92.3 to 99.6) against 73.3% with thinking off (range 62.6 to 81.9), for 2.9 times the cost.

On SQL and data queries (hardest), thinking low gets 100.0% right (range 95.3 to 100.0) against 88.5% with thinking off (range 79.5 to 93.8), for 5.7 times the cost.

Provider
DeepSeek
Model id
deepseek-flash
Price per 1M tokens
$0.30 in, $1.20 out
Context window
1,000,000
Processing location
China
Zabio default
Thinking off
Tested
25 September 2026

Math

Right answers by difficulty and thinking setting

Each dot is one thinking setting, and its bar is the 90% range.

5060708090100EasyMediumHardHardestEasy, Thinking off: 100.0% (90.2 to 100.0)Easy, Thinking low: 100.0% (90.2 to 100.0)Easy, Thinking high: 100.0% (90.2 to 100.0)Medium, Thinking off: 94.5% (87.1 to 97.8)Medium, Thinking low: 98.2% (92.3 to 99.6)Medium, Thinking high: 96.4% (89.6 to 98.8)Hard, Thinking off: 96.3% (89.4 to 98.8)Hard, Thinking low: 100.0% (95.2 to 100.0)Hard, Thinking high: 100.0% (95.2 to 100.0)Hardest, Thinking off: 73.3% (62.6 to 81.9)Hardest, Thinking low: 96.4% (89.6 to 98.8)Hardest, Thinking high: 98.2% (92.3 to 99.6)

Cost per 1,000 tasks

At list price, by difficulty. Thinking costs more because the model writes more tokens before it answers.

$0.2$1$5EasyEasy, Off: $0.36Off$0.36Easy, Low: $0.27Low$0.27Easy, High: $0.32High$0.32MediumMedium, Off: $0.59Off$0.59Medium, Low: $0.62Low$0.62Medium, High: $1.04High$1.04HardHard, Off: $1.66Off$1.66Hard, Low: $2.26Low$2.26Hard, High: $3.00High$3.00HardestHardest, Off: $3.16Off$3.16Hardest, Low: $6.38Low$6.38Hardest, High: $9.18High$9.18

The numbers behind this chart are in the table below it.

deepseek-flash, Math
DifficultyThinkingRight answers90% rangeQuestionsGroupChoicePer 1,000 tasksRight answers per $1Answer tokensThinking tokens (est.)Median timeCut off
EasyThinking off100.0%90.2 to 100.025Top$0.362,79428202.1 s0.0%
Thinking low100.0%90.2 to 100.025TopBest, Balanced, Cheap$0.273,642381691.6 s0.0%
Thinking high100.0%90.2 to 100.025Top$0.323,154402021.7 s0.0%
MediumThinking off94.5%87.1 to 97.855Near the topCheap$0.591,60647302.3 s0.0%
Thinking low98.2%92.3 to 99.655TopBest, Balanced$0.621,590634281.9 s0.0%
Thinking high96.4%89.6 to 98.855Near the top$1.04924697762.0 s0.0%
HardThinking off96.3%89.4 to 98.854Near the topCheap$1.665791,36204.8 s0.0%
Thinking low100.0%95.2 to 100.054TopBest, Balanced$2.264421391,7155.7 s0.0%
Thinking high100.0%95.2 to 100.054Top$3.003331452,3245.9 s0.0%
HardestThinking off73.3%62.6 to 81.955Below$3.162322,602010.8 s0.0%
Thinking low96.4%89.6 to 98.855Near the topCheap$6.381512235,05917.2 s1.4%
Thinking high98.2%92.3 to 99.655TopBest, Balanced$9.181072437,37222.2 s0.0%

SQL and data queries

Right answers by difficulty and thinking setting

Each dot is one thinking setting, and its bar is the 90% range.

5060708090100EasyMediumHardHardestEasy, Thinking off: 100.0% (90.2 to 100.0)Easy, Thinking low: 100.0% (90.2 to 100.0)Easy, Thinking high: 100.0% (90.2 to 100.0)Medium, Thinking off: 96.4% (89.6 to 98.8)Medium, Thinking low: 96.4% (89.6 to 98.8)Medium, Thinking high: 98.2% (92.3 to 99.6)Hard, Thinking off: 93.9% (86.3 to 97.4)Hard, Thinking low: 100.0% (95.3 to 100.0)Hard, Thinking high: 100.0% (95.3 to 100.0)Hardest, Thinking off: 88.5% (79.5 to 93.8)Hardest, Thinking low: 100.0% (95.3 to 100.0)Hardest, Thinking high: 100.0% (95.3 to 100.0)

Cost per 1,000 tasks

At list price, by difficulty. Thinking costs more because the model writes more tokens before it answers.

$0.2$1$5EasyEasy, Off: $0.24Off$0.24Easy, Low: $0.35Low$0.35Easy, High: $0.42High$0.42MediumMedium, Off: $0.45Off$0.45Medium, Low: $1.26Low$1.26Medium, High: $1.87High$1.87HardHard, Off: $0.58Off$0.58Hard, Low: $2.10Low$2.10Hard, High: $3.48High$3.48HardestHardest, Off: $0.75Off$0.75Hardest, Low: $4.29Low$4.29Hardest, High: $7.42High$7.42

The numbers behind this chart are in the table below it.

deepseek-flash, SQL and data queries
DifficultyThinkingRight answers90% rangeQuestionsGroupChoicePer 1,000 tasksRight answers per $1Answer tokensThinking tokens (est.)Median timeCut off
EasyThinking off100.0%90.2 to 100.025TopBest, Balanced, Cheap$0.244,1243201.1 s0.0%
Thinking low100.0%90.2 to 100.025Top$0.352,85928871.5 s0.0%
Thinking high100.0%90.2 to 100.025Top$0.422,359281491.6 s0.0%
MediumThinking off96.4%89.6 to 98.855Near the topCheap$0.452,12418401.5 s0.0%
Thinking low96.4%89.6 to 98.855Near the top$1.267631467074.0 s0.0%
Thinking high98.2%92.3 to 99.655TopBest, Balanced$1.875241451,2154.7 s0.0%
HardThinking off93.9%86.3 to 97.455Below$0.581,60828301.7 s0.0%
Thinking low100.0%95.3 to 100.055TopBest, Balanced, Cheap$2.104762261,3155.8 s0.0%
Thinking high100.0%95.3 to 100.055Top$3.482872282,4648.5 s0.0%
HardestThinking off88.5%79.5 to 93.855Below$0.751,17941702.2 s0.0%
Thinking low100.0%95.3 to 100.055TopBest, Balanced, Cheap$4.292333143,04912.4 s0.0%
Thinking high100.0%95.3 to 100.055Top$7.421353205,64922.0 s0.0%

Where it is a top choice

Jobs where deepseek-flash is the best, balanced or cheap choice among the models tested so far.

  • Math, EasyBest, Balanced, CheapThinking low, $0.27
  • Math, MediumBest, BalancedThinking low, $0.62
  • Math, MediumCheapThinking off, $0.59
  • Math, HardBest, BalancedThinking low, $2.26
  • Math, HardCheapThinking off, $1.66
  • Math, HardestBest, BalancedThinking high, $9.18
  • Math, HardestCheapThinking low, $6.38
  • SQL and data queries, EasyBest, Balanced, CheapThinking off, $0.24
  • SQL and data queries, MediumBest, BalancedThinking high, $1.87
  • SQL and data queries, MediumCheapThinking off, $0.45
  • SQL and data queries, HardBest, Balanced, CheapThinking low, $2.10
  • SQL and data queries, HardestBest, Balanced, CheapThinking low, $4.29

Stability

7.1%

Different result on a second run. Share of sampled questions that got a different result when asked again.

Compare with

Public results for this model

One bar per kind of job, from the sources named below. Hover a bar to see the benchmarks behind it, with the score each one reports and whether it was run at settings like ours.

Aggregated from public resultsRead on 1 October 2026

Zabio has tested this kind of job too. Those results are further up the page and are counted separately.

deepseek-flash
Writing, Medium: 66.3 +/- 3.9 Evidence, in standard errors: -0.7 LiveBench Language: 70.1 (Same settings as ours) LiveBench paraphrase: 62.4 (Same settings as ours) LMArena text style control, instruction following: 1,427.9 (Same settings as ours)Writing, Medium66.3Creative writing, Medium: 67.3 +/- 6.6 Evidence, in standard errors: -0.2 LiveBench story generation: 67.3 (Same settings as ours) LMArena text style control, creative writing: 1,407.8 (Same settings as ours)Creative writing, Medium67.3Summarizing, Medium: 68.9 +/- 6.6 Evidence, in standard errors: -0.5 LiveBench summarize: 68.9 (Same settings as ours)Summarizing, Medium68.9Pulling out data, Medium: 83.0 +/- 2.1 Evidence, in standard errors: -0.1 LiveBench Data Analysis: 68.0 (Same settings as ours) LiveBench tablereformat: 98.0 (Same settings as ours) LMArena text style control, instruction following: 1,427.9 (Same settings as ours)Pulling out data, Medium83.0Sorting into categories, Medium: 63.1 +/- 3.4 Evidence, in standard errors: -1.2 LiveBench IF: 63.1 (Same settings as ours)Sorting into categories, Medium63.1Conversation, Medium: 33.6 +/- 1.5 Evidence, in standard errors: -0.5 SimpleQA Verified: 33.6 (Different settings) LMArena text style control, multi turn: 1,453.2 (Same settings as ours)Conversation, Medium33.6Analysis and reasoning, Medium: 61.1 +/- 3.5 Evidence, in standard errors: 0.3 SimpleBench: 61.1 (Same settings as ours)Analysis and reasoning, Medium61.1Analysis and reasoning, Hard: 79.0 +/- 2.0 Evidence, in standard errors: 0.3 GPQA diamond: 91.0 (Different settings) LiveBench Reasoning: 70.6 (Same settings as ours) LMArena text style control, hard prompts: 1,458.8 (Same settings as ours)Analysis and reasoning, Hard79.0Analysis and reasoning, Hardest: 7.0 +/- 1.0 Evidence, in standard errors: -1.3 ARC-AGI-2: 2.1 (Same settings as ours) ARC-AGI: 11.8 (Same settings as ours) LMArena text style control, expert: 1,457.1 (Same settings as ours)Analysis and reasoning, Hardest7.0Math, Medium: Rating 1,424.8 Evidence, in standard errors: -0.7 LMArena text style control, math: 1,424.8 (Same settings as ours)Math, MediumRating 1,424.8Math, Hard: 79.7 +/- 2.1 Evidence, in standard errors: -1.0 LiveBench Mathematics: 79.7 (Same settings as ours)Math, Hard79.7Math, Hardest: 76.0 +/- 1.8 Evidence, in standard errors: 1.3 FrontierMath-Tiers-1-3-v2-Private: 57.5 (Different settings) OTIS Mock AIME 2024-2025: 94.4 (Different settings)Math, Hardest76.0Writing code, Hard: 53.4 +/- 3.6 Evidence, in standard errors: -0.6 LiveBench Agentic Coding: 37.6 (Same settings as ours) LiveBench Coding: 69.2 (Same settings as ours) LMArena text style control, coding: 1,483.9 (Same settings as ours) LMArena webdev, overall: 1,620.0 (Different settings)Writing code, Hard53.4Fixing code, Hard: 37.6 +/- 5.7 Evidence, in standard errors: -0.8 LiveBench Agentic Coding: 37.6 (Same settings as ours)Fixing code, Hard37.6SQL and data queries, Medium: 46.1 +/- 6.9 Evidence, in standard errors: -0.3 LiveBench tablejoin: 46.1 (Same settings as ours)SQL and data queries, Medium46.1Questions about a document, Medium: Rating 1,447.4 Evidence, in standard errors: -0.4 LMArena text style control, longer query: 1,447.4 (Same settings as ours)Questions about a document, MediumRating 1,447.4
The benchmarks behind each bar
Kind of jobBenchmarkScoreSettings
Writing, MediumLiveBench Language70.1Same settings as ours
Writing, MediumLiveBench paraphrase62.4Same settings as ours
Writing, MediumLMArena text style control, instruction followingRating 1,427.9Same settings as ours
Creative writing, MediumLiveBench story generation67.3Same settings as ours
Creative writing, MediumLMArena text style control, creative writingRating 1,407.8Same settings as ours
Summarizing, MediumLiveBench summarize68.9Same settings as ours
Pulling out data, MediumLiveBench Data Analysis68.0Same settings as ours
Pulling out data, MediumLiveBench tablereformat98.0Same settings as ours
Pulling out data, MediumLMArena text style control, instruction followingRating 1,427.9Same settings as ours
Sorting into categories, MediumLiveBench IF63.1Same settings as ours
Conversation, MediumSimpleQA Verified33.6Different settings
Conversation, MediumLMArena text style control, multi turnRating 1,453.2Same settings as ours
Analysis and reasoning, MediumSimpleBench61.1Same settings as ours
Analysis and reasoning, HardGPQA diamond91.0Different settings
Analysis and reasoning, HardLiveBench Reasoning70.6Same settings as ours
Analysis and reasoning, HardLMArena text style control, hard promptsRating 1,458.8Same settings as ours
Analysis and reasoning, HardestARC-AGI-22.1Same settings as ours
Analysis and reasoning, HardestARC-AGI11.8Same settings as ours
Analysis and reasoning, HardestLMArena text style control, expertRating 1,457.1Same settings as ours
Math, MediumLMArena text style control, mathRating 1,424.8Same settings as ours
Math, HardLiveBench Mathematics79.7Same settings as ours
Math, HardestFrontierMath-Tiers-1-3-v2-Private57.5Different settings
Math, HardestOTIS Mock AIME 2024-202594.4Different settings
Writing code, HardLiveBench Agentic Coding37.6Same settings as ours
Writing code, HardLiveBench Coding69.2Same settings as ours
Writing code, HardLMArena text style control, codingRating 1,483.9Same settings as ours
Writing code, HardLMArena webdev, overallRating 1,620.0Different settings
Fixing code, HardLiveBench Agentic Coding37.6Same settings as ours
SQL and data queries, MediumLiveBench tablejoin46.1Same settings as ours
Questions about a document, MediumLMArena text style control, longer queryRating 1,447.4Same settings as ours
Price against score in each kind of job (16)

Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.

Writing, Medium
5060708090100$1$10 gpt-5.4-nano: 64.6 +/- 3.9, $0.66 deepseek-flash: 66.3 +/- 3.9, $0.72 deepseek-flash@together: 66.3 +/- 3.9, $0.72 gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24 gemini-3.8-flash: 83.9 +/- 3.1, $2.10 gpt-5.4-mini: 64.9 +/- 3.9, $2.40 deepseek-v4-pro: 67.2 +/- 3.9, $2.64 deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64 grok-4.7: 79.1 +/- 3.4, $4.40 gemini-3.5-flash: 80.1 +/- 3.4, $4.80 claude-sonnet-5: 68.1 +/- 3.9, $5.60 gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40 gpt-5.4: 73.0 +/- 3.7, $8.00 claude-sonnet-4-6: 67.8 +/- 3.9, $8.40 claude-opus-5-5: 75.3 +/- 3.7, $11.20 claude-opus-5: 77.2 +/- 3.6, $14.00 claude-opus-4-8: 73.8 +/- 3.7, $14.00 gpt-5.5: 79.6 +/- 3.5, $16.00 claude-fable-5-1: 80.5 +/- 3.4, $28.00 gpt-6-astra: 83.8 +/- 3.2, $28.00 deepseek-flashgemini-3.8-flash Best, Balanced, Cheap
Creative writing, Medium
5060708090100$1$10 gpt-5.4-nano: 67.7 +/- 6.6, $0.66 deepseek-flash: 67.3 +/- 6.6, $0.72 deepseek-flash@together: 67.3 +/- 6.6, $0.72 gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24 gemini-3.8-flash: 82.8 +/- 5.3, $2.10 gpt-5.4-mini: 66.6 +/- 6.7, $2.40 deepseek-v4-pro: 67.8 +/- 6.6, $2.64 deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64 grok-4.7: 74.9 +/- 6.1, $4.40 gemini-3.5-flash: 75.2 +/- 6.1, $4.80 claude-sonnet-5: 62.3 +/- 6.9, $5.60 gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40 gpt-5.4: 79.0 +/- 5.8, $8.00 claude-sonnet-4-6: 68.9 +/- 6.6, $8.40 claude-opus-5-5: 70.9 +/- 6.4, $11.20 claude-opus-4-8: 80.6 +/- 5.6, $14.00 claude-opus-5: 61.3 +/- 6.9, $14.00 gpt-5.5: 72.6 +/- 6.3, $16.00 claude-fable-5-1: 78.1 +/- 5.9, $28.00 gpt-6-astra: 74.0 +/- 6.2, $28.00 deepseek-flashgemini-3.8-flash Best, Balanced, Cheap
Summarizing, Medium
5060708090100$1$10 gpt-5.4-nano: 71.8 +/- 6.4, $0.73 deepseek-flash: 68.9 +/- 6.6, $0.96 deepseek-flash@together: 68.9 +/- 6.6, $0.96 gemini-3.5-flash-lite: 68.5 +/- 6.6, $1.22 gemini-3.8-flash: 87.5 +/- 4.7, $2.55 gpt-5.4-mini: 59.6 +/- 6.9, $2.70 deepseek-v4-pro: 69.5 +/- 6.5, $3.96 deepseek-v4-pro@together: 69.5 +/- 6.5, $3.96 gemini-3.5-flash: 82.6 +/- 5.4, $5.40 grok-4.7: 78.2 +/- 5.8, $6.60 claude-sonnet-5: 71.2 +/- 6.4, $6.80 gemini-3.1-pro-preview: 88.2 +/- 4.6, $7.20 gpt-5.4: 68.4 +/- 6.6, $9.00 claude-sonnet-4-6: 61.7 +/- 6.9, $10.20 claude-opus-5-5: 69.1 +/- 6.5, $13.60 claude-opus-4-8: 77.7 +/- 5.9, $17.00 claude-opus-5: 66.4 +/- 6.7, $17.00 gpt-5.5: 72.3 +/- 6.3, $18.00 gpt-6-astra: 79.6 +/- 5.7, $34.00 claude-fable-5-1: 72.2 +/- 6.3, $34.00 gpt-5.4-nano Cheapdeepseek-flashgemini-3.8-flash Best, Balanced
Pulling out data, Medium
5060708090100$1$10 gpt-5.4-nano: 82.8 +/- 2.1, $0.41 deepseek-flash: 83.0 +/- 2.1, $0.48 deepseek-flash@together: 83.0 +/- 2.1, $0.48 gemini-3.5-flash-lite: 75.7 +/- 2.3, $0.74 gemini-3.8-flash: 75.1 +/- 2.5, $1.35 gpt-5.4-mini: 85.4 +/- 1.9, $1.50 deepseek-v4-pro: 87.3 +/- 1.9, $1.85 deepseek-v4-pro@together: 87.3 +/- 1.9, $1.85 gemini-3.5-flash: 80.5 +/- 2.4, $3.00 grok-4.7: 88.4 +/- 1.8, $3.08 claude-sonnet-5: 84.9 +/- 2.1, $3.60 gemini-3.1-pro-preview: 88.3 +/- 1.9, $4.00 gpt-5.4: 89.7 +/- 1.7, $5.00 claude-sonnet-4-6: 87.0 +/- 2.2, $5.40 claude-opus-5-5: 89.2 +/- 1.9, $7.20 claude-opus-5: 84.3 +/- 2.4, $9.00 claude-opus-4-8: 80.1 +/- 2.5, $9.00 gpt-5.5: 90.8 +/- 1.7, $10.00 gpt-6-astra: 91.5 +/- 1.6, $18.00 claude-fable-5-1: 87.2 +/- 2.3, $18.00 deepseek-flashgpt-5.4-mini Cheapdeepseek-v4-pro Balancedclaude-opus-5-5 Best
Sorting into categories, Medium
5060708090100$1$10 gpt-5.4-nano: 67.2 +/- 3.3, $0.41 deepseek-flash: 63.1 +/- 3.4, $0.48 deepseek-flash@together: 63.1 +/- 3.4, $0.48 gemini-3.5-flash-lite: 67.2 +/- 3.3, $0.74 gemini-3.8-flash: 81.4 +/- 2.8, $1.35 gpt-5.4-mini: 59.8 +/- 3.5, $1.50 deepseek-v4-pro: 62.4 +/- 3.4, $1.85 deepseek-v4-pro@together: 62.4 +/- 3.4, $1.85 gemini-3.5-flash: 75.6 +/- 3.0, $3.00 grok-4.7: 75.3 +/- 3.1, $3.08 claude-sonnet-5: 63.9 +/- 3.4, $3.60 gemini-3.1-pro-preview: 79.1 +/- 2.9, $4.00 gpt-5.4: 70.2 +/- 3.2, $5.00 claude-sonnet-4-6: 63.2 +/- 3.4, $5.40 claude-opus-5-5: 65.7 +/- 3.4, $7.20 claude-opus-4-8: 72.0 +/- 3.2, $9.00 claude-opus-5: 63.8 +/- 3.4, $9.00 gpt-5.5: 70.7 +/- 3.2, $10.00 gpt-6-astra: 75.6 +/- 3.0, $18.00 claude-fable-5-1: 73.0 +/- 3.1, $18.00 deepseek-flashgemini-3.8-flash Best, Balanced, Cheap
Conversation, Medium
0255075100$1$10 gpt-4o-mini: 8.3 +/- 0.9, $0.48 gpt-5.4-nano: 11.7 +/- 1.0, $0.82 deepseek-flash: 33.6 +/- 1.5, $0.96 deepseek-flash@together: 33.6 +/- 1.5, $0.96 gpt-5-mini: 21.6 +/- 1.3, $1.20 gpt-4.1-mini: 12.7 +/- 1.1, $1.28 gemini-3.8-flash: 69.7 +/- 1.5, $2.70 gpt-5.4-mini: 29.4 +/- 1.4, $3.00 claude-haiku-4-5: 13.2 +/- 1.1, $3.60 deepseek-v4-pro: 52.9 +/- 1.6, $3.70 deepseek-v4-pro@together: 52.9 +/- 1.6, $3.70 gemini-3.5-flash: 66.2 +/- 1.5, $6.00 gpt-4.1: 31.1 +/- 1.5, $6.40 claude-sonnet-5: 33.7 +/- 1.5, $7.20 gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00 gpt-5.4: 45.1 +/- 1.6, $10.00 claude-sonnet-4-6: 32.8 +/- 1.5, $10.80 claude-opus-5-5: 72.2 +/- 1.4, $14.40 claude-opus-4-8: 53.0 +/- 1.6, $18.00 claude-opus-5: 59.9 +/- 1.6, $18.00 gpt-5.5: 63.0 +/- 1.5, $20.00 gpt-6-astra: 75.6 +/- 1.4, $36.00 claude-fable-5-1: 70.8 +/- 1.4, $36.00 deepseek-flashgemini-3.8-flash Cheapgemini-3.1-pro-preview Best, Balanced
Analysis and reasoning, Medium
0255075100$1$10 gpt-oss-120b: 22.1 +/- 2.9, $0.36 gpt-4o-mini: 10.7 +/- 2.2, $0.36 deepseek-flash: 61.1 +/- 3.5, $0.72 deepseek-flash@together: 61.1 +/- 3.5, $0.72 mistral-large-2512: 22.5 +/- 3.0, $1.00 llama-3.3-70b-instruct-turbo: 19.9 +/- 2.8, $1.25 gemini-3.5-flash: 76.7 +/- 3.0, $4.80 gpt-4.1: 27.0 +/- 3.1, $4.80 claude-sonnet-5: 60.6 +/- 3.5, $5.60 gemini-3.1-pro-preview: 79.6 +/- 2.9, $6.40 claude-opus-5: 80.6 +/- 2.8, $14.00 claude-opus-4-8: 64.8 +/- 3.4, $14.00 gpt-5.5: 69.0 +/- 3.3, $16.00 deepseek-flashgemini-3.5-flash Best, Balanced, Cheap
Analysis and reasoning, Hard
0255075100$1$10$100 qwen3.5-9b: 79.0 +/- 2.4, $0.57 gpt-oss-120b: 75.8 +/- 2.7, $0.84 mistral-small-2603: 47.5 +/- 2.8, $0.84 gpt-4o-mini: 37.7 +/- 2.4, $0.84 gpt-5.4-nano: 66.1 +/- 2.4, $1.53 deepseek-flash: 79.0 +/- 2.0, $1.68 deepseek-flash@together: 79.0 +/- 2.0, $1.68 gpt-4.1-mini: 65.9 +/- 2.7, $2.24 gpt-5-mini: 71.7 +/- 3.2, $2.30 mistral-large-2512: 51.3 +/- 2.7, $2.35 gemini-3.5-flash-lite: 68.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02 gemini-3.8-flash: 92.3 +/- 1.3, $4.88 gpt-5.4-mini: 67.1 +/- 2.4, $5.55 deepseek-v4-pro: 78.0 +/- 2.1, $6.20 deepseek-v4-pro@together: 78.0 +/- 2.1, $6.20 claude-haiku-4-5: 60.5 +/- 2.8, $6.50 mistral-medium-3-5: 59.5 +/- 2.8, $9.75 grok-4.7: 82.7 +/- 2.7, $10.34 gemini-3.5-flash: 85.8 +/- 1.7, $11.10 gpt-4.1: 66.9 +/- 2.8, $11.20 claude-sonnet-5: 84.5 +/- 1.8, $13.00 gemini-3.1-pro-preview: 89.5 +/- 1.5, $14.80 gpt-5.4: 80.3 +/- 2.1, $18.50 claude-sonnet-4-6: 81.8 +/- 1.9, $19.50 claude-opus-5-5: 91.4 +/- 1.4, $26.00 claude-opus-5: 92.2 +/- 1.4, $32.50 claude-opus-4-8: 86.9 +/- 1.7, $32.50 gpt-5.5: 82.4 +/- 2.0, $37.00 gpt-6-astra: 94.2 +/- 1.2, $65.00 claude-fable-5-1: 91.7 +/- 1.9, $65.00 deepseek-flashgemini-3.8-flash Cheapclaude-opus-5-5 Best, Balanced
Analysis and reasoning, Hardest
0255075100$1$10$100 gpt-4o-mini: 0.0 +/- 0.4, $1.35 gpt-5.4-nano: 9.9 +/- 1.1, $2.48 deepseek-flash: 7.0 +/- 1.0, $2.70 deepseek-flash@together: 7.0 +/- 1.0, $2.70 gpt-4.1-mini: 1.8 +/- 0.5, $3.60 gpt-5-mini: 15.8 +/- 0.9, $3.75 gemini-3.5-flash-lite: 9.3 +/- 1.1, $4.65 gemini-3.8-flash: 44.5 +/- 2.0, $7.88 gpt-5.4-mini: 7.1 +/- 1.0, $9.00 deepseek-v4-pro: 6.9 +/- 0.9, $9.90 deepseek-v4-pro@together: 6.9 +/- 0.9, $9.90 claude-haiku-4-5: 7.8 +/- 1.0, $10.50 mistral-medium-3-5: 4.5 +/- 0.8, $15.75 gemini-3.5-flash: 28.9 +/- 1.8, $18.00 gpt-4.1: 3.8 +/- 0.5, $18.00 gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00 gpt-5.4: 45.0 +/- 1.8, $30.00 claude-sonnet-4-6: 73.5 +/- 2.4, $31.50 claude-opus-5-5: 94.6 +/- 1.3, $42.00 claude-opus-5: 92.9 +/- 1.5, $52.50 claude-opus-4-8: 75.1 +/- 2.4, $52.50 gpt-5.5: 54.8 +/- 2.4, $60.00 claude-fable-5-1: 73.2 +/- 1.6, $105 gpt-6-astra: 66.8 +/- 1.7, $105 deepseek-flashgemini-3.8-flash Cheapclaude-opus-5-5 Best, Balanced

Math, Medium: Nobody publishes a scored result for this one yet. The models below have a preference rating only.

Math, Hard
0255075100$1$10$100 gpt-4o-mini: 52.6 +/- 1.1, $0.84 mistral-small-2603: 46.8 +/- 1.2, $0.84 gpt-5.4-nano: 91.0 +/- 1.5, $1.53 deepseek-flash: 79.7 +/- 2.1, $1.68 deepseek-flash@together: 79.7 +/- 2.1, $1.68 gpt-4.1-mini: 87.3 +/- 0.7, $2.24 gpt-5-mini: 97.9 +/- 0.3, $2.30 mistral-large-2512: 50.3 +/- 1.2, $2.35 gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02 gemini-3.8-flash: 91.6 +/- 1.4, $4.88 gpt-5.4-mini: 78.5 +/- 2.1, $5.55 deepseek-v4-pro: 90.7 +/- 1.5, $6.20 deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20 claude-haiku-4-5: 86.9 +/- 0.8, $6.50 mistral-medium-3-5: 81.6 +/- 0.9, $9.75 grok-4.7: 95.7 +/- 1.0, $10.34 gemini-3.5-flash: 88.2 +/- 1.7, $11.10 gpt-4.1: 83.0 +/- 0.9, $11.20 claude-sonnet-5: 92.9 +/- 1.3, $13.00 gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80 gpt-5.4: 94.2 +/- 1.2, $18.50 claude-sonnet-4-6: 87.0 +/- 1.7, $19.50 claude-opus-5-5: 97.1 +/- 0.9, $26.00 claude-opus-5: 95.7 +/- 1.0, $32.50 claude-opus-4-8: 94.3 +/- 1.2, $32.50 gpt-5.5: 95.9 +/- 1.0, $37.00 claude-fable-5-1: 97.0 +/- 0.9, $65.00 gpt-6-astra: 96.8 +/- 0.9, $65.00 deepseek-flashgpt-5-mini Best, Balanced, Cheap
Math, Hardest
0255075100$1$10$100 qwen3.5-9b: 42.8 +/- 6.5, $0.89 gpt-oss-120b: 88.9 +/- 4.5, $1.35 mistral-small-2603: 5.3 +/- 2.6, $1.35 gpt-4o-mini: 3.8 +/- 1.4, $1.35 gpt-5.4-nano: 25.7 +/- 2.9, $2.48 deepseek-flash: 76.0 +/- 1.8, $2.70 deepseek-flash@together: 76.0 +/- 1.8, $2.70 gpt-4.1-mini: 18.6 +/- 2.2, $3.60 gpt-5-mini: 34.1 +/- 2.9, $3.75 mistral-large-2512: 1.1 +/- 0.5, $3.75 gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65 llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68 gemini-3.8-flash: 83.7 +/- 1.5, $7.88 gpt-5.4-mini: 23.6 +/- 2.7, $9.00 deepseek-v4-pro: 54.0 +/- 4.6, $9.90 deepseek-v4-pro@together: 54.0 +/- 4.6, $9.90 claude-haiku-4-5: 20.0 +/- 3.2, $10.50 mistral-medium-3-5: 16.3 +/- 3.0, $15.75 gemini-3.5-flash: 65.3 +/- 2.2, $18.00 gpt-4.1: 16.6 +/- 2.2, $18.00 claude-sonnet-5: 72.8 +/- 3.3, $21.00 gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00 gpt-5.4: 60.9 +/- 3.3, $30.00 claude-sonnet-4-6: 51.8 +/- 3.7, $31.50 claude-opus-5-5: 95.6 +/- 2.6, $42.00 claude-opus-5: 92.8 +/- 1.6, $52.50 claude-opus-4-8: 72.3 +/- 2.5, $52.50 gpt-5.5: 69.1 +/- 4.5, $60.00 gpt-6-astra: 96.8 +/- 2.6, $105 claude-fable-5-1: 95.1 +/- 2.7, $105 gpt-oss-120b Cheapdeepseek-flash Balancedclaude-opus-5-5 Best
Writing code, Hard
0255075100$2$5$10$20$50$100 gpt-5.4-nano: 58.8 +/- 3.6, $2.09 deepseek-flash: 53.4 +/- 3.6, $2.22 deepseek-flash@together: 53.4 +/- 3.6, $2.22 gemini-3.5-flash-lite: 60.7 +/- 3.5, $3.98 gemini-3.8-flash: 63.4 +/- 3.6, $6.56 gpt-5.4-mini: 56.7 +/- 3.6, $7.58 deepseek-v4-pro: 56.3 +/- 3.6, $7.99 deepseek-v4-pro@together: 56.3 +/- 3.6, $7.99 grok-4.7: 65.6 +/- 3.5, $13.31 gemini-3.5-flash: 63.6 +/- 3.5, $15.15 claude-sonnet-5: 70.0 +/- 3.4, $17.50 gemini-3.1-pro-preview: 43.2 +/- 2.8, $20.20 gpt-5.4: 49.0 +/- 3.5, $25.25 claude-sonnet-4-6: 61.0 +/- 3.5, $26.25 claude-opus-5-5: 79.5 +/- 3.4, $35.00 claude-opus-5: 73.3 +/- 3.3, $43.75 claude-opus-4-8: 66.2 +/- 3.4, $43.75 gpt-5.5: 48.7 +/- 3.1, $50.50 claude-fable-5-1: 75.3 +/- 3.7, $87.50 gpt-6-astra: 61.5 +/- 4.6, $87.50 deepseek-flashclaude-opus-5-5 Best, Balanced, Cheap
Fixing code, Hard
0255075100$1$10$100 qwen3.5-9b: 9.2 +/- 2.4, $0.68 gpt-oss-120b: 25.6 +/- 1.9, $1.11 gpt-4o-mini: 3.6 +/- 1.2, $1.11 codestral-2508: 11.1 +/- 2.1, $1.82 gpt-5.4-nano: 46.8 +/- 5.9, $2.09 deepseek-flash: 37.6 +/- 5.7, $2.22 deepseek-flash@together: 37.6 +/- 5.7, $2.22 gpt-4.1-mini: 32.4 +/- 3.1, $2.96 gpt-5-mini: 47.0 +/- 1.7, $3.20 gemini-3.5-flash-lite: 45.3 +/- 5.9, $3.98 gemini-3.8-flash: 54.2 +/- 5.9, $6.56 gpt-5.4-mini: 41.7 +/- 5.8, $7.58 deepseek-v4-pro: 57.1 +/- 3.5, $7.99 deepseek-v4-pro@together: 57.1 +/- 3.5, $7.99 claude-haiku-4-5: 13.9 +/- 2.7, $8.75 grok-4.7: 54.0 +/- 5.9, $13.31 gpt-4.1: 49.8 +/- 1.9, $14.80 gemini-3.5-flash: 64.2 +/- 3.1, $15.15 claude-sonnet-5: 59.4 +/- 5.8, $17.50 gemini-3.1-pro-preview: 49.8 +/- 4.0, $20.20 gpt-5.4: 69.7 +/- 2.3, $25.25 claude-sonnet-4-6: 59.9 +/- 2.2, $26.25 claude-opus-5-5: 71.7 +/- 5.3, $35.00 claude-opus-5: 65.2 +/- 5.6, $43.75 claude-opus-4-8: 50.5 +/- 5.9, $43.75 gpt-5.5: 59.0 +/- 3.6, $50.50 claude-fable-5-1: 66.1 +/- 5.6, $87.50 gpt-6-astra: 57.3 +/- 5.8, $87.50 deepseek-flashclaude-sonnet-5 Cheapclaude-opus-5-5 Best, Balanced
SQL and data queries, Medium
0255075100$1$10 gpt-5.4-nano: 50.6 +/- 6.9, $0.41 deepseek-flash: 46.1 +/- 6.9, $0.48 deepseek-flash@together: 46.1 +/- 6.9, $0.48 gemini-3.5-flash-lite: 40.6 +/- 6.8, $0.74 gemini-3.8-flash: 45.3 +/- 6.9, $1.35 gpt-5.4-mini: 49.4 +/- 6.9, $1.50 deepseek-v4-pro: 48.2 +/- 6.9, $1.85 deepseek-v4-pro@together: 48.2 +/- 6.9, $1.85 gemini-3.5-flash: 50.5 +/- 6.9, $3.00 grok-4.7: 44.9 +/- 6.9, $3.08 claude-sonnet-5: 42.4 +/- 6.9, $3.60 gemini-3.1-pro-preview: 52.4 +/- 6.9, $4.00 gpt-5.4: 51.8 +/- 6.9, $5.00 claude-sonnet-4-6: 45.1 +/- 6.9, $5.40 claude-opus-5-5: 52.3 +/- 6.9, $7.20 claude-opus-5: 52.0 +/- 6.9, $9.00 claude-opus-4-8: 51.1 +/- 6.9, $9.00 gpt-5.5: 55.9 +/- 6.9, $10.00 gpt-6-astra: 58.4 +/- 6.8, $18.00 claude-fable-5-1: 56.2 +/- 6.9, $18.00 gpt-5.4-nano Best, Balanced, Cheapdeepseek-flash

Questions about a document, Medium: Nobody publishes a scored result for this one yet. The models below have a preference rating only.

Sources

What these numbers are

  • These numbers come from the sources named on each row, not from Zabio's own tests. Zabio's tests are shown separately and never combined with them into one figure.
  • Sources test models at their own settings. A settings mismatch against how Zabio calls the model is marked on the row and lowers its weight.
  • Scores are compared only inside one benchmark. Task scores combine benchmarks by evidence in standard errors, so a task score is a summary, not a benchmark result.
  • A model missing from every source is listed as such rather than left out.

Results from our own tests have their own section.

How we tested

Called through Zabio's own API at each thinking setting, question set pilot-2026-09-25, methodology 1.0. Without a thinking setting in the request, Zabio uses thinking off.

Model id to use through Zabio: deepseek-flash

Changes

  • 25 September 2026First results: math and SQL at three thinking settings.

Read this first

  • AIME problems are public and may appear in model training data, so math results at the frontier level may be inflated. Differences between settings are the reliable part.
  • SQL questions were kept only when an independent model outside the test answered them correctly, which limits how hard the hardest questions are.
  • These results compare DeepSeek models and settings with each other. Other providers' models have not been tested in this run.
  • Results are not ranked when their ranges overlap. Top means nothing tested is probably better on that job. Near the top means not clearly worse than the top.

Zabio sells access to these models and earns a fee on the traffic it carries. That is why the method is public, every number has its range and date, and corrections are published as changes. No vendor pays for tests, results or placement.

DeepSeek is a trademark of its owner. Zabio uses DeepSeek models through the DeepSeek API and is not affiliated with DeepSeek.

Use the data

The results are free to reuse under CC BY 4.0. Please credit Zabio with the citation below.

License
CC BY 4.0
Suggested citation
Zabio, Model benchmarks, zabio.ai/benchmarks, 2026-09-25
Methodology
1.0
Tested
25 September 2026
Download benchmarks.json