Open beta Zabio is in open beta: anyone can sign up and use it in production. Models, prices and features can still change while we learn from the first customers. Tell us what breaks at hello@zabio.ai.
Menu
Start free

Model from DeepSeek

deepseek-v4-pro benchmarks

On Math (hardest), thinking high gets 98.2% right (range 92.3 to 99.6) against 44.2% with thinking off (range 33.8 to 55.3), for 7.5 times the cost.

On SQL and data queries (hardest), thinking high gets 98.2% right (range 92.3 to 99.6) against 84.8% with thinking off (range 75.3 to 91.1), for 13.7 times the cost.

Provider
DeepSeek
Model id
deepseek-v4-pro
Price per 1M tokens
$1.32 in, $3.96 out
Context window
1,000,000
Processing location
China
Zabio default
Thinking off
Tested
25 September 2026

Math

Right answers by difficulty and thinking setting

Each dot is one thinking setting, and its bar is the 90% range.

0255075100EasyMediumHardHardestEasy, Thinking off: 96.0% (83.9 to 99.1)Easy, Thinking low: 100.0% (90.2 to 100.0)Easy, Thinking high: 100.0% (90.2 to 100.0)Medium, Thinking off: 94.5% (87.1 to 97.8)Medium, Thinking low: 98.2% (92.3 to 99.6)Medium, Thinking high: 98.2% (92.3 to 99.6)Hard, Thinking off: 86.4% (77.0 to 92.4)Hard, Thinking low: 98.1% (92.0 to 99.6)Hard, Thinking high: 98.2% (92.3 to 99.6)Hardest, Thinking off: 44.2% (33.8 to 55.3)Hardest, Thinking low: 89.1% (80.3 to 94.3)Hardest, Thinking high: 98.2% (92.3 to 99.6)

Cost per 1,000 tasks

At list price, by difficulty. Thinking costs more because the model writes more tokens before it answers.

$1$10EasyEasy, Off: $1.04Off$1.04Easy, Low: $0.81Low$0.81Easy, High: $1.28High$1.28MediumMedium, Off: $1.60Off$1.60Medium, Low: $1.57Low$1.57Medium, High: $3.44High$3.44HardHard, Off: $3.94Off$3.94Hard, Low: $5.98Low$5.98Hard, High: $15.63High$15.63HardestHardest, Off: $6.52Off$6.52Hardest, Low: $25.10Low$25.10Hardest, High: $48.68High$48.68

The numbers behind this chart are in the table below it.

deepseek-v4-pro, Math
DifficultyThinkingRight answers90% rangeQuestionsGroupChoicePer 1,000 tasksRight answers per $1Answer tokensThinking tokens (est.)Median timeCut off
EasyThinking off96.0%83.9 to 99.125Near the top$1.0492124202.9 s0.0%
Thinking low100.0%90.2 to 100.025Top$0.811,238361462.1 s0.0%
Thinking high100.0%90.2 to 100.025Top$1.28779402362.7 s0.0%
MediumThinking off94.5%87.1 to 97.855Near the top$1.6059138103.3 s0.0%
Thinking low98.2%92.3 to 99.655Top$1.57626593142.9 s0.0%
Thinking high98.2%92.3 to 99.655Top$3.44285697514.0 s0.0%
HardThinking off86.4%77.0 to 92.454Below$3.9422096006.8 s0.0%
Thinking low98.1%92.0 to 99.653Near the top$5.981641121,36410.0 s0.0%
Thinking high98.2%92.3 to 99.655Top$15.63631413,74717.7 s0.0%
HardestThinking off44.2%33.8 to 55.355Below$6.52681,611011.6 s0.0%
Thinking low89.1%80.3 to 94.355Below$25.10361666,13632.6 s6.8%
Thinking high98.2%92.3 to 99.655Top$48.682025111,97857.0 s1.4%

SQL and data queries

Right answers by difficulty and thinking setting

Each dot is one thinking setting, and its bar is the 90% range.

5060708090100EasyMediumHardHardestEasy, Thinking off: 100.0% (90.2 to 100.0)Easy, Thinking low: 100.0% (90.2 to 100.0)Easy, Thinking high: 100.0% (90.2 to 100.0)Medium, Thinking off: 90.9% (82.5 to 95.5)Medium, Thinking low: 98.2% (92.3 to 99.6)Medium, Thinking high: 98.2% (92.3 to 99.6)Hard, Thinking off: 84.8% (75.3 to 91.1)Hard, Thinking low: 97.6% (91.3 to 99.4)Hard, Thinking high: 100.0% (95.3 to 100.0)Hardest, Thinking off: 84.8% (75.3 to 91.1)Hardest, Thinking low: 94.5% (87.1 to 97.8)Hardest, Thinking high: 98.2% (92.3 to 99.6)

Cost per 1,000 tasks

At list price, by difficulty. Thinking costs more because the model writes more tokens before it answers.

$1$5$20EasyEasy, Off: $1.03Off$1.03Easy, Low: $1.76Low$1.76Easy, High: $2.35High$2.35MediumMedium, Off: $1.76Off$1.76Medium, Low: $8.92Low$8.92Medium, High: $12.51High$12.51HardHard, Off: $2.24Off$2.24Hard, Low: $13.45Low$13.45Hard, High: $22.80High$22.80HardestHardest, Off: $3.06Off$3.06Hardest, Low: $26.46Low$26.46Hardest, High: $41.95High$41.95

The numbers behind this chart are in the table below it.

deepseek-v4-pro, SQL and data queries
DifficultyThinkingRight answers90% rangeQuestionsGroupChoicePer 1,000 tasksRight answers per $1Answer tokensThinking tokens (est.)Median timeCut off
EasyThinking off100.0%90.2 to 100.025Top$1.039743301.1 s0.0%
Thinking low100.0%90.2 to 100.025Top$1.76569281902.2 s0.0%
Thinking high100.0%90.2 to 100.025Top$2.35425283132.2 s0.0%
MediumThinking off90.9%82.5 to 95.555Below$1.7651618601.9 s0.0%
Thinking low98.2%92.3 to 99.655Top$8.921101451,84913.7 s0.0%
Thinking high98.2%92.3 to 99.655Top$12.51791462,72915.2 s0.0%
HardThinking off84.8%75.3 to 91.155Below$2.2437929302.3 s0.0%
Thinking low97.6%91.3 to 99.455Near the top$13.45732252,90117.9 s0.0%
Thinking high100.0%95.3 to 100.055Top$22.80442295,23025.6 s0.0%
HardestThinking off84.8%75.3 to 91.155Below$3.0627849403.2 s0.0%
Thinking low94.5%87.1 to 97.855Below$26.46363266,07747.1 s0.0%
Thinking high98.2%92.3 to 99.655Near the top$41.95233269,96372.9 s0.0%

Where it is a top choice

Jobs where deepseek-v4-pro is the best, balanced or cheap choice among the models tested so far.

Not the best, balanced or cheap choice for any job tested so far.

Stability

12.7%

Different result on a second run. Share of sampled questions that got a different result when asked again.

Compare with

Public results for this model

One bar per kind of job, from the sources named below. Hover a bar to see the benchmarks behind it, with the score each one reports and whether it was run at settings like ours.

Aggregated from public resultsRead on 1 October 2026

Zabio has tested this kind of job too. Those results are further up the page and are counted separately.

deepseek-v4-pro
Writing, Medium: 67.2 +/- 3.9 Evidence, in standard errors: 0.0 LiveBench Language: 78.1 (Same settings as ours) LiveBench paraphrase: 56.2 (Same settings as ours) LMArena text style control, instruction following: 1,452.4 (Same settings as ours)Writing, Medium67.2Creative writing, Medium: 67.8 +/- 6.6 Evidence, in standard errors: 0.7 LiveBench story generation: 67.8 (Same settings as ours) LMArena text style control, creative writing: 1,446.1 (Same settings as ours)Creative writing, Medium67.8Summarizing, Medium: 69.5 +/- 6.5 Evidence, in standard errors: -0.4 LiveBench summarize: 69.5 (Same settings as ours)Summarizing, Medium69.5Pulling out data, Medium: 87.3 +/- 1.9 Evidence, in standard errors: 1.2 LiveBench Data Analysis: 74.5 (Same settings as ours) LiveBench tablereformat: 100.0 (Same settings as ours) LMArena text style control, instruction following: 1,452.4 (Same settings as ours)Pulling out data, Medium87.3Sorting into categories, Medium: 62.4 +/- 3.4 Evidence, in standard errors: -1.4 LiveBench IF: 62.4 (Same settings as ours)Sorting into categories, Medium62.4Conversation, Medium: 52.9 +/- 1.6 Evidence, in standard errors: 1.1 SimpleQA Verified: 52.9 (Different settings) LMArena text style control, multi turn: 1,475.0 (Same settings as ours)Conversation, Medium52.9Analysis and reasoning, Hard: 78.0 +/- 2.1 Evidence, in standard errors: 0.1 GPQA diamond: 73.2 (Same settings as ours) LiveBench Reasoning: 82.7 (Same settings as ours) LMArena text style control, hard prompts: 1,480.2 (Same settings as ours)Analysis and reasoning, Hard78.0Analysis and reasoning, Hardest: 6.9 +/- 0.9 Evidence, in standard errors: -0.8 ARC-AGI-2: 0.8 (Same settings as ours) ARC-AGI: 13.0 (Same settings as ours) LMArena text style control, expert: 1,479.7 (Same settings as ours)Analysis and reasoning, Hardest6.9Math, Medium: Rating 1,444.8 Evidence, in standard errors: 0.1 LMArena text style control, math: 1,444.8 (Same settings as ours)Math, MediumRating 1,444.8Math, Hard: 90.7 +/- 1.5 Evidence, in standard errors: 1.5 LiveBench Mathematics: 90.7 (Same settings as ours)Math, Hard90.7Math, Hardest: 54.0 +/- 4.6 Evidence, in standard errors: -0.5 FrontierMath-Tiers-1-3-v2-Private: 64.6 (Different settings) OTIS Mock AIME 2024-2025: 46.7 (Same settings as ours)Math, Hardest54.0Writing code, Hard: 56.3 +/- 3.6 Evidence, in standard errors: -0.4 LiveBench Agentic Coding: 42.6 (Same settings as ours) LiveBench Coding: 70.0 (Same settings as ours) LMArena text style control, coding: 1,500.4 (Same settings as ours) LMArena webdev, overall: 1,446.2 (Same settings as ours)Writing code, Hard56.3Fixing code, Hard: 57.1 +/- 3.5 Evidence, in standard errors: 0.6 SWE-Bench verified: 77.6 (Different settings) LiveBench Agentic Coding: 42.6 (Same settings as ours)Fixing code, Hard57.1SQL and data queries, Medium: 48.2 +/- 6.9 Evidence, in standard errors: -0.1 LiveBench tablejoin: 48.2 (Same settings as ours)SQL and data queries, Medium48.2Questions about a document, Medium: Rating 1,471.7 Evidence, in standard errors: 0.5 LMArena text style control, longer query: 1,471.7 (Same settings as ours)Questions about a document, MediumRating 1,471.7
The benchmarks behind each bar
Kind of jobBenchmarkScoreSettings
Writing, MediumLiveBench Language78.1Same settings as ours
Writing, MediumLiveBench paraphrase56.2Same settings as ours
Writing, MediumLMArena text style control, instruction followingRating 1,452.4Same settings as ours
Creative writing, MediumLiveBench story generation67.8Same settings as ours
Creative writing, MediumLMArena text style control, creative writingRating 1,446.1Same settings as ours
Summarizing, MediumLiveBench summarize69.5Same settings as ours
Pulling out data, MediumLiveBench Data Analysis74.5Same settings as ours
Pulling out data, MediumLiveBench tablereformat100.0Same settings as ours
Pulling out data, MediumLMArena text style control, instruction followingRating 1,452.4Same settings as ours
Sorting into categories, MediumLiveBench IF62.4Same settings as ours
Conversation, MediumSimpleQA Verified52.9Different settings
Conversation, MediumLMArena text style control, multi turnRating 1,475.0Same settings as ours
Analysis and reasoning, HardGPQA diamond73.2Same settings as ours
Analysis and reasoning, HardLiveBench Reasoning82.7Same settings as ours
Analysis and reasoning, HardLMArena text style control, hard promptsRating 1,480.2Same settings as ours
Analysis and reasoning, HardestARC-AGI-20.8Same settings as ours
Analysis and reasoning, HardestARC-AGI13.0Same settings as ours
Analysis and reasoning, HardestLMArena text style control, expertRating 1,479.7Same settings as ours
Math, MediumLMArena text style control, mathRating 1,444.8Same settings as ours
Math, HardLiveBench Mathematics90.7Same settings as ours
Math, HardestFrontierMath-Tiers-1-3-v2-Private64.6Different settings
Math, HardestOTIS Mock AIME 2024-202546.7Same settings as ours
Writing code, HardLiveBench Agentic Coding42.6Same settings as ours
Writing code, HardLiveBench Coding70.0Same settings as ours
Writing code, HardLMArena text style control, codingRating 1,500.4Same settings as ours
Writing code, HardLMArena webdev, overallRating 1,446.2Same settings as ours
Fixing code, HardSWE-Bench verified77.6Different settings
Fixing code, HardLiveBench Agentic Coding42.6Same settings as ours
SQL and data queries, MediumLiveBench tablejoin48.2Same settings as ours
Questions about a document, MediumLMArena text style control, longer queryRating 1,471.7Same settings as ours
Price against score in each kind of job (15)

Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.

Writing, Medium
5060708090100$1$10 gpt-5.4-nano: 64.6 +/- 3.9, $0.66 deepseek-flash: 66.3 +/- 3.9, $0.72 deepseek-flash@together: 66.3 +/- 3.9, $0.72 gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24 gemini-3.8-flash: 83.9 +/- 3.1, $2.10 gpt-5.4-mini: 64.9 +/- 3.9, $2.40 deepseek-v4-pro: 67.2 +/- 3.9, $2.64 deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64 grok-4.7: 79.1 +/- 3.4, $4.40 gemini-3.5-flash: 80.1 +/- 3.4, $4.80 claude-sonnet-5: 68.1 +/- 3.9, $5.60 gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40 gpt-5.4: 73.0 +/- 3.7, $8.00 claude-sonnet-4-6: 67.8 +/- 3.9, $8.40 claude-opus-5-5: 75.3 +/- 3.7, $11.20 claude-opus-5: 77.2 +/- 3.6, $14.00 claude-opus-4-8: 73.8 +/- 3.7, $14.00 gpt-5.5: 79.6 +/- 3.5, $16.00 claude-fable-5-1: 80.5 +/- 3.4, $28.00 gpt-6-astra: 83.8 +/- 3.2, $28.00 gemini-3.8-flash Best, Balanced, Cheapdeepseek-v4-pro
Creative writing, Medium
5060708090100$1$10 gpt-5.4-nano: 67.7 +/- 6.6, $0.66 deepseek-flash: 67.3 +/- 6.6, $0.72 deepseek-flash@together: 67.3 +/- 6.6, $0.72 gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24 gemini-3.8-flash: 82.8 +/- 5.3, $2.10 gpt-5.4-mini: 66.6 +/- 6.7, $2.40 deepseek-v4-pro: 67.8 +/- 6.6, $2.64 deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64 grok-4.7: 74.9 +/- 6.1, $4.40 gemini-3.5-flash: 75.2 +/- 6.1, $4.80 claude-sonnet-5: 62.3 +/- 6.9, $5.60 gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40 gpt-5.4: 79.0 +/- 5.8, $8.00 claude-sonnet-4-6: 68.9 +/- 6.6, $8.40 claude-opus-5-5: 70.9 +/- 6.4, $11.20 claude-opus-4-8: 80.6 +/- 5.6, $14.00 claude-opus-5: 61.3 +/- 6.9, $14.00 gpt-5.5: 72.6 +/- 6.3, $16.00 claude-fable-5-1: 78.1 +/- 5.9, $28.00 gpt-6-astra: 74.0 +/- 6.2, $28.00 gemini-3.8-flash Best, Balanced, Cheapdeepseek-v4-pro
Summarizing, Medium
5060708090100$1$10 gpt-5.4-nano: 71.8 +/- 6.4, $0.73 deepseek-flash: 68.9 +/- 6.6, $0.96 deepseek-flash@together: 68.9 +/- 6.6, $0.96 gemini-3.5-flash-lite: 68.5 +/- 6.6, $1.22 gemini-3.8-flash: 87.5 +/- 4.7, $2.55 gpt-5.4-mini: 59.6 +/- 6.9, $2.70 deepseek-v4-pro: 69.5 +/- 6.5, $3.96 deepseek-v4-pro@together: 69.5 +/- 6.5, $3.96 gemini-3.5-flash: 82.6 +/- 5.4, $5.40 grok-4.7: 78.2 +/- 5.8, $6.60 claude-sonnet-5: 71.2 +/- 6.4, $6.80 gemini-3.1-pro-preview: 88.2 +/- 4.6, $7.20 gpt-5.4: 68.4 +/- 6.6, $9.00 claude-sonnet-4-6: 61.7 +/- 6.9, $10.20 claude-opus-5-5: 69.1 +/- 6.5, $13.60 claude-opus-4-8: 77.7 +/- 5.9, $17.00 claude-opus-5: 66.4 +/- 6.7, $17.00 gpt-5.5: 72.3 +/- 6.3, $18.00 gpt-6-astra: 79.6 +/- 5.7, $34.00 claude-fable-5-1: 72.2 +/- 6.3, $34.00 gpt-5.4-nano Cheapgemini-3.8-flash Best, Balanceddeepseek-v4-pro
Pulling out data, Medium
5060708090100$1$10 gpt-5.4-nano: 82.8 +/- 2.1, $0.41 deepseek-flash: 83.0 +/- 2.1, $0.48 deepseek-flash@together: 83.0 +/- 2.1, $0.48 gemini-3.5-flash-lite: 75.7 +/- 2.3, $0.74 gemini-3.8-flash: 75.1 +/- 2.5, $1.35 gpt-5.4-mini: 85.4 +/- 1.9, $1.50 deepseek-v4-pro: 87.3 +/- 1.9, $1.85 deepseek-v4-pro@together: 87.3 +/- 1.9, $1.85 gemini-3.5-flash: 80.5 +/- 2.4, $3.00 grok-4.7: 88.4 +/- 1.8, $3.08 claude-sonnet-5: 84.9 +/- 2.1, $3.60 gemini-3.1-pro-preview: 88.3 +/- 1.9, $4.00 gpt-5.4: 89.7 +/- 1.7, $5.00 claude-sonnet-4-6: 87.0 +/- 2.2, $5.40 claude-opus-5-5: 89.2 +/- 1.9, $7.20 claude-opus-5: 84.3 +/- 2.4, $9.00 claude-opus-4-8: 80.1 +/- 2.5, $9.00 gpt-5.5: 90.8 +/- 1.7, $10.00 gpt-6-astra: 91.5 +/- 1.6, $18.00 claude-fable-5-1: 87.2 +/- 2.3, $18.00 gpt-5.4-mini Cheapdeepseek-v4-pro Balancedclaude-opus-5-5 Best
Sorting into categories, Medium
5060708090100$1$10 gpt-5.4-nano: 67.2 +/- 3.3, $0.41 deepseek-flash: 63.1 +/- 3.4, $0.48 deepseek-flash@together: 63.1 +/- 3.4, $0.48 gemini-3.5-flash-lite: 67.2 +/- 3.3, $0.74 gemini-3.8-flash: 81.4 +/- 2.8, $1.35 gpt-5.4-mini: 59.8 +/- 3.5, $1.50 deepseek-v4-pro: 62.4 +/- 3.4, $1.85 deepseek-v4-pro@together: 62.4 +/- 3.4, $1.85 gemini-3.5-flash: 75.6 +/- 3.0, $3.00 grok-4.7: 75.3 +/- 3.1, $3.08 claude-sonnet-5: 63.9 +/- 3.4, $3.60 gemini-3.1-pro-preview: 79.1 +/- 2.9, $4.00 gpt-5.4: 70.2 +/- 3.2, $5.00 claude-sonnet-4-6: 63.2 +/- 3.4, $5.40 claude-opus-5-5: 65.7 +/- 3.4, $7.20 claude-opus-4-8: 72.0 +/- 3.2, $9.00 claude-opus-5: 63.8 +/- 3.4, $9.00 gpt-5.5: 70.7 +/- 3.2, $10.00 gpt-6-astra: 75.6 +/- 3.0, $18.00 claude-fable-5-1: 73.0 +/- 3.1, $18.00 gemini-3.8-flash Best, Balanced, Cheapdeepseek-v4-pro
Conversation, Medium
0255075100$1$10 gpt-4o-mini: 8.3 +/- 0.9, $0.48 gpt-5.4-nano: 11.7 +/- 1.0, $0.82 deepseek-flash: 33.6 +/- 1.5, $0.96 deepseek-flash@together: 33.6 +/- 1.5, $0.96 gpt-5-mini: 21.6 +/- 1.3, $1.20 gpt-4.1-mini: 12.7 +/- 1.1, $1.28 gemini-3.8-flash: 69.7 +/- 1.5, $2.70 gpt-5.4-mini: 29.4 +/- 1.4, $3.00 claude-haiku-4-5: 13.2 +/- 1.1, $3.60 deepseek-v4-pro: 52.9 +/- 1.6, $3.70 deepseek-v4-pro@together: 52.9 +/- 1.6, $3.70 gemini-3.5-flash: 66.2 +/- 1.5, $6.00 gpt-4.1: 31.1 +/- 1.5, $6.40 claude-sonnet-5: 33.7 +/- 1.5, $7.20 gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00 gpt-5.4: 45.1 +/- 1.6, $10.00 claude-sonnet-4-6: 32.8 +/- 1.5, $10.80 claude-opus-5-5: 72.2 +/- 1.4, $14.40 claude-opus-4-8: 53.0 +/- 1.6, $18.00 claude-opus-5: 59.9 +/- 1.6, $18.00 gpt-5.5: 63.0 +/- 1.5, $20.00 gpt-6-astra: 75.6 +/- 1.4, $36.00 claude-fable-5-1: 70.8 +/- 1.4, $36.00 gemini-3.8-flash Cheapdeepseek-v4-progemini-3.1-pro-preview Best, Balanced
Analysis and reasoning, Hard
0255075100$1$10$100 qwen3.5-9b: 79.0 +/- 2.4, $0.57 gpt-oss-120b: 75.8 +/- 2.7, $0.84 mistral-small-2603: 47.5 +/- 2.8, $0.84 gpt-4o-mini: 37.7 +/- 2.4, $0.84 gpt-5.4-nano: 66.1 +/- 2.4, $1.53 deepseek-flash: 79.0 +/- 2.0, $1.68 deepseek-flash@together: 79.0 +/- 2.0, $1.68 gpt-4.1-mini: 65.9 +/- 2.7, $2.24 gpt-5-mini: 71.7 +/- 3.2, $2.30 mistral-large-2512: 51.3 +/- 2.7, $2.35 gemini-3.5-flash-lite: 68.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02 gemini-3.8-flash: 92.3 +/- 1.3, $4.88 gpt-5.4-mini: 67.1 +/- 2.4, $5.55 deepseek-v4-pro: 78.0 +/- 2.1, $6.20 deepseek-v4-pro@together: 78.0 +/- 2.1, $6.20 claude-haiku-4-5: 60.5 +/- 2.8, $6.50 mistral-medium-3-5: 59.5 +/- 2.8, $9.75 grok-4.7: 82.7 +/- 2.7, $10.34 gemini-3.5-flash: 85.8 +/- 1.7, $11.10 gpt-4.1: 66.9 +/- 2.8, $11.20 claude-sonnet-5: 84.5 +/- 1.8, $13.00 gemini-3.1-pro-preview: 89.5 +/- 1.5, $14.80 gpt-5.4: 80.3 +/- 2.1, $18.50 claude-sonnet-4-6: 81.8 +/- 1.9, $19.50 claude-opus-5-5: 91.4 +/- 1.4, $26.00 claude-opus-5: 92.2 +/- 1.4, $32.50 claude-opus-4-8: 86.9 +/- 1.7, $32.50 gpt-5.5: 82.4 +/- 2.0, $37.00 gpt-6-astra: 94.2 +/- 1.2, $65.00 claude-fable-5-1: 91.7 +/- 1.9, $65.00 gemini-3.8-flash Cheapdeepseek-v4-proclaude-opus-5-5 Best, Balanced
Analysis and reasoning, Hardest
0255075100$1$10$100 gpt-4o-mini: 0.0 +/- 0.4, $1.35 gpt-5.4-nano: 9.9 +/- 1.1, $2.48 deepseek-flash: 7.0 +/- 1.0, $2.70 deepseek-flash@together: 7.0 +/- 1.0, $2.70 gpt-4.1-mini: 1.8 +/- 0.5, $3.60 gpt-5-mini: 15.8 +/- 0.9, $3.75 gemini-3.5-flash-lite: 9.3 +/- 1.1, $4.65 gemini-3.8-flash: 44.5 +/- 2.0, $7.88 gpt-5.4-mini: 7.1 +/- 1.0, $9.00 deepseek-v4-pro: 6.9 +/- 0.9, $9.90 deepseek-v4-pro@together: 6.9 +/- 0.9, $9.90 claude-haiku-4-5: 7.8 +/- 1.0, $10.50 mistral-medium-3-5: 4.5 +/- 0.8, $15.75 gemini-3.5-flash: 28.9 +/- 1.8, $18.00 gpt-4.1: 3.8 +/- 0.5, $18.00 gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00 gpt-5.4: 45.0 +/- 1.8, $30.00 claude-sonnet-4-6: 73.5 +/- 2.4, $31.50 claude-opus-5-5: 94.6 +/- 1.3, $42.00 claude-opus-5: 92.9 +/- 1.5, $52.50 claude-opus-4-8: 75.1 +/- 2.4, $52.50 gpt-5.5: 54.8 +/- 2.4, $60.00 claude-fable-5-1: 73.2 +/- 1.6, $105 gpt-6-astra: 66.8 +/- 1.7, $105 gemini-3.8-flash Cheapdeepseek-v4-proclaude-opus-5-5 Best, Balanced

Math, Medium: Nobody publishes a scored result for this one yet. The models below have a preference rating only.

Math, Hard
0255075100$1$10$100 gpt-4o-mini: 52.6 +/- 1.1, $0.84 mistral-small-2603: 46.8 +/- 1.2, $0.84 gpt-5.4-nano: 91.0 +/- 1.5, $1.53 deepseek-flash: 79.7 +/- 2.1, $1.68 deepseek-flash@together: 79.7 +/- 2.1, $1.68 gpt-4.1-mini: 87.3 +/- 0.7, $2.24 gpt-5-mini: 97.9 +/- 0.3, $2.30 mistral-large-2512: 50.3 +/- 1.2, $2.35 gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02 gemini-3.8-flash: 91.6 +/- 1.4, $4.88 gpt-5.4-mini: 78.5 +/- 2.1, $5.55 deepseek-v4-pro: 90.7 +/- 1.5, $6.20 deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20 claude-haiku-4-5: 86.9 +/- 0.8, $6.50 mistral-medium-3-5: 81.6 +/- 0.9, $9.75 grok-4.7: 95.7 +/- 1.0, $10.34 gemini-3.5-flash: 88.2 +/- 1.7, $11.10 gpt-4.1: 83.0 +/- 0.9, $11.20 claude-sonnet-5: 92.9 +/- 1.3, $13.00 gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80 gpt-5.4: 94.2 +/- 1.2, $18.50 claude-sonnet-4-6: 87.0 +/- 1.7, $19.50 claude-opus-5-5: 97.1 +/- 0.9, $26.00 claude-opus-5: 95.7 +/- 1.0, $32.50 claude-opus-4-8: 94.3 +/- 1.2, $32.50 gpt-5.5: 95.9 +/- 1.0, $37.00 claude-fable-5-1: 97.0 +/- 0.9, $65.00 gpt-6-astra: 96.8 +/- 0.9, $65.00 gpt-5-mini Best, Balanced, Cheapdeepseek-v4-pro
Math, Hardest
0255075100$1$10$100 qwen3.5-9b: 42.8 +/- 6.5, $0.89 gpt-oss-120b: 88.9 +/- 4.5, $1.35 mistral-small-2603: 5.3 +/- 2.6, $1.35 gpt-4o-mini: 3.8 +/- 1.4, $1.35 gpt-5.4-nano: 25.7 +/- 2.9, $2.48 deepseek-flash: 76.0 +/- 1.8, $2.70 deepseek-flash@together: 76.0 +/- 1.8, $2.70 gpt-4.1-mini: 18.6 +/- 2.2, $3.60 gpt-5-mini: 34.1 +/- 2.9, $3.75 mistral-large-2512: 1.1 +/- 0.5, $3.75 gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65 llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68 gemini-3.8-flash: 83.7 +/- 1.5, $7.88 gpt-5.4-mini: 23.6 +/- 2.7, $9.00 deepseek-v4-pro: 54.0 +/- 4.6, $9.90 deepseek-v4-pro@together: 54.0 +/- 4.6, $9.90 claude-haiku-4-5: 20.0 +/- 3.2, $10.50 mistral-medium-3-5: 16.3 +/- 3.0, $15.75 gemini-3.5-flash: 65.3 +/- 2.2, $18.00 gpt-4.1: 16.6 +/- 2.2, $18.00 claude-sonnet-5: 72.8 +/- 3.3, $21.00 gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00 gpt-5.4: 60.9 +/- 3.3, $30.00 claude-sonnet-4-6: 51.8 +/- 3.7, $31.50 claude-opus-5-5: 95.6 +/- 2.6, $42.00 claude-opus-5: 92.8 +/- 1.6, $52.50 claude-opus-4-8: 72.3 +/- 2.5, $52.50 gpt-5.5: 69.1 +/- 4.5, $60.00 gpt-6-astra: 96.8 +/- 2.6, $105 claude-fable-5-1: 95.1 +/- 2.7, $105 gpt-oss-120b Cheapdeepseek-flash Balanceddeepseek-v4-proclaude-opus-5-5 Best
Writing code, Hard
0255075100$2$5$10$20$50$100 gpt-5.4-nano: 58.8 +/- 3.6, $2.09 deepseek-flash: 53.4 +/- 3.6, $2.22 deepseek-flash@together: 53.4 +/- 3.6, $2.22 gemini-3.5-flash-lite: 60.7 +/- 3.5, $3.98 gemini-3.8-flash: 63.4 +/- 3.6, $6.56 gpt-5.4-mini: 56.7 +/- 3.6, $7.58 deepseek-v4-pro: 56.3 +/- 3.6, $7.99 deepseek-v4-pro@together: 56.3 +/- 3.6, $7.99 grok-4.7: 65.6 +/- 3.5, $13.31 gemini-3.5-flash: 63.6 +/- 3.5, $15.15 claude-sonnet-5: 70.0 +/- 3.4, $17.50 gemini-3.1-pro-preview: 43.2 +/- 2.8, $20.20 gpt-5.4: 49.0 +/- 3.5, $25.25 claude-sonnet-4-6: 61.0 +/- 3.5, $26.25 claude-opus-5-5: 79.5 +/- 3.4, $35.00 claude-opus-5: 73.3 +/- 3.3, $43.75 claude-opus-4-8: 66.2 +/- 3.4, $43.75 gpt-5.5: 48.7 +/- 3.1, $50.50 claude-fable-5-1: 75.3 +/- 3.7, $87.50 gpt-6-astra: 61.5 +/- 4.6, $87.50 deepseek-v4-proclaude-opus-5-5 Best, Balanced, Cheap
Fixing code, Hard
0255075100$1$10$100 qwen3.5-9b: 9.2 +/- 2.4, $0.68 gpt-oss-120b: 25.6 +/- 1.9, $1.11 gpt-4o-mini: 3.6 +/- 1.2, $1.11 codestral-2508: 11.1 +/- 2.1, $1.82 gpt-5.4-nano: 46.8 +/- 5.9, $2.09 deepseek-flash: 37.6 +/- 5.7, $2.22 deepseek-flash@together: 37.6 +/- 5.7, $2.22 gpt-4.1-mini: 32.4 +/- 3.1, $2.96 gpt-5-mini: 47.0 +/- 1.7, $3.20 gemini-3.5-flash-lite: 45.3 +/- 5.9, $3.98 gemini-3.8-flash: 54.2 +/- 5.9, $6.56 gpt-5.4-mini: 41.7 +/- 5.8, $7.58 deepseek-v4-pro: 57.1 +/- 3.5, $7.99 deepseek-v4-pro@together: 57.1 +/- 3.5, $7.99 claude-haiku-4-5: 13.9 +/- 2.7, $8.75 grok-4.7: 54.0 +/- 5.9, $13.31 gpt-4.1: 49.8 +/- 1.9, $14.80 gemini-3.5-flash: 64.2 +/- 3.1, $15.15 claude-sonnet-5: 59.4 +/- 5.8, $17.50 gemini-3.1-pro-preview: 49.8 +/- 4.0, $20.20 gpt-5.4: 69.7 +/- 2.3, $25.25 claude-sonnet-4-6: 59.9 +/- 2.2, $26.25 claude-opus-5-5: 71.7 +/- 5.3, $35.00 claude-opus-5: 65.2 +/- 5.6, $43.75 claude-opus-4-8: 50.5 +/- 5.9, $43.75 gpt-5.5: 59.0 +/- 3.6, $50.50 claude-fable-5-1: 66.1 +/- 5.6, $87.50 gpt-6-astra: 57.3 +/- 5.8, $87.50 deepseek-v4-proclaude-sonnet-5 Cheapclaude-opus-5-5 Best, Balanced
SQL and data queries, Medium
0255075100$1$10 gpt-5.4-nano: 50.6 +/- 6.9, $0.41 deepseek-flash: 46.1 +/- 6.9, $0.48 deepseek-flash@together: 46.1 +/- 6.9, $0.48 gemini-3.5-flash-lite: 40.6 +/- 6.8, $0.74 gemini-3.8-flash: 45.3 +/- 6.9, $1.35 gpt-5.4-mini: 49.4 +/- 6.9, $1.50 deepseek-v4-pro: 48.2 +/- 6.9, $1.85 deepseek-v4-pro@together: 48.2 +/- 6.9, $1.85 gemini-3.5-flash: 50.5 +/- 6.9, $3.00 grok-4.7: 44.9 +/- 6.9, $3.08 claude-sonnet-5: 42.4 +/- 6.9, $3.60 gemini-3.1-pro-preview: 52.4 +/- 6.9, $4.00 gpt-5.4: 51.8 +/- 6.9, $5.00 claude-sonnet-4-6: 45.1 +/- 6.9, $5.40 claude-opus-5-5: 52.3 +/- 6.9, $7.20 claude-opus-5: 52.0 +/- 6.9, $9.00 claude-opus-4-8: 51.1 +/- 6.9, $9.00 gpt-5.5: 55.9 +/- 6.9, $10.00 gpt-6-astra: 58.4 +/- 6.8, $18.00 claude-fable-5-1: 56.2 +/- 6.9, $18.00 gpt-5.4-nano Best, Balanced, Cheapdeepseek-v4-pro

Questions about a document, Medium: Nobody publishes a scored result for this one yet. The models below have a preference rating only.

Sources

What these numbers are

  • These numbers come from the sources named on each row, not from Zabio's own tests. Zabio's tests are shown separately and never combined with them into one figure.
  • Sources test models at their own settings. A settings mismatch against how Zabio calls the model is marked on the row and lowers its weight.
  • Scores are compared only inside one benchmark. Task scores combine benchmarks by evidence in standard errors, so a task score is a summary, not a benchmark result.
  • A model missing from every source is listed as such rather than left out.

Results from our own tests have their own section.

How we tested

Called through Zabio's own API at each thinking setting, question set pilot-2026-09-25, methodology 1.0. Without a thinking setting in the request, Zabio uses thinking off.

Model id to use through Zabio: deepseek-v4-pro

Changes

  • 25 September 2026First results: math and SQL at three thinking settings.

Read this first

  • AIME problems are public and may appear in model training data, so math results at the frontier level may be inflated. Differences between settings are the reliable part.
  • SQL questions were kept only when an independent model outside the test answered them correctly, which limits how hard the hardest questions are.
  • These results compare DeepSeek models and settings with each other. Other providers' models have not been tested in this run.
  • Results are not ranked when their ranges overlap. Top means nothing tested is probably better on that job. Near the top means not clearly worse than the top.

Zabio sells access to these models and earns a fee on the traffic it carries. That is why the method is public, every number has its range and date, and corrections are published as changes. No vendor pays for tests, results or placement.

DeepSeek is a trademark of its owner. Zabio uses DeepSeek models through the DeepSeek API and is not affiliated with DeepSeek.

Use the data

The results are free to reuse under CC BY 4.0. Please credit Zabio with the citation below.

License
CC BY 4.0
Suggested citation
Zabio, Model benchmarks, zabio.ai/benchmarks, 2026-09-25
Methodology
1.0
Tested
25 September 2026
Download benchmarks.json