Open beta Zabio is in open beta: anyone can sign up and use it in production. Models, prices and features can still change while we learn from the first customers. Tell us what breaks at hello@zabio.ai.
Menu
Start free

Zabio benchmarks

AI model benchmarks by task

Tested on 25 September 2026: 2 models, Math, SQL and data queries, at 3 thinking settings, on 380 questions. Every number shows its 90% range and how many questions it rests on.

Tested
25 September 2026
Models
2
Questions
380
Question set
pilot-2026-09-25
Methodology
1.0

The best choice for each job

For each task and difficulty, three choices among the options tested so far: best is the cheapest option that nothing else is probably better than, cheap is the cheapest option not clearly worse than the top, and balanced sits between the two.

Math

DifficultyBestBalancedCheap
Easydeepseek-flashThinking low$0.27deepseek-flashThinking low$0.27deepseek-flashThinking low$0.27
Mediumdeepseek-flashThinking low$0.62deepseek-flashThinking low$0.62deepseek-flashThinking off$0.59
Harddeepseek-flashThinking low$2.26deepseek-flashThinking low$2.26deepseek-flashThinking off$1.66
Hardestdeepseek-flashThinking high$9.18deepseek-flashThinking high$9.18deepseek-flashThinking low$6.38

Models compared: deepseek-flash, deepseek-v4-pro

SQL and data queries

DifficultyBestBalancedCheap
Easydeepseek-flashThinking off$0.24deepseek-flashThinking off$0.24deepseek-flashThinking off$0.24
Mediumdeepseek-flashThinking high$1.87deepseek-flashThinking high$1.87deepseek-flashThinking off$0.45
Harddeepseek-flashThinking low$2.10deepseek-flashThinking low$2.10deepseek-flashThinking low$2.10
Hardestdeepseek-flashThinking low$4.29deepseek-flashThinking low$4.29deepseek-flashThinking low$4.29

Models compared: deepseek-flash, deepseek-v4-pro

Price and right answers

Each dot is a model at one thinking setting: cost per 1,000 tasks across, right answers up. Up and to the left is better.

Math

Easy
5060708090100$0.2$0.5$1$2deepseek-flash, Thinking high: 100.0%, $0.32highdeepseek-flash, Thinking low: 100.0%, $0.27lowdeepseek-flash, Thinking off: 100.0%, $0.36offdeepseek-v4-pro, Thinking low: 100.0%, $0.81lowdeepseek-v4-pro, Thinking high: 100.0%, $1.28highdeepseek-v4-pro, Thinking off: 96.0%, $1.04off
Medium
5060708090100$0.5$1$2$5deepseek-flash, Thinking high: 96.4%, $1.04highdeepseek-flash, Thinking low: 98.2%, $0.62lowdeepseek-flash, Thinking off: 94.5%, $0.59offdeepseek-v4-pro, Thinking low: 98.2%, $1.57lowdeepseek-v4-pro, Thinking high: 98.2%, $3.44highdeepseek-v4-pro, Thinking off: 94.5%, $1.60off
Hard
5060708090100$1$2$5$10$20deepseek-flash, Thinking off: 96.3%, $1.66offdeepseek-flash, Thinking low: 100.0%, $2.26lowdeepseek-flash, Thinking high: 100.0%, $3.00highdeepseek-v4-pro, Thinking off: 86.4%, $3.94offdeepseek-v4-pro, Thinking low: 98.1%, $5.98lowdeepseek-v4-pro, Thinking high: 98.2%, $15.63high
Hardest
0255075100$2$5$10$20$50deepseek-flash, Thinking low: 96.4%, $6.38lowdeepseek-flash, Thinking high: 98.2%, $9.18highdeepseek-flash, Thinking off: 73.3%, $3.16offdeepseek-v4-pro, Thinking off: 44.2%, $6.52offdeepseek-v4-pro, Thinking low: 89.1%, $25.10lowdeepseek-v4-pro, Thinking high: 98.2%, $48.68high

SQL and data queries

Easy
5060708090100$0.2$0.5$1$2$5deepseek-flash, Thinking off: 100.0%, $0.24offdeepseek-flash, Thinking low: 100.0%, $0.35lowdeepseek-flash, Thinking high: 100.0%, $0.42highdeepseek-v4-pro, Thinking off: 100.0%, $1.03offdeepseek-v4-pro, Thinking low: 100.0%, $1.76lowdeepseek-v4-pro, Thinking high: 100.0%, $2.35high
Medium
5060708090100$1$10deepseek-flash, Thinking off: 96.4%, $0.45offdeepseek-flash, Thinking high: 98.2%, $1.87highdeepseek-flash, Thinking low: 96.4%, $1.26lowdeepseek-v4-pro, Thinking off: 90.9%, $1.76offdeepseek-v4-pro, Thinking high: 98.2%, $12.51highdeepseek-v4-pro, Thinking low: 98.2%, $8.92low
Hard
5060708090100$1$10deepseek-flash, Thinking off: 93.9%, $0.58offdeepseek-flash, Thinking high: 100.0%, $3.48highdeepseek-flash, Thinking low: 100.0%, $2.10lowdeepseek-v4-pro, Thinking off: 84.8%, $2.24offdeepseek-v4-pro, Thinking low: 97.6%, $13.45lowdeepseek-v4-pro, Thinking high: 100.0%, $22.80high
Hardest
5060708090100$1$10deepseek-flash, Thinking off: 88.5%, $0.75offdeepseek-flash, Thinking low: 100.0%, $4.29lowdeepseek-flash, Thinking high: 100.0%, $7.42highdeepseek-v4-pro, Thinking off: 84.8%, $3.06offdeepseek-v4-pro, Thinking low: 94.5%, $26.46lowdeepseek-v4-pro, Thinking high: 98.2%, $41.95high

Cost per 1,000 tasks, list price, log scale. Right answers, %.

How every model compares on public benchmarks

Weekly results from Epoch AI, LMArena and LiveBench, shown as price against score for each kind of job. Zabio uses these same results to pick the model for each job.

Aggregated from public resultsRead on 1 October 2026

Writing

Price against score, Medium
5060708090100$1$10 gpt-5.4-nano: 64.6 +/- 3.9, $0.66 deepseek-flash: 66.3 +/- 3.9, $0.72 deepseek-flash@together: 66.3 +/- 3.9, $0.72 gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24 gemini-3.8-flash: 83.9 +/- 3.1, $2.10 gpt-5.4-mini: 64.9 +/- 3.9, $2.40 deepseek-v4-pro: 67.2 +/- 3.9, $2.64 deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64 grok-4.7: 79.1 +/- 3.4, $4.40 gemini-3.5-flash: 80.1 +/- 3.4, $4.80 claude-sonnet-5: 68.1 +/- 3.9, $5.60 gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40 gpt-5.4: 73.0 +/- 3.7, $8.00 claude-sonnet-4-6: 67.8 +/- 3.9, $8.40 claude-opus-5-5: 75.3 +/- 3.7, $11.20 claude-opus-5: 77.2 +/- 3.6, $14.00 claude-opus-4-8: 73.8 +/- 3.7, $14.00 gpt-5.5: 79.6 +/- 3.5, $16.00 claude-fable-5-1: 80.5 +/- 3.4, $28.00 gpt-6-astra: 83.8 +/- 3.2, $28.00 gemini-3.8-flash Best, Balanced, Cheap
Ranking, Medium
1. gemini-3.8-flash: 83.9 +/- 3.11gemini-3.8-flash83.92. claude-fable-5-1: 80.5 +/- 3.42claude-fable-5-180.53. gpt-6-astra: 83.8 +/- 3.23gpt-6-astra83.84. gemini-3.1-pro-preview: 82.6 +/- 3.24gemini-3.1-pro-preview82.65. claude-opus-5: 77.2 +/- 3.65claude-opus-577.26. gpt-5.5: 79.6 +/- 3.56gpt-5.579.67. claude-opus-5-5: 75.3 +/- 3.77claude-opus-5-575.38. gemini-3.5-flash: 80.1 +/- 3.48gemini-3.5-flash80.19. claude-opus-4-8: 73.8 +/- 3.79claude-opus-4-873.810. grok-4.7: 79.1 +/- 3.410grok-4.779.111. gpt-5.4: 73.0 +/- 3.711gpt-5.473.012. claude-sonnet-4-6: 67.8 +/- 3.912claude-sonnet-4-667.813. claude-sonnet-5: 68.1 +/- 3.913claude-sonnet-568.114. deepseek-v4-pro: 67.2 +/- 3.914deepseek-v4-pro67.215. deepseek-v4-pro@together: 67.2 +/- 3.915deepseek-v4-pro@together67.216. gemini-3.5-flash-lite: 68.4 +/- 3.816gemini-3.5-flash-lite68.417. gpt-5.4-mini: 64.9 +/- 3.917gpt-5.4-mini64.918. deepseek-flash: 66.3 +/- 3.918deepseek-flash66.319. deepseek-flash@together: 66.3 +/- 3.919deepseek-flash@together66.324. gpt-5.4-nano: 64.6 +/- 3.924gpt-5.4-nano64.6Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Rating only, not on the chart (10)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-small-2603Rating 1,337.7$0.36
  • gpt-oss-120bRating 1,323.4$0.36
  • gpt-4o-miniRating 1,293.1$0.36
  • gpt-4.1-miniRating 1,372.9$0.96
  • mistral-large-2512Rating 1,400.5$1.00
  • gpt-5-miniRating 1,373.3$1.00
  • llama-3.3-70b-instruct-turboRating 1,292.6$1.25
  • claude-haiku-4-5Rating 1,415.0$2.80
  • mistral-medium-3-5Rating 1,421.2$4.20
  • gpt-4.1Rating 1,403.3$4.80

Creative writing

Price against score, Medium
5060708090100$1$10 gpt-5.4-nano: 67.7 +/- 6.6, $0.66 deepseek-flash: 67.3 +/- 6.6, $0.72 deepseek-flash@together: 67.3 +/- 6.6, $0.72 gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24 gemini-3.8-flash: 82.8 +/- 5.3, $2.10 gpt-5.4-mini: 66.6 +/- 6.7, $2.40 deepseek-v4-pro: 67.8 +/- 6.6, $2.64 deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64 grok-4.7: 74.9 +/- 6.1, $4.40 gemini-3.5-flash: 75.2 +/- 6.1, $4.80 claude-sonnet-5: 62.3 +/- 6.9, $5.60 gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40 gpt-5.4: 79.0 +/- 5.8, $8.00 claude-sonnet-4-6: 68.9 +/- 6.6, $8.40 claude-opus-5-5: 70.9 +/- 6.4, $11.20 claude-opus-4-8: 80.6 +/- 5.6, $14.00 claude-opus-5: 61.3 +/- 6.9, $14.00 gpt-5.5: 72.6 +/- 6.3, $16.00 claude-fable-5-1: 78.1 +/- 5.9, $28.00 gpt-6-astra: 74.0 +/- 6.2, $28.00 gemini-3.8-flash Best, Balanced, Cheap
Ranking, Medium
1. gemini-3.8-flash: 82.8 +/- 5.31gemini-3.8-flash82.82. gemini-3.1-pro-preview: 77.3 +/- 5.92gemini-3.1-pro-preview77.33. claude-fable-5-1: 78.1 +/- 5.93claude-fable-5-178.14. claude-opus-4-8: 80.6 +/- 5.64claude-opus-4-880.65. claude-opus-5-5: 70.9 +/- 6.45claude-opus-5-570.96. gemini-3.5-flash: 75.2 +/- 6.16gemini-3.5-flash75.27. gpt-6-astra: 74.0 +/- 6.27gpt-6-astra74.08. gpt-5.5: 72.6 +/- 6.38gpt-5.572.69. gpt-5.4: 79.0 +/- 5.89gpt-5.479.010. claude-opus-5: 61.3 +/- 6.910claude-opus-561.311. claude-sonnet-4-6: 68.9 +/- 6.611claude-sonnet-4-668.912. grok-4.7: 74.9 +/- 6.112grok-4.774.913. deepseek-v4-pro: 67.8 +/- 6.613deepseek-v4-pro67.814. deepseek-v4-pro@together: 67.8 +/- 6.614deepseek-v4-pro@together67.815. gemini-3.5-flash-lite: 67.5 +/- 6.615gemini-3.5-flash-lite67.516. claude-sonnet-5: 62.3 +/- 6.916claude-sonnet-562.317. deepseek-flash: 67.3 +/- 6.617deepseek-flash67.318. deepseek-flash@together: 67.3 +/- 6.618deepseek-flash@together67.319. gpt-5.4-mini: 66.6 +/- 6.719gpt-5.4-mini66.623. gpt-5.4-nano: 67.7 +/- 6.623gpt-5.4-nano67.7Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Rating only, not on the chart (10)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-small-2603Rating 1,323.2$0.36
  • gpt-4o-miniRating 1,293.9$0.36
  • gpt-oss-120bRating 1,277.8$0.36
  • gpt-4.1-miniRating 1,349.1$0.96
  • mistral-large-2512Rating 1,373.4$1.00
  • gpt-5-miniRating 1,323.7$1.00
  • llama-3.3-70b-instruct-turboRating 1,285.3$1.25
  • claude-haiku-4-5Rating 1,387.4$2.80
  • mistral-medium-3-5Rating 1,400.3$4.20
  • gpt-4.1Rating 1,402.4$4.80

Summarizing

Price against score, Medium
5060708090100$1$10 gpt-5.4-nano: 71.8 +/- 6.4, $0.73 deepseek-flash: 68.9 +/- 6.6, $0.96 deepseek-flash@together: 68.9 +/- 6.6, $0.96 gemini-3.5-flash-lite: 68.5 +/- 6.6, $1.22 gemini-3.8-flash: 87.5 +/- 4.7, $2.55 gpt-5.4-mini: 59.6 +/- 6.9, $2.70 deepseek-v4-pro: 69.5 +/- 6.5, $3.96 deepseek-v4-pro@together: 69.5 +/- 6.5, $3.96 gemini-3.5-flash: 82.6 +/- 5.4, $5.40 grok-4.7: 78.2 +/- 5.8, $6.60 claude-sonnet-5: 71.2 +/- 6.4, $6.80 gemini-3.1-pro-preview: 88.2 +/- 4.6, $7.20 gpt-5.4: 68.4 +/- 6.6, $9.00 claude-sonnet-4-6: 61.7 +/- 6.9, $10.20 claude-opus-5-5: 69.1 +/- 6.5, $13.60 claude-opus-4-8: 77.7 +/- 5.9, $17.00 claude-opus-5: 66.4 +/- 6.7, $17.00 gpt-5.5: 72.3 +/- 6.3, $18.00 gpt-6-astra: 79.6 +/- 5.7, $34.00 claude-fable-5-1: 72.2 +/- 6.3, $34.00 gpt-5.4-nano Cheapgemini-3.8-flash Best, Balanced
Ranking, Medium
1. gemini-3.1-pro-preview: 88.2 +/- 4.61gemini-3.1-pro-preview88.22. gemini-3.8-flash: 87.5 +/- 4.72gemini-3.8-flash87.53. gemini-3.5-flash: 82.6 +/- 5.43gemini-3.5-flash82.64. gpt-6-astra: 79.6 +/- 5.74gpt-6-astra79.65. grok-4.7: 78.2 +/- 5.85grok-4.778.26. claude-opus-4-8: 77.7 +/- 5.96claude-opus-4-877.77. gpt-5.5: 72.3 +/- 6.37gpt-5.572.38. claude-fable-5-1: 72.2 +/- 6.38claude-fable-5-172.29. gpt-5.4-nano: 71.8 +/- 6.49gpt-5.4-nano71.810. claude-sonnet-5: 71.2 +/- 6.410claude-sonnet-571.211. deepseek-v4-pro: 69.5 +/- 6.511deepseek-v4-pro69.512. deepseek-v4-pro@together: 69.5 +/- 6.512deepseek-v4-pro@together69.513. claude-opus-5-5: 69.1 +/- 6.513claude-opus-5-569.114. deepseek-flash: 68.9 +/- 6.614deepseek-flash68.915. deepseek-flash@together: 68.9 +/- 6.615deepseek-flash@together68.916. gemini-3.5-flash-lite: 68.5 +/- 6.616gemini-3.5-flash-lite68.517. gpt-5.4: 68.4 +/- 6.617gpt-5.468.418. claude-opus-5: 66.4 +/- 6.718claude-opus-566.419. claude-sonnet-4-6: 61.7 +/- 6.919claude-sonnet-4-661.720. gpt-5.4-mini: 59.6 +/- 6.920gpt-5.4-mini59.6Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model

Pulling out data

Price against score, Medium
5060708090100$1$10 gpt-5.4-nano: 82.8 +/- 2.1, $0.41 deepseek-flash: 83.0 +/- 2.1, $0.48 deepseek-flash@together: 83.0 +/- 2.1, $0.48 gemini-3.5-flash-lite: 75.7 +/- 2.3, $0.74 gemini-3.8-flash: 75.1 +/- 2.5, $1.35 gpt-5.4-mini: 85.4 +/- 1.9, $1.50 deepseek-v4-pro: 87.3 +/- 1.9, $1.85 deepseek-v4-pro@together: 87.3 +/- 1.9, $1.85 gemini-3.5-flash: 80.5 +/- 2.4, $3.00 grok-4.7: 88.4 +/- 1.8, $3.08 claude-sonnet-5: 84.9 +/- 2.1, $3.60 gemini-3.1-pro-preview: 88.3 +/- 1.9, $4.00 gpt-5.4: 89.7 +/- 1.7, $5.00 claude-sonnet-4-6: 87.0 +/- 2.2, $5.40 claude-opus-5-5: 89.2 +/- 1.9, $7.20 claude-opus-5: 84.3 +/- 2.4, $9.00 claude-opus-4-8: 80.1 +/- 2.5, $9.00 gpt-5.5: 90.8 +/- 1.7, $10.00 gpt-6-astra: 91.5 +/- 1.6, $18.00 claude-fable-5-1: 87.2 +/- 2.3, $18.00 gpt-5.4-mini Cheapdeepseek-v4-pro Balancedclaude-opus-5-5 Best
Ranking, Medium
1. gpt-6-astra: 91.5 +/- 1.61gpt-6-astra91.52. gpt-5.5: 90.8 +/- 1.72gpt-5.590.83. claude-opus-5-5: 89.2 +/- 1.93claude-opus-5-589.24. gemini-3.1-pro-preview: 88.3 +/- 1.94gemini-3.1-pro-preview88.35. claude-fable-5-1: 87.2 +/- 2.35claude-fable-5-187.26. gpt-5.4: 89.7 +/- 1.76gpt-5.489.77. claude-sonnet-4-6: 87.0 +/- 2.27claude-sonnet-4-687.08. claude-opus-5: 84.3 +/- 2.48claude-opus-584.39. grok-4.7: 88.4 +/- 1.89grok-4.788.410. deepseek-v4-pro: 87.3 +/- 1.910deepseek-v4-pro87.311. deepseek-v4-pro@together: 87.3 +/- 1.911deepseek-v4-pro@together87.312. claude-sonnet-5: 84.9 +/- 2.112claude-sonnet-584.913. gpt-5.4-mini: 85.4 +/- 1.913gpt-5.4-mini85.414. claude-opus-4-8: 80.1 +/- 2.514claude-opus-4-880.115. gemini-3.5-flash: 80.5 +/- 2.415gemini-3.5-flash80.516. gemini-3.8-flash: 75.1 +/- 2.516gemini-3.8-flash75.117. deepseek-flash: 83.0 +/- 2.117deepseek-flash83.018. deepseek-flash@together: 83.0 +/- 2.118deepseek-flash@together83.019. gemini-3.5-flash-lite: 75.7 +/- 2.319gemini-3.5-flash-lite75.721. gpt-5.4-nano: 82.8 +/- 2.121gpt-5.4-nano82.8Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Rating only, not on the chart (10)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-small-2603Rating 1,337.7$0.24
  • gpt-oss-120bRating 1,323.4$0.24
  • gpt-4o-miniRating 1,293.1$0.24
  • gpt-5-miniRating 1,373.3$0.60
  • gpt-4.1-miniRating 1,372.9$0.64
  • mistral-large-2512Rating 1,400.5$0.70
  • llama-3.3-70b-instruct-turboRating 1,292.6$1.04
  • claude-haiku-4-5Rating 1,415.0$1.80
  • mistral-medium-3-5Rating 1,421.2$2.70
  • gpt-4.1Rating 1,403.3$3.20

Sorting into categories

Price against score, Medium
5060708090100$1$10 gpt-5.4-nano: 67.2 +/- 3.3, $0.41 deepseek-flash: 63.1 +/- 3.4, $0.48 deepseek-flash@together: 63.1 +/- 3.4, $0.48 gemini-3.5-flash-lite: 67.2 +/- 3.3, $0.74 gemini-3.8-flash: 81.4 +/- 2.8, $1.35 gpt-5.4-mini: 59.8 +/- 3.5, $1.50 deepseek-v4-pro: 62.4 +/- 3.4, $1.85 deepseek-v4-pro@together: 62.4 +/- 3.4, $1.85 gemini-3.5-flash: 75.6 +/- 3.0, $3.00 grok-4.7: 75.3 +/- 3.1, $3.08 claude-sonnet-5: 63.9 +/- 3.4, $3.60 gemini-3.1-pro-preview: 79.1 +/- 2.9, $4.00 gpt-5.4: 70.2 +/- 3.2, $5.00 claude-sonnet-4-6: 63.2 +/- 3.4, $5.40 claude-opus-5-5: 65.7 +/- 3.4, $7.20 claude-opus-4-8: 72.0 +/- 3.2, $9.00 claude-opus-5: 63.8 +/- 3.4, $9.00 gpt-5.5: 70.7 +/- 3.2, $10.00 gpt-6-astra: 75.6 +/- 3.0, $18.00 claude-fable-5-1: 73.0 +/- 3.1, $18.00 gemini-3.8-flash Best, Balanced, Cheap
Ranking, Medium
1. gemini-3.8-flash: 81.4 +/- 2.81gemini-3.8-flash81.42. gemini-3.1-pro-preview: 79.1 +/- 2.92gemini-3.1-pro-preview79.13. gemini-3.5-flash: 75.6 +/- 3.03gemini-3.5-flash75.64. gpt-6-astra: 75.6 +/- 3.04gpt-6-astra75.65. grok-4.7: 75.3 +/- 3.15grok-4.775.36. claude-fable-5-1: 73.0 +/- 3.16claude-fable-5-173.07. claude-opus-4-8: 72.0 +/- 3.27claude-opus-4-872.08. gpt-5.5: 70.7 +/- 3.28gpt-5.570.79. gpt-5.4: 70.2 +/- 3.29gpt-5.470.210. gemini-3.5-flash-lite: 67.2 +/- 3.310gemini-3.5-flash-lite67.211. gpt-5.4-nano: 67.2 +/- 3.311gpt-5.4-nano67.212. claude-opus-5-5: 65.7 +/- 3.412claude-opus-5-565.713. claude-sonnet-5: 63.9 +/- 3.413claude-sonnet-563.914. claude-opus-5: 63.8 +/- 3.414claude-opus-563.815. claude-sonnet-4-6: 63.2 +/- 3.415claude-sonnet-4-663.216. deepseek-flash: 63.1 +/- 3.416deepseek-flash63.117. deepseek-flash@together: 63.1 +/- 3.417deepseek-flash@together63.118. deepseek-v4-pro: 62.4 +/- 3.418deepseek-v4-pro62.419. deepseek-v4-pro@together: 62.4 +/- 3.419deepseek-v4-pro@together62.420. gpt-5.4-mini: 59.8 +/- 3.520gpt-5.4-mini59.8Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model

Conversation

Price against score, Medium
0255075100$1$10 gpt-4o-mini: 8.3 +/- 0.9, $0.48 gpt-5.4-nano: 11.7 +/- 1.0, $0.82 deepseek-flash: 33.6 +/- 1.5, $0.96 deepseek-flash@together: 33.6 +/- 1.5, $0.96 gpt-5-mini: 21.6 +/- 1.3, $1.20 gpt-4.1-mini: 12.7 +/- 1.1, $1.28 gemini-3.8-flash: 69.7 +/- 1.5, $2.70 gpt-5.4-mini: 29.4 +/- 1.4, $3.00 claude-haiku-4-5: 13.2 +/- 1.1, $3.60 deepseek-v4-pro: 52.9 +/- 1.6, $3.70 deepseek-v4-pro@together: 52.9 +/- 1.6, $3.70 gemini-3.5-flash: 66.2 +/- 1.5, $6.00 gpt-4.1: 31.1 +/- 1.5, $6.40 claude-sonnet-5: 33.7 +/- 1.5, $7.20 gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00 gpt-5.4: 45.1 +/- 1.6, $10.00 claude-sonnet-4-6: 32.8 +/- 1.5, $10.80 claude-opus-5-5: 72.2 +/- 1.4, $14.40 claude-opus-4-8: 53.0 +/- 1.6, $18.00 claude-opus-5: 59.9 +/- 1.6, $18.00 gpt-5.5: 63.0 +/- 1.5, $20.00 gpt-6-astra: 75.6 +/- 1.4, $36.00 claude-fable-5-1: 70.8 +/- 1.4, $36.00 gemini-3.8-flash Cheapgemini-3.1-pro-preview Best, Balanced
Ranking, Medium
1. gemini-3.1-pro-preview: 73.5 +/- 1.41gemini-3.1-pro-preview73.52. gpt-6-astra: 75.6 +/- 1.42gpt-6-astra75.63. gemini-3.8-flash: 69.7 +/- 1.53gemini-3.8-flash69.74. claude-opus-5-5: 72.2 +/- 1.44claude-opus-5-572.25. claude-fable-5-1: 70.8 +/- 1.45claude-fable-5-170.86. gemini-3.5-flash: 66.2 +/- 1.56gemini-3.5-flash66.27. gpt-5.5: 63.0 +/- 1.57gpt-5.563.08. claude-opus-4-8: 53.0 +/- 1.68claude-opus-4-853.09. claude-opus-5: 59.9 +/- 1.69claude-opus-559.910. deepseek-v4-pro: 52.9 +/- 1.610deepseek-v4-pro52.911. deepseek-v4-pro@together: 52.9 +/- 1.611deepseek-v4-pro@together52.912. gpt-5.4: 45.1 +/- 1.612gpt-5.445.113. claude-sonnet-4-6: 32.8 +/- 1.513claude-sonnet-4-632.814. claude-sonnet-5: 33.7 +/- 1.514claude-sonnet-533.716. deepseek-flash: 33.6 +/- 1.516deepseek-flash33.617. deepseek-flash@together: 33.6 +/- 1.517deepseek-flash@together33.618. gpt-5.4-mini: 29.4 +/- 1.418gpt-5.4-mini29.420. gpt-4.1: 31.1 +/- 1.520gpt-4.131.123. claude-haiku-4-5: 13.2 +/- 1.123claude-haiku-4-513.225. gpt-5.4-nano: 11.7 +/- 1.025gpt-5.4-nano11.726. gpt-5-mini: 21.6 +/- 1.326gpt-5-mini21.628. gpt-4.1-mini: 12.7 +/- 1.128gpt-4.1-mini12.730. gpt-4o-mini: 8.3 +/- 0.930gpt-4o-mini8.3Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Rating only, not on the chart (7)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-small-2603Rating 1,364.9$0.48
  • gpt-oss-120bRating 1,326.8$0.48
  • mistral-large-2512Rating 1,419.6$1.40
  • gemini-3.5-flash-liteRating 1,464.6$1.48
  • llama-3.3-70b-instruct-turboRating 1,316.5$2.08
  • mistral-medium-3-5Rating 1,432.9$5.40
  • grok-4.7Rating 1,424.7$6.16

Analysis and reasoning

Price against score, Medium
0255075100$1$10 gpt-oss-120b: 22.1 +/- 2.9, $0.36 gpt-4o-mini: 10.7 +/- 2.2, $0.36 deepseek-flash: 61.1 +/- 3.5, $0.72 deepseek-flash@together: 61.1 +/- 3.5, $0.72 mistral-large-2512: 22.5 +/- 3.0, $1.00 llama-3.3-70b-instruct-turbo: 19.9 +/- 2.8, $1.25 gemini-3.5-flash: 76.7 +/- 3.0, $4.80 gpt-4.1: 27.0 +/- 3.1, $4.80 claude-sonnet-5: 60.6 +/- 3.5, $5.60 gemini-3.1-pro-preview: 79.6 +/- 2.9, $6.40 claude-opus-5: 80.6 +/- 2.8, $14.00 claude-opus-4-8: 64.8 +/- 3.4, $14.00 gpt-5.5: 69.0 +/- 3.3, $16.00 gemini-3.5-flash Best, Balanced, Cheap
Ranking, Medium
1. claude-opus-5: 80.6 +/- 2.81claude-opus-580.62. gemini-3.1-pro-preview: 79.6 +/- 2.92gemini-3.1-pro-preview79.63. gemini-3.5-flash: 76.7 +/- 3.03gemini-3.5-flash76.74. gpt-5.5: 69.0 +/- 3.34gpt-5.569.05. claude-opus-4-8: 64.8 +/- 3.45claude-opus-4-864.86. deepseek-flash: 61.1 +/- 3.56deepseek-flash61.17. deepseek-flash@together: 61.1 +/- 3.57deepseek-flash@together61.18. claude-sonnet-5: 60.6 +/- 3.58claude-sonnet-560.69. gpt-4.1: 27.0 +/- 3.19gpt-4.127.010. mistral-large-2512: 22.5 +/- 3.010mistral-large-251222.511. gpt-oss-120b: 22.1 +/- 2.911gpt-oss-120b22.112. llama-3.3-70b-instruct-turbo: 19.9 +/- 2.812llama-3.3-70b-instruct-turbo19.913. gpt-4o-mini: 10.7 +/- 2.213gpt-4o-mini10.7Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Price against score, Hard
0255075100$1$10$100 qwen3.5-9b: 79.0 +/- 2.4, $0.57 gpt-oss-120b: 75.8 +/- 2.7, $0.84 mistral-small-2603: 47.5 +/- 2.8, $0.84 gpt-4o-mini: 37.7 +/- 2.4, $0.84 gpt-5.4-nano: 66.1 +/- 2.4, $1.53 deepseek-flash: 79.0 +/- 2.0, $1.68 deepseek-flash@together: 79.0 +/- 2.0, $1.68 gpt-4.1-mini: 65.9 +/- 2.7, $2.24 gpt-5-mini: 71.7 +/- 3.2, $2.30 mistral-large-2512: 51.3 +/- 2.7, $2.35 gemini-3.5-flash-lite: 68.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02 gemini-3.8-flash: 92.3 +/- 1.3, $4.88 gpt-5.4-mini: 67.1 +/- 2.4, $5.55 deepseek-v4-pro: 78.0 +/- 2.1, $6.20 deepseek-v4-pro@together: 78.0 +/- 2.1, $6.20 claude-haiku-4-5: 60.5 +/- 2.8, $6.50 mistral-medium-3-5: 59.5 +/- 2.8, $9.75 grok-4.7: 82.7 +/- 2.7, $10.34 gemini-3.5-flash: 85.8 +/- 1.7, $11.10 gpt-4.1: 66.9 +/- 2.8, $11.20 claude-sonnet-5: 84.5 +/- 1.8, $13.00 gemini-3.1-pro-preview: 89.5 +/- 1.5, $14.80 gpt-5.4: 80.3 +/- 2.1, $18.50 claude-sonnet-4-6: 81.8 +/- 1.9, $19.50 claude-opus-5-5: 91.4 +/- 1.4, $26.00 claude-opus-5: 92.2 +/- 1.4, $32.50 claude-opus-4-8: 86.9 +/- 1.7, $32.50 gpt-5.5: 82.4 +/- 2.0, $37.00 gpt-6-astra: 94.2 +/- 1.2, $65.00 claude-fable-5-1: 91.7 +/- 1.9, $65.00 gemini-3.8-flash Cheapclaude-opus-5-5 Best, Balanced
Ranking, Hard
1. claude-opus-5-5: 91.4 +/- 1.41claude-opus-5-591.42. gpt-6-astra: 94.2 +/- 1.22gpt-6-astra94.23. gemini-3.8-flash: 92.3 +/- 1.33gemini-3.8-flash92.34. claude-opus-5: 92.2 +/- 1.44claude-opus-592.25. claude-fable-5-1: 91.7 +/- 1.95claude-fable-5-191.76. gemini-3.1-pro-preview: 89.5 +/- 1.56gemini-3.1-pro-preview89.57. claude-opus-4-8: 86.9 +/- 1.77claude-opus-4-886.98. gemini-3.5-flash: 85.8 +/- 1.78gemini-3.5-flash85.89. gpt-5.5: 82.4 +/- 2.09gpt-5.582.410. claude-sonnet-5: 84.5 +/- 1.810claude-sonnet-584.511. claude-sonnet-4-6: 81.8 +/- 1.911claude-sonnet-4-681.812. gpt-5.4: 80.3 +/- 2.112gpt-5.480.313. qwen3.5-9b: 79.0 +/- 2.413qwen3.5-9b79.014. deepseek-flash: 79.0 +/- 2.014deepseek-flash79.015. deepseek-flash@together: 79.0 +/- 2.015deepseek-flash@together79.016. deepseek-v4-pro: 78.0 +/- 2.116deepseek-v4-pro78.017. deepseek-v4-pro@together: 78.0 +/- 2.117deepseek-v4-pro@together78.018. grok-4.7: 82.7 +/- 2.718grok-4.782.719. gemini-3.5-flash-lite: 68.7 +/- 2.319gemini-3.5-flash-lite68.720. gpt-5-mini: 71.7 +/- 3.220gpt-5-mini71.721. gpt-oss-120b: 75.8 +/- 2.721gpt-oss-120b75.822. gpt-4.1: 66.9 +/- 2.822gpt-4.166.923. gpt-5.4-mini: 67.1 +/- 2.423gpt-5.4-mini67.124. mistral-medium-3-5: 59.5 +/- 2.824mistral-medium-3-559.525. claude-haiku-4-5: 60.5 +/- 2.825claude-haiku-4-560.526. gpt-4.1-mini: 65.9 +/- 2.726gpt-4.1-mini65.927. gpt-5.4-nano: 66.1 +/- 2.427gpt-5.4-nano66.128. mistral-large-2512: 51.3 +/- 2.728mistral-large-251251.329. mistral-small-2603: 47.5 +/- 2.829mistral-small-260347.530. llama-3.3-70b-instruct-turbo: 47.4 +/- 2.930llama-3.3-70b-instruct-turbo47.431. gpt-4o-mini: 37.7 +/- 2.431gpt-4o-mini37.7Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Price against score, Hardest
0255075100$1$10$100 gpt-4o-mini: 0.0 +/- 0.4, $1.35 gpt-5.4-nano: 9.9 +/- 1.1, $2.48 deepseek-flash: 7.0 +/- 1.0, $2.70 deepseek-flash@together: 7.0 +/- 1.0, $2.70 gpt-4.1-mini: 1.8 +/- 0.5, $3.60 gpt-5-mini: 15.8 +/- 0.9, $3.75 gemini-3.5-flash-lite: 9.3 +/- 1.1, $4.65 gemini-3.8-flash: 44.5 +/- 2.0, $7.88 gpt-5.4-mini: 7.1 +/- 1.0, $9.00 deepseek-v4-pro: 6.9 +/- 0.9, $9.90 deepseek-v4-pro@together: 6.9 +/- 0.9, $9.90 claude-haiku-4-5: 7.8 +/- 1.0, $10.50 mistral-medium-3-5: 4.5 +/- 0.8, $15.75 gemini-3.5-flash: 28.9 +/- 1.8, $18.00 gpt-4.1: 3.8 +/- 0.5, $18.00 gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00 gpt-5.4: 45.0 +/- 1.8, $30.00 claude-sonnet-4-6: 73.5 +/- 2.4, $31.50 claude-opus-5-5: 94.6 +/- 1.3, $42.00 claude-opus-5: 92.9 +/- 1.5, $52.50 claude-opus-4-8: 75.1 +/- 2.4, $52.50 gpt-5.5: 54.8 +/- 2.4, $60.00 claude-fable-5-1: 73.2 +/- 1.6, $105 gpt-6-astra: 66.8 +/- 1.7, $105 gemini-3.8-flash Cheapclaude-opus-5-5 Best, Balanced
Ranking, Hardest
1. claude-opus-5: 92.9 +/- 1.51claude-opus-592.92. claude-opus-5-5: 94.6 +/- 1.32claude-opus-5-594.63. claude-fable-5-1: 73.2 +/- 1.63claude-fable-5-173.24. gemini-3.1-pro-preview: 73.8 +/- 1.54gemini-3.1-pro-preview73.85. gpt-6-astra: 66.8 +/- 1.75gpt-6-astra66.86. claude-opus-4-8: 75.1 +/- 2.46claude-opus-4-875.17. gemini-3.8-flash: 44.5 +/- 2.07gemini-3.8-flash44.58. claude-sonnet-4-6: 73.5 +/- 2.48claude-sonnet-4-673.59. gpt-5.5: 54.8 +/- 2.49gpt-5.554.811. gpt-5.4: 45.0 +/- 1.811gpt-5.445.012. gemini-3.5-flash: 28.9 +/- 1.812gemini-3.5-flash28.914. gemini-3.5-flash-lite: 9.3 +/- 1.114gemini-3.5-flash-lite9.315. deepseek-v4-pro: 6.9 +/- 0.915deepseek-v4-pro6.916. deepseek-v4-pro@together: 6.9 +/- 0.916deepseek-v4-pro@together6.917. gpt-5.4-mini: 7.1 +/- 1.017gpt-5.4-mini7.118. deepseek-flash: 7.0 +/- 1.018deepseek-flash7.019. deepseek-flash@together: 7.0 +/- 1.019deepseek-flash@together7.020. claude-haiku-4-5: 7.8 +/- 1.020claude-haiku-4-57.821. gpt-5.4-nano: 9.9 +/- 1.121gpt-5.4-nano9.923. mistral-medium-3-5: 4.5 +/- 0.823mistral-medium-3-54.524. gpt-5-mini: 15.8 +/- 0.924gpt-5-mini15.828. gpt-4.1: 3.8 +/- 0.528gpt-4.13.829. gpt-4.1-mini: 1.8 +/- 0.529gpt-4.1-mini1.830. gpt-4o-mini: 0.0 +/- 0.430gpt-4o-mini0.0Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Rating only, not on the chart (6)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • gpt-oss-120bRating 1,356.9$1.35
  • mistral-small-2603Rating 1,334.6$1.35
  • mistral-large-2512Rating 1,427.4$3.75
  • llama-3.3-70b-instruct-turboRating 1,297.5$4.68
  • grok-4.7Rating 1,458.6$16.50
  • claude-sonnet-5Rating 1,511.8$21.00

Math

Nobody publishes a scored result for this one yet. The models below have a preference rating only.

Ranking, Medium
Rating only, not on the chart (30)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • gpt-oss-120bRating 1,381.1$0.36
  • mistral-small-2603Rating 1,338.0$0.36
  • gpt-4o-miniRating 1,275.4$0.36
  • gpt-5.4-nanoRating 1,424.7$0.66
  • deepseek-flashRating 1,424.8$0.72
  • deepseek-flash@togetherRating 1,424.8$0.72
  • gpt-4.1-miniRating 1,353.4$0.96
  • gpt-5-miniRating 1,405.0$1.00
  • mistral-large-2512Rating 1,401.5$1.00
  • gemini-3.5-flash-liteRating 1,441.9$1.24
  • llama-3.3-70b-instruct-turboRating 1,295.8$1.25
  • gemini-3.8-flashRating 1,516.4$2.10
  • gpt-5.4-miniRating 1,438.8$2.40
  • deepseek-v4-proRating 1,444.8$2.64
  • deepseek-v4-pro@togetherRating 1,444.8$2.64
  • claude-haiku-4-5Rating 1,400.2$2.80
  • mistral-medium-3-5Rating 1,428.4$4.20
  • grok-4.7Rating 1,444.9$4.40
  • gemini-3.5-flashRating 1,496.3$4.80
  • gpt-4.1Rating 1,374.6$4.80
  • claude-sonnet-5Rating 1,473.6$5.60
  • gemini-3.1-pro-previewRating 1,487.9$6.40
  • gpt-5.4Rating 1,459.9$8.00
  • claude-sonnet-4-6Rating 1,463.2$8.40
  • claude-opus-5-5Rating 1,510.2$11.20
  • claude-opus-5Rating 1,522.5$14.00
  • claude-opus-4-8Rating 1,476.3$14.00
  • gpt-5.5Rating 1,499.4$16.00
  • claude-fable-5-1Rating 1,517.5$28.00
  • gpt-6-astraRating 1,490.0$28.00
Price against score, Hard
0255075100$1$10$100 gpt-4o-mini: 52.6 +/- 1.1, $0.84 mistral-small-2603: 46.8 +/- 1.2, $0.84 gpt-5.4-nano: 91.0 +/- 1.5, $1.53 deepseek-flash: 79.7 +/- 2.1, $1.68 deepseek-flash@together: 79.7 +/- 2.1, $1.68 gpt-4.1-mini: 87.3 +/- 0.7, $2.24 gpt-5-mini: 97.9 +/- 0.3, $2.30 mistral-large-2512: 50.3 +/- 1.2, $2.35 gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02 gemini-3.8-flash: 91.6 +/- 1.4, $4.88 gpt-5.4-mini: 78.5 +/- 2.1, $5.55 deepseek-v4-pro: 90.7 +/- 1.5, $6.20 deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20 claude-haiku-4-5: 86.9 +/- 0.8, $6.50 mistral-medium-3-5: 81.6 +/- 0.9, $9.75 grok-4.7: 95.7 +/- 1.0, $10.34 gemini-3.5-flash: 88.2 +/- 1.7, $11.10 gpt-4.1: 83.0 +/- 0.9, $11.20 claude-sonnet-5: 92.9 +/- 1.3, $13.00 gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80 gpt-5.4: 94.2 +/- 1.2, $18.50 claude-sonnet-4-6: 87.0 +/- 1.7, $19.50 claude-opus-5-5: 97.1 +/- 0.9, $26.00 claude-opus-5: 95.7 +/- 1.0, $32.50 claude-opus-4-8: 94.3 +/- 1.2, $32.50 gpt-5.5: 95.9 +/- 1.0, $37.00 claude-fable-5-1: 97.0 +/- 0.9, $65.00 gpt-6-astra: 96.8 +/- 0.9, $65.00 gpt-5-mini Best, Balanced, Cheap
Ranking, Hard
1. claude-opus-5-5: 97.1 +/- 0.91claude-opus-5-597.12. claude-fable-5-1: 97.0 +/- 0.92claude-fable-5-197.03. gpt-6-astra: 96.8 +/- 0.93gpt-6-astra96.84. gpt-5.5: 95.9 +/- 1.04gpt-5.595.95. claude-opus-5: 95.7 +/- 1.05claude-opus-595.76. grok-4.7: 95.7 +/- 1.06grok-4.795.77. claude-opus-4-8: 94.3 +/- 1.27claude-opus-4-894.38. gpt-5.4: 94.2 +/- 1.28gpt-5.494.29. gpt-5-mini: 97.9 +/- 0.39gpt-5-mini97.910. claude-sonnet-5: 92.9 +/- 1.310claude-sonnet-592.911. gemini-3.8-flash: 91.6 +/- 1.411gemini-3.8-flash91.612. gemini-3.1-pro-preview: 91.0 +/- 1.512gemini-3.1-pro-preview91.013. gpt-5.4-nano: 91.0 +/- 1.513gpt-5.4-nano91.014. deepseek-v4-pro: 90.7 +/- 1.514deepseek-v4-pro90.715. deepseek-v4-pro@together: 90.7 +/- 1.515deepseek-v4-pro@together90.716. gemini-3.5-flash: 88.2 +/- 1.716gemini-3.5-flash88.217. claude-sonnet-4-6: 87.0 +/- 1.717claude-sonnet-4-687.018. gpt-4.1-mini: 87.3 +/- 0.718gpt-4.1-mini87.319. claude-haiku-4-5: 86.9 +/- 0.819claude-haiku-4-586.920. deepseek-flash: 79.7 +/- 2.120deepseek-flash79.721. deepseek-flash@together: 79.7 +/- 2.121deepseek-flash@together79.722. gpt-5.4-mini: 78.5 +/- 2.122gpt-5.4-mini78.523. gemini-3.5-flash-lite: 73.7 +/- 2.323gemini-3.5-flash-lite73.724. gpt-4.1: 83.0 +/- 0.924gpt-4.183.025. mistral-medium-3-5: 81.6 +/- 0.925mistral-medium-3-581.626. gpt-4o-mini: 52.6 +/- 1.126gpt-4o-mini52.627. mistral-large-2512: 50.3 +/- 1.227mistral-large-251250.328. mistral-small-2603: 46.8 +/- 1.228mistral-small-260346.829. llama-3.3-70b-instruct-turbo: 41.6 +/- 1.229llama-3.3-70b-instruct-turbo41.6Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Price against score, Hardest
0255075100$1$10$100 qwen3.5-9b: 42.8 +/- 6.5, $0.89 gpt-oss-120b: 88.9 +/- 4.5, $1.35 mistral-small-2603: 5.3 +/- 2.6, $1.35 gpt-4o-mini: 3.8 +/- 1.4, $1.35 gpt-5.4-nano: 25.7 +/- 2.9, $2.48 deepseek-flash: 76.0 +/- 1.8, $2.70 deepseek-flash@together: 76.0 +/- 1.8, $2.70 gpt-4.1-mini: 18.6 +/- 2.2, $3.60 gpt-5-mini: 34.1 +/- 2.9, $3.75 mistral-large-2512: 1.1 +/- 0.5, $3.75 gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65 llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68 gemini-3.8-flash: 83.7 +/- 1.5, $7.88 gpt-5.4-mini: 23.6 +/- 2.7, $9.00 deepseek-v4-pro: 54.0 +/- 4.6, $9.90 deepseek-v4-pro@together: 54.0 +/- 4.6, $9.90 claude-haiku-4-5: 20.0 +/- 3.2, $10.50 mistral-medium-3-5: 16.3 +/- 3.0, $15.75 gemini-3.5-flash: 65.3 +/- 2.2, $18.00 gpt-4.1: 16.6 +/- 2.2, $18.00 claude-sonnet-5: 72.8 +/- 3.3, $21.00 gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00 gpt-5.4: 60.9 +/- 3.3, $30.00 claude-sonnet-4-6: 51.8 +/- 3.7, $31.50 claude-opus-5-5: 95.6 +/- 2.6, $42.00 claude-opus-5: 92.8 +/- 1.6, $52.50 claude-opus-4-8: 72.3 +/- 2.5, $52.50 gpt-5.5: 69.1 +/- 4.5, $60.00 gpt-6-astra: 96.8 +/- 2.6, $105 claude-fable-5-1: 95.1 +/- 2.7, $105 gpt-oss-120b Cheapdeepseek-flash Balancedclaude-opus-5-5 Best
Ranking, Hardest
1. gpt-6-astra: 96.8 +/- 2.61gpt-6-astra96.82. claude-opus-5-5: 95.6 +/- 2.62claude-opus-5-595.63. claude-fable-5-1: 95.1 +/- 2.73claude-fable-5-195.14. claude-opus-5: 92.8 +/- 1.64claude-opus-592.85. claude-opus-4-8: 72.3 +/- 2.55claude-opus-4-872.36. gemini-3.8-flash: 83.7 +/- 1.56gemini-3.8-flash83.77. gemini-3.1-pro-preview: 64.1 +/- 1.77gemini-3.1-pro-preview64.18. gemini-3.5-flash: 65.3 +/- 2.28gemini-3.5-flash65.39. gpt-oss-120b: 88.9 +/- 4.59gpt-oss-120b88.910. deepseek-flash: 76.0 +/- 1.810deepseek-flash76.011. deepseek-flash@together: 76.0 +/- 1.811deepseek-flash@together76.012. gpt-5.4: 60.9 +/- 3.312gpt-5.460.913. claude-sonnet-5: 72.8 +/- 3.313claude-sonnet-572.814. gpt-5.5: 69.1 +/- 4.514gpt-5.569.115. claude-sonnet-4-6: 51.8 +/- 3.715claude-sonnet-4-651.816. deepseek-v4-pro: 54.0 +/- 4.616deepseek-v4-pro54.017. deepseek-v4-pro@together: 54.0 +/- 4.617deepseek-v4-pro@together54.018. gemini-3.5-flash-lite: 44.6 +/- 4.218gemini-3.5-flash-lite44.619. gpt-5-mini: 34.1 +/- 2.919gpt-5-mini34.120. qwen3.5-9b: 42.8 +/- 6.520qwen3.5-9b42.821. claude-haiku-4-5: 20.0 +/- 3.221claude-haiku-4-520.022. gpt-5.4-nano: 25.7 +/- 2.922gpt-5.4-nano25.723. gpt-5.4-mini: 23.6 +/- 2.723gpt-5.4-mini23.624. gpt-4.1-mini: 18.6 +/- 2.224gpt-4.1-mini18.625. mistral-medium-3-5: 16.3 +/- 3.025mistral-medium-3-516.326. gpt-4.1: 16.6 +/- 2.226gpt-4.116.627. mistral-small-2603: 5.3 +/- 2.627mistral-small-26035.328. llama-3.3-70b-instruct-turbo: 5.1 +/- 2.428llama-3.3-70b-instruct-turbo5.129. mistral-large-2512: 1.1 +/- 0.529mistral-large-25121.130. gpt-4o-mini: 3.8 +/- 1.430gpt-4o-mini3.8Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model

Writing code

Price against score, Hard
0255075100$2$5$10$20$50$100 gpt-5.4-nano: 58.8 +/- 3.6, $2.09 deepseek-flash: 53.4 +/- 3.6, $2.22 deepseek-flash@together: 53.4 +/- 3.6, $2.22 gemini-3.5-flash-lite: 60.7 +/- 3.5, $3.98 gemini-3.8-flash: 63.4 +/- 3.6, $6.56 gpt-5.4-mini: 56.7 +/- 3.6, $7.58 deepseek-v4-pro: 56.3 +/- 3.6, $7.99 deepseek-v4-pro@together: 56.3 +/- 3.6, $7.99 grok-4.7: 65.6 +/- 3.5, $13.31 gemini-3.5-flash: 63.6 +/- 3.5, $15.15 claude-sonnet-5: 70.0 +/- 3.4, $17.50 gemini-3.1-pro-preview: 43.2 +/- 2.8, $20.20 gpt-5.4: 49.0 +/- 3.5, $25.25 claude-sonnet-4-6: 61.0 +/- 3.5, $26.25 claude-opus-5-5: 79.5 +/- 3.4, $35.00 claude-opus-5: 73.3 +/- 3.3, $43.75 claude-opus-4-8: 66.2 +/- 3.4, $43.75 gpt-5.5: 48.7 +/- 3.1, $50.50 claude-fable-5-1: 75.3 +/- 3.7, $87.50 gpt-6-astra: 61.5 +/- 4.6, $87.50 claude-opus-5-5 Best, Balanced, Cheap
Ranking, Hard
1. claude-opus-5-5: 79.5 +/- 3.41claude-opus-5-579.52. claude-fable-5-1: 75.3 +/- 3.72claude-fable-5-175.33. claude-opus-5: 73.3 +/- 3.33claude-opus-573.34. gpt-6-astra: 61.5 +/- 4.64gpt-6-astra61.55. claude-sonnet-5: 70.0 +/- 3.45claude-sonnet-570.06. claude-opus-4-8: 66.2 +/- 3.46claude-opus-4-866.27. gemini-3.8-flash: 63.4 +/- 3.67gemini-3.8-flash63.48. gpt-5.5: 48.7 +/- 3.18gpt-5.548.79. claude-sonnet-4-6: 61.0 +/- 3.59claude-sonnet-4-661.010. grok-4.7: 65.6 +/- 3.510grok-4.765.611. gemini-3.5-flash: 63.6 +/- 3.511gemini-3.5-flash63.612. gpt-5.4: 49.0 +/- 3.512gpt-5.449.013. gemini-3.1-pro-preview: 43.2 +/- 2.813gemini-3.1-pro-preview43.214. gemini-3.5-flash-lite: 60.7 +/- 3.514gemini-3.5-flash-lite60.715. deepseek-v4-pro: 56.3 +/- 3.615deepseek-v4-pro56.316. deepseek-v4-pro@together: 56.3 +/- 3.616deepseek-v4-pro@together56.317. deepseek-flash: 53.4 +/- 3.617deepseek-flash53.418. deepseek-flash@together: 53.4 +/- 3.618deepseek-flash@together53.419. gpt-5.4-mini: 56.7 +/- 3.619gpt-5.4-mini56.720. gpt-5.4-nano: 58.8 +/- 3.620gpt-5.4-nano58.8Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model
Rating only, not on the chart (10)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-small-2603Rating 1,412.1$1.11
  • gpt-oss-120bRating 1,389.7$1.11
  • gpt-4o-miniRating 1,348.0$1.11
  • gpt-4.1-miniRating 1,433.0$2.96
  • mistral-large-2512Rating 1,469.3$3.03
  • gpt-5-miniRating 1,431.2$3.20
  • llama-3.3-70b-instruct-turboRating 1,345.7$3.48
  • claude-haiku-4-5Rating 1,480.8$8.75
  • mistral-medium-3-5Rating 1,478.6$13.13
  • gpt-4.1Rating 1,456.1$14.80

Fixing code

Price against score, Hard
0255075100$1$10$100 qwen3.5-9b: 9.2 +/- 2.4, $0.68 gpt-oss-120b: 25.6 +/- 1.9, $1.11 gpt-4o-mini: 3.6 +/- 1.2, $1.11 codestral-2508: 11.1 +/- 2.1, $1.82 gpt-5.4-nano: 46.8 +/- 5.9, $2.09 deepseek-flash: 37.6 +/- 5.7, $2.22 deepseek-flash@together: 37.6 +/- 5.7, $2.22 gpt-4.1-mini: 32.4 +/- 3.1, $2.96 gpt-5-mini: 47.0 +/- 1.7, $3.20 gemini-3.5-flash-lite: 45.3 +/- 5.9, $3.98 gemini-3.8-flash: 54.2 +/- 5.9, $6.56 gpt-5.4-mini: 41.7 +/- 5.8, $7.58 deepseek-v4-pro: 57.1 +/- 3.5, $7.99 deepseek-v4-pro@together: 57.1 +/- 3.5, $7.99 claude-haiku-4-5: 13.9 +/- 2.7, $8.75 grok-4.7: 54.0 +/- 5.9, $13.31 gpt-4.1: 49.8 +/- 1.9, $14.80 gemini-3.5-flash: 64.2 +/- 3.1, $15.15 claude-sonnet-5: 59.4 +/- 5.8, $17.50 gemini-3.1-pro-preview: 49.8 +/- 4.0, $20.20 gpt-5.4: 69.7 +/- 2.3, $25.25 claude-sonnet-4-6: 59.9 +/- 2.2, $26.25 claude-opus-5-5: 71.7 +/- 5.3, $35.00 claude-opus-5: 65.2 +/- 5.6, $43.75 claude-opus-4-8: 50.5 +/- 5.9, $43.75 gpt-5.5: 59.0 +/- 3.6, $50.50 claude-fable-5-1: 66.1 +/- 5.6, $87.50 gpt-6-astra: 57.3 +/- 5.8, $87.50 claude-sonnet-5 Cheapclaude-opus-5-5 Best, Balanced
Ranking, Hard
1. claude-opus-5-5: 71.7 +/- 5.31claude-opus-5-571.72. claude-fable-5-1: 66.1 +/- 5.62claude-fable-5-166.13. claude-opus-5: 65.2 +/- 5.63claude-opus-565.24. gpt-5.4: 69.7 +/- 2.34gpt-5.469.75. claude-sonnet-5: 59.4 +/- 5.85claude-sonnet-559.46. gemini-3.5-flash: 64.2 +/- 3.16gemini-3.5-flash64.27. gpt-6-astra: 57.3 +/- 5.87gpt-6-astra57.38. gpt-5.5: 59.0 +/- 3.68gpt-5.559.09. gemini-3.8-flash: 54.2 +/- 5.99gemini-3.8-flash54.210. grok-4.7: 54.0 +/- 5.910grok-4.754.011. deepseek-v4-pro: 57.1 +/- 3.511deepseek-v4-pro57.112. deepseek-v4-pro@together: 57.1 +/- 3.512deepseek-v4-pro@together57.113. claude-opus-4-8: 50.5 +/- 5.913claude-opus-4-850.514. claude-sonnet-4-6: 59.9 +/- 2.214claude-sonnet-4-659.915. gemini-3.1-pro-preview: 49.8 +/- 4.015gemini-3.1-pro-preview49.816. gpt-5.4-nano: 46.8 +/- 5.916gpt-5.4-nano46.817. gpt-4.1: 49.8 +/- 1.917gpt-4.149.818. gemini-3.5-flash-lite: 45.3 +/- 5.918gemini-3.5-flash-lite45.319. gpt-5-mini: 47.0 +/- 1.719gpt-5-mini47.020. gpt-5.4-mini: 41.7 +/- 5.820gpt-5.4-mini41.721. deepseek-flash: 37.6 +/- 5.721deepseek-flash37.622. deepseek-flash@together: 37.6 +/- 5.722deepseek-flash@together37.623. gpt-4.1-mini: 32.4 +/- 3.123gpt-4.1-mini32.424. gpt-oss-120b: 25.6 +/- 1.924gpt-oss-120b25.625. claude-haiku-4-5: 13.9 +/- 2.725claude-haiku-4-513.926. qwen3.5-9b: 9.2 +/- 2.426qwen3.5-9b9.227. codestral-2508: 11.1 +/- 2.127codestral-250811.128. gpt-4o-mini: 3.6 +/- 1.228gpt-4o-mini3.6Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model

SQL and data queries

Price against score, Medium
0255075100$1$10 gpt-5.4-nano: 50.6 +/- 6.9, $0.41 deepseek-flash: 46.1 +/- 6.9, $0.48 deepseek-flash@together: 46.1 +/- 6.9, $0.48 gemini-3.5-flash-lite: 40.6 +/- 6.8, $0.74 gemini-3.8-flash: 45.3 +/- 6.9, $1.35 gpt-5.4-mini: 49.4 +/- 6.9, $1.50 deepseek-v4-pro: 48.2 +/- 6.9, $1.85 deepseek-v4-pro@together: 48.2 +/- 6.9, $1.85 gemini-3.5-flash: 50.5 +/- 6.9, $3.00 grok-4.7: 44.9 +/- 6.9, $3.08 claude-sonnet-5: 42.4 +/- 6.9, $3.60 gemini-3.1-pro-preview: 52.4 +/- 6.9, $4.00 gpt-5.4: 51.8 +/- 6.9, $5.00 claude-sonnet-4-6: 45.1 +/- 6.9, $5.40 claude-opus-5-5: 52.3 +/- 6.9, $7.20 claude-opus-5: 52.0 +/- 6.9, $9.00 claude-opus-4-8: 51.1 +/- 6.9, $9.00 gpt-5.5: 55.9 +/- 6.9, $10.00 gpt-6-astra: 58.4 +/- 6.8, $18.00 claude-fable-5-1: 56.2 +/- 6.9, $18.00 gpt-5.4-nano Best, Balanced, Cheap
Ranking, Medium
1. gpt-6-astra: 58.4 +/- 6.81gpt-6-astra58.42. claude-fable-5-1: 56.2 +/- 6.92claude-fable-5-156.23. gpt-5.5: 55.9 +/- 6.93gpt-5.555.94. gemini-3.1-pro-preview: 52.4 +/- 6.94gemini-3.1-pro-preview52.45. claude-opus-5-5: 52.3 +/- 6.95claude-opus-5-552.36. claude-opus-5: 52.0 +/- 6.96claude-opus-552.07. gpt-5.4: 51.8 +/- 6.97gpt-5.451.88. claude-opus-4-8: 51.1 +/- 6.98claude-opus-4-851.19. gpt-5.4-nano: 50.6 +/- 6.99gpt-5.4-nano50.610. gemini-3.5-flash: 50.5 +/- 6.910gemini-3.5-flash50.511. gpt-5.4-mini: 49.4 +/- 6.911gpt-5.4-mini49.412. deepseek-v4-pro: 48.2 +/- 6.912deepseek-v4-pro48.213. deepseek-v4-pro@together: 48.2 +/- 6.913deepseek-v4-pro@together48.214. deepseek-flash: 46.1 +/- 6.914deepseek-flash46.115. deepseek-flash@together: 46.1 +/- 6.915deepseek-flash@together46.116. gemini-3.8-flash: 45.3 +/- 6.916gemini-3.8-flash45.317. claude-sonnet-4-6: 45.1 +/- 6.917claude-sonnet-4-645.118. grok-4.7: 44.9 +/- 6.918grok-4.744.919. claude-sonnet-5: 42.4 +/- 6.919claude-sonnet-542.420. gemini-3.5-flash-lite: 40.6 +/- 6.820gemini-3.5-flash-lite40.6Tied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next modelTied with the next model

Using tools

Price against score, Hard
0255075100$1$10$100 qwen3.5-9b: 9.2 +/- 2.4, $1.13 gpt-oss-120b: 14.2 +/- 2.3, $1.68 gpt-5-mini: 22.3 +/- 2.6, $4.60 claude-haiku-4-5: 13.9 +/- 2.7, $13.00 gemini-3.1-pro-preview: 59.4 +/- 4.2, $29.60 gpt-5.4: 81.8 +/- 2.0, $37.00 claude-sonnet-4-6: 62.8 +/- 1.8, $39.00 gpt-5.5: 66.1 +/- 2.5, $74.00 gpt-5.4 Best, Balanced, Cheap
Ranking, Hard
1. gpt-5.4: 81.8 +/- 2.01gpt-5.481.82. gpt-5.5: 66.1 +/- 2.52gpt-5.566.13. gemini-3.1-pro-preview: 59.4 +/- 4.23gemini-3.1-pro-preview59.44. claude-sonnet-4-6: 62.8 +/- 1.84claude-sonnet-4-662.86. gpt-5-mini: 22.3 +/- 2.66gpt-5-mini22.37. gpt-oss-120b: 14.2 +/- 2.37gpt-oss-120b14.28. claude-haiku-4-5: 13.9 +/- 2.78claude-haiku-4-513.99. qwen3.5-9b: 9.2 +/- 2.49qwen3.5-9b9.2Tied with the next model
Rating only, not on the chart (1)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-medium-3-5Rating -0.1$19.50

Reading images

Nobody publishes a scored result for this one yet. The models below have a preference rating only.

Ranking, Medium
Rating only, not on the chart (21)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-small-2603Rating 1,141.6$0.36
  • gpt-4o-miniRating 1,065.5$0.36
  • gpt-5.4-nanoRating 1,196.1$0.66
  • gpt-4.1-miniRating 1,181.3$0.96
  • mistral-large-2512Rating 1,221.5$1.00
  • gpt-5-miniRating 1,202.0$1.00
  • gemini-3.5-flash-liteRating 1,266.1$1.24
  • gemini-3.8-flashRating 1,312.0$2.10
  • gpt-5.4-miniRating 1,244.9$2.40
  • mistral-medium-3-5Rating 1,222.4$4.20
  • gemini-3.5-flashRating 1,310.5$4.80
  • gpt-4.1Rating 1,210.3$4.80
  • claude-sonnet-5Rating 1,275.4$5.60
  • gemini-3.1-pro-previewRating 1,296.2$6.40
  • gpt-5.4Rating 1,291.9$8.00
  • claude-sonnet-4-6Rating 1,282.8$8.40
  • claude-opus-5Rating 1,320.9$14.00
  • claude-opus-4-8Rating 1,287.9$14.00
  • gpt-5.5Rating 1,297.3$16.00
  • claude-fable-5-1Rating 1,318.9$28.00
  • gpt-6-astraRating 1,280.1$28.00

Questions about a document

Nobody publishes a scored result for this one yet. The models below have a preference rating only.

Ranking, Medium
Rating only, not on the chart (30)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • mistral-small-2603Rating 1,358.8$0.72
  • gpt-4o-miniRating 1,326.2$0.72
  • gpt-oss-120bRating 1,324.4$0.72
  • gpt-5.4-nanoRating 1,402.6$1.14
  • deepseek-flashRating 1,447.4$1.44
  • deepseek-flash@togetherRating 1,447.4$1.44
  • gpt-5-miniRating 1,370.3$1.60
  • gpt-4.1-miniRating 1,389.9$1.92
  • gemini-3.5-flash-liteRating 1,461.6$1.96
  • mistral-large-2512Rating 1,417.0$2.20
  • llama-3.3-70b-instruct-turboRating 1,313.4$3.74
  • gemini-3.8-flashRating 1,507.9$3.90
  • gpt-5.4-miniRating 1,446.7$4.20
  • claude-haiku-4-5Rating 1,436.3$5.20
  • deepseek-v4-proRating 1,471.7$5.81
  • deepseek-v4-pro@togetherRating 1,471.7$5.81
  • mistral-medium-3-5Rating 1,429.6$7.80
  • gemini-3.5-flashRating 1,481.5$8.40
  • gpt-4.1Rating 1,426.9$9.60
  • grok-4.7Rating 1,450.5$9.68
  • claude-sonnet-5Rating 1,481.8$10.40
  • gemini-3.1-pro-previewRating 1,498.9$11.20
  • gpt-5.4Rating 1,477.4$14.00
  • claude-sonnet-4-6Rating 1,492.4$15.60
  • claude-opus-5-5Rating 1,527.8$20.80
  • claude-opus-5Rating 1,504.8$26.00
  • claude-opus-4-8Rating 1,499.8$26.00
  • gpt-5.5Rating 1,486.6$28.00
  • claude-fable-5-1Rating 1,511.4$52.00
  • gpt-6-astraRating 1,485.6$52.00
Price against score, Hard
0255075100$1$2$5$10$20$50 gpt-oss-120b: 36.1 +/- 5.0, $1.74 gpt-5-mini: 69.4 +/- 5.0, $3.80 gpt-4.1-mini: 44.4 +/- 5.0, $4.64 llama-3.3-70b-instruct-turbo: 33.3 +/- 5.0, $9.26 gpt-4.1: 63.9 +/- 5.0, $23.20
Ranking, Hard
3. gpt-5-mini: 69.4 +/- 5.03gpt-5-mini69.44. gpt-4.1: 63.9 +/- 5.04gpt-4.163.910. gpt-4.1-mini: 44.4 +/- 5.010gpt-4.1-mini44.413. gpt-oss-120b: 36.1 +/- 5.013gpt-oss-120b36.114. llama-3.3-70b-instruct-turbo: 33.3 +/- 5.014llama-3.3-70b-instruct-turbo33.3Tied with the next modelTied with the next model
Rating only, not on the chart (11)

A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.

  • claude-haiku-4-5Rating 1,420.0$12.50
  • gemini-3.5-flashRating 1,463.3$20.10
  • claude-sonnet-5Rating 1,466.1$25.00
  • gemini-3.1-pro-previewRating 1,443.9$26.80
  • gpt-5.4Rating 1,471.0$33.50
  • claude-sonnet-4-6Rating 1,482.2$37.50
  • claude-opus-5Rating 1,516.3$62.50
  • claude-opus-4-8Rating 1,464.1$62.50
  • gpt-5.5Rating 1,485.7$67.00
  • claude-fable-5-1Rating 1,512.6$125
  • gpt-6-astraRating 1,467.5$125

Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.

Sources

What these numbers are

  • These numbers come from the sources named on each row, not from Zabio's own tests. Zabio's tests are shown separately and never combined with them into one figure.
  • Sources test models at their own settings. A settings mismatch against how Zabio calls the model is marked on the row and lowers its weight.
  • Scores are compared only inside one benchmark. Task scores combine benchmarks by evidence in standard errors, so a task score is a summary, not a benchmark result.
  • A model missing from every source is listed as such rather than left out.

Results from our own tests have their own section.

Models tested

Each model page shows every thinking setting side by side, by task and difficulty.

Head to head

Changes

  • 25 September 2026deepseek-flash: First results: math and SQL at three thinking settings.
  • 25 September 2026deepseek-v4-pro: First results: math and SQL at three thinking settings.

Questions

What is the best model for math?

Among the models tested so far, deepseek-flash with thinking high is the best choice on the hardest math questions, at $9.18 per 1,000 tasks (tested 25 September 2026).

What is the cheapest model that does math well?

On the hardest math questions, the cheap choice among the models tested so far is deepseek-flash with thinking low, at $6.38 per 1,000 tasks.

What is the best model for SQL and data queries?

Among the models tested so far, deepseek-flash with thinking low is the best choice on the hardest SQL and data queries questions, at $4.29 per 1,000 tasks (tested 25 September 2026).

What is the cheapest model that does SQL and data queries well?

On the hardest SQL and data queries questions, the cheap choice among the models tested so far is deepseek-flash with thinking low, at $4.29 per 1,000 tasks.

Does thinking make a model better?

On easy questions it barely changes the result and costs more. On the hardest questions it can add many right answers. Each model page shows every setting side by side.

Where do these numbers come from?

From the leading public benchmarks, Epoch AI, LMArena and LiveBench, refreshed every week, and from Zabio's own tests, where each model is called through Zabio's API and every answer is graded automatically. Zabio uses these same results to pick the model for each job. Each chart names its sources.

Read this first

  • AIME problems are public and may appear in model training data, so math results at the frontier level may be inflated. Differences between settings are the reliable part.
  • SQL questions were kept only when an independent model outside the test answered them correctly, which limits how hard the hardest questions are.
  • These results compare DeepSeek models and settings with each other. Other providers' models have not been tested in this run.
  • Results are not ranked when their ranges overlap. Top means nothing tested is probably better on that job. Near the top means not clearly worse than the top.

Zabio sells access to these models and earns a fee on the traffic it carries. That is why the method is public, every number has its range and date, and corrections are published as changes. No vendor pays for tests, results or placement.

DeepSeek is a trademark of its owner. Zabio uses DeepSeek models through the DeepSeek API and is not affiliated with DeepSeek.

Use the data

The results are free to reuse under CC BY 4.0. Please credit Zabio with the citation below.

License
CC BY 4.0
Suggested citation
Zabio, Model benchmarks, zabio.ai/benchmarks, 2026-09-25
Methodology
1.0
Tested
25 September 2026
Download benchmarks.json