Open beta Zabio is in open beta: anyone can sign up and use it in production. Models, prices and features can still change while we learn from the first customers. Tell us what breaks at hello@zabio.ai.
Menu
Start free

Model

gemini-3.1-pro-preview

gemini-3.1-pro-preview from Google, Powerful class. List prices per million tokens, and what the public benchmarks say about it.

Class
Powerful
Input, per million
$2.00
Output, per million
$12.00
Cached input
$0.20
Context window
200,000
Longest answer
65,536
Processed in
global
Prices from
14 September 2026

Calling it

Name it in the model field, or let auto choose it for you.

"model": "gemini-3.1-pro-preview"

Supports: images, tools, JSON schema

auto can pick this model when it suits the job and the line you chose.

What it costs

  • Input $2.00 and output $12.00 per million tokens, at the provider list price.
  • Input read from cache: $0.20 per million tokens.
  • gemini-3.1-pro-preview: prompts over 200,001 tokens cost $4.00 in and $18.00 out per 1M tokens.

Public results for this model

Zabio has not run its own tests on this model. Everything below is aggregated from public benchmarks, with the sources named.

Aggregated from public resultsRead on 1 October 2026

gemini-3.1-pro-preview
Writing, Medium: 82.6 +/- 3.2 Evidence, in standard errors: 2.5 LiveBench Language: 85.4 (Close to ours) LiveBench paraphrase: 79.9 (Close to ours) LMArena text style control, instruction following: 1,479.3 (Same settings as ours)Writing, Medium82.6Creative writing, Medium: 77.3 +/- 5.9 Evidence, in standard errors: 2.2 LiveBench story generation: 77.3 (Close to ours) LMArena text style control, creative writing: 1,480.3 (Same settings as ours)Creative writing, Medium77.3Summarizing, Medium: 88.2 +/- 4.6 Evidence, in standard errors: 2.0 LiveBench summarize: 88.2 (Close to ours)Summarizing, Medium88.2Pulling out data, Medium: 88.3 +/- 1.9 Evidence, in standard errors: 1.9 LiveBench Data Analysis: 78.5 (Close to ours) LiveBench tablereformat: 98.0 (Close to ours) LMArena text style control, instruction following: 1,479.3 (Same settings as ours)Pulling out data, Medium88.3Sorting into categories, Medium: 79.1 +/- 2.9 Evidence, in standard errors: 2.2 LiveBench IF: 79.1 (Close to ours)Sorting into categories, Medium79.1Conversation, Medium: 73.5 +/- 1.4 Evidence, in standard errors: 3.3 SimpleQA Verified: 73.5 (Close to ours) LMArena text style control, multi turn: 1,496.5 (Same settings as ours)Conversation, Medium73.5Analysis and reasoning, Medium: 79.6 +/- 2.9 Evidence, in standard errors: 2.5 SimpleBench: 79.6 (Same settings as ours)Analysis and reasoning, Medium79.6Analysis and reasoning, Hard: 89.5 +/- 1.5 Evidence, in standard errors: 2.6 GPQA diamond: 94.1 (Same settings as ours) LiveBench Reasoning: 84.0 (Close to ours) LMArena text style control, hard prompts: 1,507.3 (Same settings as ours)Analysis and reasoning, Hard89.5Analysis and reasoning, Hardest: 73.8 +/- 1.5 Evidence, in standard errors: 3.3 ARC-AGI-2: 77.1 (Same settings as ours) ARC-AGI: 98.0 (Same settings as ours) HLE: 46.4 (Same settings as ours) LMArena text style control, expert: 1,508.6 (Same settings as ours)Analysis and reasoning, Hardest73.8Math, Medium: Rating 1,487.9 Evidence, in standard errors: 1.7 LMArena text style control, math: 1,487.9 (Same settings as ours)Math, MediumRating 1,487.9Math, Hard: 91.0 +/- 1.5 Evidence, in standard errors: 1.6 LiveBench Mathematics: 91.0 (Close to ours)Math, Hard91.0Math, Hardest: 64.1 +/- 1.7 Evidence, in standard errors: 1.8 FrontierMath-2025-02-28-Private: 36.9 (Same settings as ours) FrontierMath-Tiers-1-3-v2-Private: 59.7 (Same settings as ours) OTIS Mock AIME 2024-2025: 95.6 (Same settings as ours)Math, Hardest64.1Writing code, Hard: 43.2 +/- 2.8 Evidence, in standard errors: 0.6 MirrorCode: 8.9 (Close to ours) LiveBench Agentic Coding: 44.1 (Close to ours) LiveBench Coding: 76.5 (Close to ours) LMArena text style control, coding: 1,521.3 (Same settings as ours) LMArena webdev, overall: 1,446.2 (Same settings as ours)Writing code, Hard43.2Fixing code, Hard: 49.8 +/- 4.0 Evidence, in standard errors: 0.2 Terminal Bench: 59.4 (Same settings as ours) LiveBench Agentic Coding: 44.1 (Close to ours)Fixing code, Hard49.8SQL and data queries, Medium: 52.4 +/- 6.9 Evidence, in standard errors: 0.3 LiveBench tablejoin: 52.4 (Close to ours)SQL and data queries, Medium52.4Using tools, Hard: 59.4 +/- 4.2 Evidence, in standard errors: 1.3 Terminal Bench: 59.4 (Same settings as ours) LMArena agent, overall: -0.1 (Same settings as ours)Using tools, Hard59.4Reading images, Medium: Rating 1,296.2 Evidence, in standard errors: 1.5 LMArena vision, overall: 1,296.2 (Same settings as ours)Reading images, MediumRating 1,296.2Questions about a document, Medium: Rating 1,498.9 Evidence, in standard errors: 2.1 LMArena text style control, longer query: 1,498.9 (Same settings as ours)Questions about a document, MediumRating 1,498.9Questions about a document, Hard: Rating 1,443.9 Evidence, in standard errors: -2.4 LMArena document, overall: 1,443.9 (Same settings as ours)Questions about a document, HardRating 1,443.9
The benchmarks behind each bar
Kind of jobBenchmarkScoreSettings
Writing, MediumLiveBench Language85.4Close to ours
Writing, MediumLiveBench paraphrase79.9Close to ours
Writing, MediumLMArena text style control, instruction followingRating 1,479.3Same settings as ours
Creative writing, MediumLiveBench story generation77.3Close to ours
Creative writing, MediumLMArena text style control, creative writingRating 1,480.3Same settings as ours
Summarizing, MediumLiveBench summarize88.2Close to ours
Pulling out data, MediumLiveBench Data Analysis78.5Close to ours
Pulling out data, MediumLiveBench tablereformat98.0Close to ours
Pulling out data, MediumLMArena text style control, instruction followingRating 1,479.3Same settings as ours
Sorting into categories, MediumLiveBench IF79.1Close to ours
Conversation, MediumSimpleQA Verified73.5Close to ours
Conversation, MediumLMArena text style control, multi turnRating 1,496.5Same settings as ours
Analysis and reasoning, MediumSimpleBench79.6Same settings as ours
Analysis and reasoning, HardGPQA diamond94.1Same settings as ours
Analysis and reasoning, HardLiveBench Reasoning84.0Close to ours
Analysis and reasoning, HardLMArena text style control, hard promptsRating 1,507.3Same settings as ours
Analysis and reasoning, HardestARC-AGI-277.1Same settings as ours
Analysis and reasoning, HardestARC-AGI98.0Same settings as ours
Analysis and reasoning, HardestHLE46.4Same settings as ours
Analysis and reasoning, HardestLMArena text style control, expertRating 1,508.6Same settings as ours
Math, MediumLMArena text style control, mathRating 1,487.9Same settings as ours
Math, HardLiveBench Mathematics91.0Close to ours
Math, HardestFrontierMath-2025-02-28-Private36.9Same settings as ours
Math, HardestFrontierMath-Tiers-1-3-v2-Private59.7Same settings as ours
Math, HardestOTIS Mock AIME 2024-202595.6Same settings as ours
Writing code, HardMirrorCode8.9Close to ours
Writing code, HardLiveBench Agentic Coding44.1Close to ours
Writing code, HardLiveBench Coding76.5Close to ours
Writing code, HardLMArena text style control, codingRating 1,521.3Same settings as ours
Writing code, HardLMArena webdev, overallRating 1,446.2Same settings as ours
Fixing code, HardTerminal Bench59.4Same settings as ours
Fixing code, HardLiveBench Agentic Coding44.1Close to ours
SQL and data queries, MediumLiveBench tablejoin52.4Close to ours
Using tools, HardTerminal Bench59.4Same settings as ours
Using tools, HardLMArena agent, overallRating -0.1Same settings as ours
Reading images, MediumLMArena vision, overallRating 1,296.2Same settings as ours
Questions about a document, MediumLMArena text style control, longer queryRating 1,498.9Same settings as ours
Questions about a document, HardLMArena document, overallRating 1,443.9Same settings as ours
Price against score in each kind of job (19)

Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.

Writing, Medium
5060708090100$1$10 gpt-5.4-nano: 64.6 +/- 3.9, $0.66 deepseek-flash: 66.3 +/- 3.9, $0.72 deepseek-flash@together: 66.3 +/- 3.9, $0.72 gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24 gemini-3.8-flash: 83.9 +/- 3.1, $2.10 gpt-5.4-mini: 64.9 +/- 3.9, $2.40 deepseek-v4-pro: 67.2 +/- 3.9, $2.64 deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64 grok-4.7: 79.1 +/- 3.4, $4.40 gemini-3.5-flash: 80.1 +/- 3.4, $4.80 claude-sonnet-5: 68.1 +/- 3.9, $5.60 gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40 gpt-5.4: 73.0 +/- 3.7, $8.00 claude-sonnet-4-6: 67.8 +/- 3.9, $8.40 claude-opus-5-5: 75.3 +/- 3.7, $11.20 claude-opus-5: 77.2 +/- 3.6, $14.00 claude-opus-4-8: 73.8 +/- 3.7, $14.00 gpt-5.5: 79.6 +/- 3.5, $16.00 claude-fable-5-1: 80.5 +/- 3.4, $28.00 gpt-6-astra: 83.8 +/- 3.2, $28.00 gemini-3.8-flash Best, Balanced, Cheapgemini-3.1-pro-preview
Creative writing, Medium
5060708090100$1$10 gpt-5.4-nano: 67.7 +/- 6.6, $0.66 deepseek-flash: 67.3 +/- 6.6, $0.72 deepseek-flash@together: 67.3 +/- 6.6, $0.72 gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24 gemini-3.8-flash: 82.8 +/- 5.3, $2.10 gpt-5.4-mini: 66.6 +/- 6.7, $2.40 deepseek-v4-pro: 67.8 +/- 6.6, $2.64 deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64 grok-4.7: 74.9 +/- 6.1, $4.40 gemini-3.5-flash: 75.2 +/- 6.1, $4.80 claude-sonnet-5: 62.3 +/- 6.9, $5.60 gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40 gpt-5.4: 79.0 +/- 5.8, $8.00 claude-sonnet-4-6: 68.9 +/- 6.6, $8.40 claude-opus-5-5: 70.9 +/- 6.4, $11.20 claude-opus-4-8: 80.6 +/- 5.6, $14.00 claude-opus-5: 61.3 +/- 6.9, $14.00 gpt-5.5: 72.6 +/- 6.3, $16.00 claude-fable-5-1: 78.1 +/- 5.9, $28.00 gpt-6-astra: 74.0 +/- 6.2, $28.00 gemini-3.8-flash Best, Balanced, Cheapgemini-3.1-pro-preview
Summarizing, Medium
5060708090100$1$10 gpt-5.4-nano: 71.8 +/- 6.4, $0.73 deepseek-flash: 68.9 +/- 6.6, $0.96 deepseek-flash@together: 68.9 +/- 6.6, $0.96 gemini-3.5-flash-lite: 68.5 +/- 6.6, $1.22 gemini-3.8-flash: 87.5 +/- 4.7, $2.55 gpt-5.4-mini: 59.6 +/- 6.9, $2.70 deepseek-v4-pro: 69.5 +/- 6.5, $3.96 deepseek-v4-pro@together: 69.5 +/- 6.5, $3.96 gemini-3.5-flash: 82.6 +/- 5.4, $5.40 grok-4.7: 78.2 +/- 5.8, $6.60 claude-sonnet-5: 71.2 +/- 6.4, $6.80 gemini-3.1-pro-preview: 88.2 +/- 4.6, $7.20 gpt-5.4: 68.4 +/- 6.6, $9.00 claude-sonnet-4-6: 61.7 +/- 6.9, $10.20 claude-opus-5-5: 69.1 +/- 6.5, $13.60 claude-opus-4-8: 77.7 +/- 5.9, $17.00 claude-opus-5: 66.4 +/- 6.7, $17.00 gpt-5.5: 72.3 +/- 6.3, $18.00 gpt-6-astra: 79.6 +/- 5.7, $34.00 claude-fable-5-1: 72.2 +/- 6.3, $34.00 gpt-5.4-nano Cheapgemini-3.8-flash Best, Balancedgemini-3.1-pro-preview
Pulling out data, Medium
5060708090100$1$10 gpt-5.4-nano: 82.8 +/- 2.1, $0.41 deepseek-flash: 83.0 +/- 2.1, $0.48 deepseek-flash@together: 83.0 +/- 2.1, $0.48 gemini-3.5-flash-lite: 75.7 +/- 2.3, $0.74 gemini-3.8-flash: 75.1 +/- 2.5, $1.35 gpt-5.4-mini: 85.4 +/- 1.9, $1.50 deepseek-v4-pro: 87.3 +/- 1.9, $1.85 deepseek-v4-pro@together: 87.3 +/- 1.9, $1.85 gemini-3.5-flash: 80.5 +/- 2.4, $3.00 grok-4.7: 88.4 +/- 1.8, $3.08 claude-sonnet-5: 84.9 +/- 2.1, $3.60 gemini-3.1-pro-preview: 88.3 +/- 1.9, $4.00 gpt-5.4: 89.7 +/- 1.7, $5.00 claude-sonnet-4-6: 87.0 +/- 2.2, $5.40 claude-opus-5-5: 89.2 +/- 1.9, $7.20 claude-opus-5: 84.3 +/- 2.4, $9.00 claude-opus-4-8: 80.1 +/- 2.5, $9.00 gpt-5.5: 90.8 +/- 1.7, $10.00 gpt-6-astra: 91.5 +/- 1.6, $18.00 claude-fable-5-1: 87.2 +/- 2.3, $18.00 gpt-5.4-mini Cheapdeepseek-v4-pro Balancedgemini-3.1-pro-previewclaude-opus-5-5 Best
Sorting into categories, Medium
5060708090100$1$10 gpt-5.4-nano: 67.2 +/- 3.3, $0.41 deepseek-flash: 63.1 +/- 3.4, $0.48 deepseek-flash@together: 63.1 +/- 3.4, $0.48 gemini-3.5-flash-lite: 67.2 +/- 3.3, $0.74 gemini-3.8-flash: 81.4 +/- 2.8, $1.35 gpt-5.4-mini: 59.8 +/- 3.5, $1.50 deepseek-v4-pro: 62.4 +/- 3.4, $1.85 deepseek-v4-pro@together: 62.4 +/- 3.4, $1.85 gemini-3.5-flash: 75.6 +/- 3.0, $3.00 grok-4.7: 75.3 +/- 3.1, $3.08 claude-sonnet-5: 63.9 +/- 3.4, $3.60 gemini-3.1-pro-preview: 79.1 +/- 2.9, $4.00 gpt-5.4: 70.2 +/- 3.2, $5.00 claude-sonnet-4-6: 63.2 +/- 3.4, $5.40 claude-opus-5-5: 65.7 +/- 3.4, $7.20 claude-opus-4-8: 72.0 +/- 3.2, $9.00 claude-opus-5: 63.8 +/- 3.4, $9.00 gpt-5.5: 70.7 +/- 3.2, $10.00 gpt-6-astra: 75.6 +/- 3.0, $18.00 claude-fable-5-1: 73.0 +/- 3.1, $18.00 gemini-3.8-flash Best, Balanced, Cheapgemini-3.1-pro-preview
Conversation, Medium
0255075100$1$10 gpt-4o-mini: 8.3 +/- 0.9, $0.48 gpt-5.4-nano: 11.7 +/- 1.0, $0.82 deepseek-flash: 33.6 +/- 1.5, $0.96 deepseek-flash@together: 33.6 +/- 1.5, $0.96 gpt-5-mini: 21.6 +/- 1.3, $1.20 gpt-4.1-mini: 12.7 +/- 1.1, $1.28 gemini-3.8-flash: 69.7 +/- 1.5, $2.70 gpt-5.4-mini: 29.4 +/- 1.4, $3.00 claude-haiku-4-5: 13.2 +/- 1.1, $3.60 deepseek-v4-pro: 52.9 +/- 1.6, $3.70 deepseek-v4-pro@together: 52.9 +/- 1.6, $3.70 gemini-3.5-flash: 66.2 +/- 1.5, $6.00 gpt-4.1: 31.1 +/- 1.5, $6.40 claude-sonnet-5: 33.7 +/- 1.5, $7.20 gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00 gpt-5.4: 45.1 +/- 1.6, $10.00 claude-sonnet-4-6: 32.8 +/- 1.5, $10.80 claude-opus-5-5: 72.2 +/- 1.4, $14.40 claude-opus-4-8: 53.0 +/- 1.6, $18.00 claude-opus-5: 59.9 +/- 1.6, $18.00 gpt-5.5: 63.0 +/- 1.5, $20.00 gpt-6-astra: 75.6 +/- 1.4, $36.00 claude-fable-5-1: 70.8 +/- 1.4, $36.00 gemini-3.8-flash Cheapgemini-3.1-pro-preview Best, Balanced
Analysis and reasoning, Medium
0255075100$1$10 gpt-oss-120b: 22.1 +/- 2.9, $0.36 gpt-4o-mini: 10.7 +/- 2.2, $0.36 deepseek-flash: 61.1 +/- 3.5, $0.72 deepseek-flash@together: 61.1 +/- 3.5, $0.72 mistral-large-2512: 22.5 +/- 3.0, $1.00 llama-3.3-70b-instruct-turbo: 19.9 +/- 2.8, $1.25 gemini-3.5-flash: 76.7 +/- 3.0, $4.80 gpt-4.1: 27.0 +/- 3.1, $4.80 claude-sonnet-5: 60.6 +/- 3.5, $5.60 gemini-3.1-pro-preview: 79.6 +/- 2.9, $6.40 claude-opus-5: 80.6 +/- 2.8, $14.00 claude-opus-4-8: 64.8 +/- 3.4, $14.00 gpt-5.5: 69.0 +/- 3.3, $16.00 gemini-3.5-flash Best, Balanced, Cheapgemini-3.1-pro-preview
Analysis and reasoning, Hard
0255075100$1$10$100 qwen3.5-9b: 79.0 +/- 2.4, $0.57 gpt-oss-120b: 75.8 +/- 2.7, $0.84 mistral-small-2603: 47.5 +/- 2.8, $0.84 gpt-4o-mini: 37.7 +/- 2.4, $0.84 gpt-5.4-nano: 66.1 +/- 2.4, $1.53 deepseek-flash: 79.0 +/- 2.0, $1.68 deepseek-flash@together: 79.0 +/- 2.0, $1.68 gpt-4.1-mini: 65.9 +/- 2.7, $2.24 gpt-5-mini: 71.7 +/- 3.2, $2.30 mistral-large-2512: 51.3 +/- 2.7, $2.35 gemini-3.5-flash-lite: 68.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02 gemini-3.8-flash: 92.3 +/- 1.3, $4.88 gpt-5.4-mini: 67.1 +/- 2.4, $5.55 deepseek-v4-pro: 78.0 +/- 2.1, $6.20 deepseek-v4-pro@together: 78.0 +/- 2.1, $6.20 claude-haiku-4-5: 60.5 +/- 2.8, $6.50 mistral-medium-3-5: 59.5 +/- 2.8, $9.75 grok-4.7: 82.7 +/- 2.7, $10.34 gemini-3.5-flash: 85.8 +/- 1.7, $11.10 gpt-4.1: 66.9 +/- 2.8, $11.20 claude-sonnet-5: 84.5 +/- 1.8, $13.00 gemini-3.1-pro-preview: 89.5 +/- 1.5, $14.80 gpt-5.4: 80.3 +/- 2.1, $18.50 claude-sonnet-4-6: 81.8 +/- 1.9, $19.50 claude-opus-5-5: 91.4 +/- 1.4, $26.00 claude-opus-5: 92.2 +/- 1.4, $32.50 claude-opus-4-8: 86.9 +/- 1.7, $32.50 gpt-5.5: 82.4 +/- 2.0, $37.00 gpt-6-astra: 94.2 +/- 1.2, $65.00 claude-fable-5-1: 91.7 +/- 1.9, $65.00 gemini-3.8-flash Cheapgemini-3.1-pro-previewclaude-opus-5-5 Best, Balanced
Analysis and reasoning, Hardest
0255075100$1$10$100 gpt-4o-mini: 0.0 +/- 0.4, $1.35 gpt-5.4-nano: 9.9 +/- 1.1, $2.48 deepseek-flash: 7.0 +/- 1.0, $2.70 deepseek-flash@together: 7.0 +/- 1.0, $2.70 gpt-4.1-mini: 1.8 +/- 0.5, $3.60 gpt-5-mini: 15.8 +/- 0.9, $3.75 gemini-3.5-flash-lite: 9.3 +/- 1.1, $4.65 gemini-3.8-flash: 44.5 +/- 2.0, $7.88 gpt-5.4-mini: 7.1 +/- 1.0, $9.00 deepseek-v4-pro: 6.9 +/- 0.9, $9.90 deepseek-v4-pro@together: 6.9 +/- 0.9, $9.90 claude-haiku-4-5: 7.8 +/- 1.0, $10.50 mistral-medium-3-5: 4.5 +/- 0.8, $15.75 gemini-3.5-flash: 28.9 +/- 1.8, $18.00 gpt-4.1: 3.8 +/- 0.5, $18.00 gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00 gpt-5.4: 45.0 +/- 1.8, $30.00 claude-sonnet-4-6: 73.5 +/- 2.4, $31.50 claude-opus-5-5: 94.6 +/- 1.3, $42.00 claude-opus-5: 92.9 +/- 1.5, $52.50 claude-opus-4-8: 75.1 +/- 2.4, $52.50 gpt-5.5: 54.8 +/- 2.4, $60.00 claude-fable-5-1: 73.2 +/- 1.6, $105 gpt-6-astra: 66.8 +/- 1.7, $105 gemini-3.8-flash Cheapgemini-3.1-pro-previewclaude-opus-5-5 Best, Balanced
Math, Hard
0255075100$1$10$100 gpt-4o-mini: 52.6 +/- 1.1, $0.84 mistral-small-2603: 46.8 +/- 1.2, $0.84 gpt-5.4-nano: 91.0 +/- 1.5, $1.53 deepseek-flash: 79.7 +/- 2.1, $1.68 deepseek-flash@together: 79.7 +/- 2.1, $1.68 gpt-4.1-mini: 87.3 +/- 0.7, $2.24 gpt-5-mini: 97.9 +/- 0.3, $2.30 mistral-large-2512: 50.3 +/- 1.2, $2.35 gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02 gemini-3.8-flash: 91.6 +/- 1.4, $4.88 gpt-5.4-mini: 78.5 +/- 2.1, $5.55 deepseek-v4-pro: 90.7 +/- 1.5, $6.20 deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20 claude-haiku-4-5: 86.9 +/- 0.8, $6.50 mistral-medium-3-5: 81.6 +/- 0.9, $9.75 grok-4.7: 95.7 +/- 1.0, $10.34 gemini-3.5-flash: 88.2 +/- 1.7, $11.10 gpt-4.1: 83.0 +/- 0.9, $11.20 claude-sonnet-5: 92.9 +/- 1.3, $13.00 gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80 gpt-5.4: 94.2 +/- 1.2, $18.50 claude-sonnet-4-6: 87.0 +/- 1.7, $19.50 claude-opus-5-5: 97.1 +/- 0.9, $26.00 claude-opus-5: 95.7 +/- 1.0, $32.50 claude-opus-4-8: 94.3 +/- 1.2, $32.50 gpt-5.5: 95.9 +/- 1.0, $37.00 claude-fable-5-1: 97.0 +/- 0.9, $65.00 gpt-6-astra: 96.8 +/- 0.9, $65.00 gpt-5-mini Best, Balanced, Cheapgemini-3.1-pro-preview
Math, Hardest
0255075100$1$10$100 qwen3.5-9b: 42.8 +/- 6.5, $0.89 gpt-oss-120b: 88.9 +/- 4.5, $1.35 mistral-small-2603: 5.3 +/- 2.6, $1.35 gpt-4o-mini: 3.8 +/- 1.4, $1.35 gpt-5.4-nano: 25.7 +/- 2.9, $2.48 deepseek-flash: 76.0 +/- 1.8, $2.70 deepseek-flash@together: 76.0 +/- 1.8, $2.70 gpt-4.1-mini: 18.6 +/- 2.2, $3.60 gpt-5-mini: 34.1 +/- 2.9, $3.75 mistral-large-2512: 1.1 +/- 0.5, $3.75 gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65 llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68 gemini-3.8-flash: 83.7 +/- 1.5, $7.88 gpt-5.4-mini: 23.6 +/- 2.7, $9.00 deepseek-v4-pro: 54.0 +/- 4.6, $9.90 deepseek-v4-pro@together: 54.0 +/- 4.6, $9.90 claude-haiku-4-5: 20.0 +/- 3.2, $10.50 mistral-medium-3-5: 16.3 +/- 3.0, $15.75 gemini-3.5-flash: 65.3 +/- 2.2, $18.00 gpt-4.1: 16.6 +/- 2.2, $18.00 claude-sonnet-5: 72.8 +/- 3.3, $21.00 gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00 gpt-5.4: 60.9 +/- 3.3, $30.00 claude-sonnet-4-6: 51.8 +/- 3.7, $31.50 claude-opus-5-5: 95.6 +/- 2.6, $42.00 claude-opus-5: 92.8 +/- 1.6, $52.50 claude-opus-4-8: 72.3 +/- 2.5, $52.50 gpt-5.5: 69.1 +/- 4.5, $60.00 gpt-6-astra: 96.8 +/- 2.6, $105 claude-fable-5-1: 95.1 +/- 2.7, $105 gpt-oss-120b Cheapdeepseek-flash Balancedgemini-3.1-pro-previewclaude-opus-5-5 Best
Writing code, Hard
0255075100$2$5$10$20$50$100 gpt-5.4-nano: 58.8 +/- 3.6, $2.09 deepseek-flash: 53.4 +/- 3.6, $2.22 deepseek-flash@together: 53.4 +/- 3.6, $2.22 gemini-3.5-flash-lite: 60.7 +/- 3.5, $3.98 gemini-3.8-flash: 63.4 +/- 3.6, $6.56 gpt-5.4-mini: 56.7 +/- 3.6, $7.58 deepseek-v4-pro: 56.3 +/- 3.6, $7.99 deepseek-v4-pro@together: 56.3 +/- 3.6, $7.99 grok-4.7: 65.6 +/- 3.5, $13.31 gemini-3.5-flash: 63.6 +/- 3.5, $15.15 claude-sonnet-5: 70.0 +/- 3.4, $17.50 gemini-3.1-pro-preview: 43.2 +/- 2.8, $20.20 gpt-5.4: 49.0 +/- 3.5, $25.25 claude-sonnet-4-6: 61.0 +/- 3.5, $26.25 claude-opus-5-5: 79.5 +/- 3.4, $35.00 claude-opus-5: 73.3 +/- 3.3, $43.75 claude-opus-4-8: 66.2 +/- 3.4, $43.75 gpt-5.5: 48.7 +/- 3.1, $50.50 claude-fable-5-1: 75.3 +/- 3.7, $87.50 gpt-6-astra: 61.5 +/- 4.6, $87.50 gemini-3.1-pro-previewclaude-opus-5-5 Best, Balanced, Cheap
Fixing code, Hard
0255075100$1$10$100 qwen3.5-9b: 9.2 +/- 2.4, $0.68 gpt-oss-120b: 25.6 +/- 1.9, $1.11 gpt-4o-mini: 3.6 +/- 1.2, $1.11 codestral-2508: 11.1 +/- 2.1, $1.82 gpt-5.4-nano: 46.8 +/- 5.9, $2.09 deepseek-flash: 37.6 +/- 5.7, $2.22 deepseek-flash@together: 37.6 +/- 5.7, $2.22 gpt-4.1-mini: 32.4 +/- 3.1, $2.96 gpt-5-mini: 47.0 +/- 1.7, $3.20 gemini-3.5-flash-lite: 45.3 +/- 5.9, $3.98 gemini-3.8-flash: 54.2 +/- 5.9, $6.56 gpt-5.4-mini: 41.7 +/- 5.8, $7.58 deepseek-v4-pro: 57.1 +/- 3.5, $7.99 deepseek-v4-pro@together: 57.1 +/- 3.5, $7.99 claude-haiku-4-5: 13.9 +/- 2.7, $8.75 grok-4.7: 54.0 +/- 5.9, $13.31 gpt-4.1: 49.8 +/- 1.9, $14.80 gemini-3.5-flash: 64.2 +/- 3.1, $15.15 claude-sonnet-5: 59.4 +/- 5.8, $17.50 gemini-3.1-pro-preview: 49.8 +/- 4.0, $20.20 gpt-5.4: 69.7 +/- 2.3, $25.25 claude-sonnet-4-6: 59.9 +/- 2.2, $26.25 claude-opus-5-5: 71.7 +/- 5.3, $35.00 claude-opus-5: 65.2 +/- 5.6, $43.75 claude-opus-4-8: 50.5 +/- 5.9, $43.75 gpt-5.5: 59.0 +/- 3.6, $50.50 claude-fable-5-1: 66.1 +/- 5.6, $87.50 gpt-6-astra: 57.3 +/- 5.8, $87.50 claude-sonnet-5 Cheapgemini-3.1-pro-previewclaude-opus-5-5 Best, Balanced
SQL and data queries, Medium
0255075100$1$10 gpt-5.4-nano: 50.6 +/- 6.9, $0.41 deepseek-flash: 46.1 +/- 6.9, $0.48 deepseek-flash@together: 46.1 +/- 6.9, $0.48 gemini-3.5-flash-lite: 40.6 +/- 6.8, $0.74 gemini-3.8-flash: 45.3 +/- 6.9, $1.35 gpt-5.4-mini: 49.4 +/- 6.9, $1.50 deepseek-v4-pro: 48.2 +/- 6.9, $1.85 deepseek-v4-pro@together: 48.2 +/- 6.9, $1.85 gemini-3.5-flash: 50.5 +/- 6.9, $3.00 grok-4.7: 44.9 +/- 6.9, $3.08 claude-sonnet-5: 42.4 +/- 6.9, $3.60 gemini-3.1-pro-preview: 52.4 +/- 6.9, $4.00 gpt-5.4: 51.8 +/- 6.9, $5.00 claude-sonnet-4-6: 45.1 +/- 6.9, $5.40 claude-opus-5-5: 52.3 +/- 6.9, $7.20 claude-opus-5: 52.0 +/- 6.9, $9.00 claude-opus-4-8: 51.1 +/- 6.9, $9.00 gpt-5.5: 55.9 +/- 6.9, $10.00 gpt-6-astra: 58.4 +/- 6.8, $18.00 claude-fable-5-1: 56.2 +/- 6.9, $18.00 gpt-5.4-nano Best, Balanced, Cheapgemini-3.1-pro-preview
Using tools, Hard
0255075100$1$10$100 qwen3.5-9b: 9.2 +/- 2.4, $1.13 gpt-oss-120b: 14.2 +/- 2.3, $1.68 gpt-5-mini: 22.3 +/- 2.6, $4.60 claude-haiku-4-5: 13.9 +/- 2.7, $13.00 gemini-3.1-pro-preview: 59.4 +/- 4.2, $29.60 gpt-5.4: 81.8 +/- 2.0, $37.00 claude-sonnet-4-6: 62.8 +/- 1.8, $39.00 gpt-5.5: 66.1 +/- 2.5, $74.00 gemini-3.1-pro-previewgpt-5.4 Best, Balanced, Cheap
Questions about a document, Hard
0255075100$1$2$5$10$20$50 gpt-oss-120b: 36.1 +/- 5.0, $1.74 gpt-5-mini: 69.4 +/- 5.0, $3.80 gpt-4.1-mini: 44.4 +/- 5.0, $4.64 llama-3.3-70b-instruct-turbo: 33.3 +/- 5.0, $9.26 gpt-4.1: 63.9 +/- 5.0, $23.20

Sources

What these numbers are

  • These numbers come from the sources named on each row, not from Zabio's own tests. Zabio's tests are shown separately and never combined with them into one figure.
  • Sources test models at their own settings. A settings mismatch against how Zabio calls the model is marked on the row and lowers its weight.
  • Scores are compared only inside one benchmark. Task scores combine benchmarks by evidence in standard errors, so a task score is a summary, not a benchmark result.
  • A model missing from every source is listed as such rather than left out.

Results from our own tests have their own section.

Where to next

How routing picks between these models

Model and company names are trademarks of their owners. Zabio uses these models through their vendors' APIs and is not affiliated with the vendors.