Open beta Zabio is in open beta: anyone can sign up and use it in production. Models, prices and features can still change while we learn from the first customers. Tell us what breaks at hello@zabio.ai.
Menu
Start free

Model

gemini-3.5-flash

gemini-3.5-flash from Google, Balanced class. List prices per million tokens, and what the public benchmarks say about it.

Class
Balanced
Input, per million
$0.75
Output, per million
$3.75
Cached input
$0.08
Context window
1,000,000
Longest answer
65,536
Processed in
global
Prices from
11 October 2026

Calling it

Name it in the model field, or let auto choose it for you.

"model": "gemini-3.5-flash"

Supports: images, tools, JSON schema

What it costs

  • Input $0.75 and output $3.75 per million tokens, at the provider list price.
  • Input read from cache: $0.08 per million tokens.

Public results for this model

Zabio has not run its own tests on this model. Everything below is aggregated from public benchmarks, with the sources named.

Aggregated from public resultsRead on 1 October 2026

gemini-3.5-flash
Writing, Medium: 80.1 +/- 3.4 Evidence, in standard errors: 1.9 LiveBench Language: 84.6 (Different settings) LiveBench paraphrase: 75.5 (Different settings) LMArena text style control, instruction following: 1,465.5 (Different settings)Writing, Medium80.1Creative writing, Medium: 75.2 +/- 6.1 Evidence, in standard errors: 1.8 LiveBench story generation: 75.2 (Different settings) LMArena text style control, creative writing: 1,469.1 (Different settings)Creative writing, Medium75.2Summarizing, Medium: 82.6 +/- 5.4 Evidence, in standard errors: 1.2 LiveBench summarize: 82.6 (Different settings)Summarizing, Medium82.6Pulling out data, Medium: 80.5 +/- 2.4 Evidence, in standard errors: 0.1 LiveBench Data Analysis: 64.9 (Different settings) LiveBench tablereformat: 96.1 (Different settings) LMArena text style control, instruction following: 1,465.5 (Different settings)Pulling out data, Medium80.5Sorting into categories, Medium: 75.6 +/- 3.0 Evidence, in standard errors: 1.6 LiveBench IF: 75.6 (Different settings)Sorting into categories, Medium75.6Conversation, Medium: 66.2 +/- 1.5 Evidence, in standard errors: 2.1 SimpleQA Verified: 66.2 (Different settings) LMArena text style control, multi turn: 1,480.8 (Different settings)Conversation, Medium66.2Analysis and reasoning, Medium: 76.7 +/- 3.0 Evidence, in standard errors: 2.2 SimpleBench: 76.7 (Same settings as ours)Analysis and reasoning, Medium76.7Analysis and reasoning, Hard: 85.8 +/- 1.7 Evidence, in standard errors: 1.6 GPQA diamond: 88.9 (Close to ours) LiveBench Reasoning: 82.0 (Different settings) LMArena text style control, hard prompts: 1,494.8 (Different settings)Analysis and reasoning, Hard85.8Analysis and reasoning, Hardest: 28.9 +/- 1.8 Evidence, in standard errors: 0.9 ARC-AGI-2: 8.9 (Close to ours) ARC-AGI: 48.8 (Close to ours) LMArena text style control, expert: 1,492.3 (Different settings)Analysis and reasoning, Hardest28.9Math, Medium: Rating 1,496.3 Evidence, in standard errors: 2.0 LMArena text style control, math: 1,496.3 (Different settings)Math, MediumRating 1,496.3Math, Hard: 88.2 +/- 1.7 Evidence, in standard errors: 0.8 LiveBench Mathematics: 88.2 (Different settings)Math, Hard88.2Math, Hardest: 65.3 +/- 2.2 Evidence, in standard errors: 1.6 FrontierMath-2025-02-28-Private: 39.0 (Different settings) FrontierMath-Tiers-1-3-v2-Private: 62.8 (Different settings) OTIS Mock AIME 2024-2025: 88.9 (Close to ours)Math, Hardest65.3Writing code, Hard: 63.6 +/- 3.5 Evidence, in standard errors: 0.8 LiveBench Agentic Coding: 49.0 (Different settings) LiveBench Coding: 78.2 (Different settings) LMArena text style control, coding: 1,506.9 (Different settings) LMArena webdev, overall: 1,499.1 (Different settings)Writing code, Hard63.6Fixing code, Hard: 64.2 +/- 3.1 Evidence, in standard errors: 1.6 SWE-Bench verified: 79.3 (Different settings) LiveBench Agentic Coding: 49.0 (Different settings)Fixing code, Hard64.2SQL and data queries, Medium: 50.5 +/- 6.9 Evidence, in standard errors: 0.1 LiveBench tablejoin: 50.5 (Different settings)SQL and data queries, Medium50.5Reading images, Medium: Rating 1,310.5 Evidence, in standard errors: 2.4 LMArena vision, overall: 1,310.5 (Different settings)Reading images, MediumRating 1,310.5Questions about a document, Medium: Rating 1,481.5 Evidence, in standard errors: 1.1 LMArena text style control, longer query: 1,481.5 (Different settings)Questions about a document, MediumRating 1,481.5Questions about a document, Hard: Rating 1,463.3 Evidence, in standard errors: -0.9 LMArena document, overall: 1,463.3 (Different settings)Questions about a document, HardRating 1,463.3
The benchmarks behind each bar
Kind of jobBenchmarkScoreSettings
Writing, MediumLiveBench Language84.6Different settings
Writing, MediumLiveBench paraphrase75.5Different settings
Writing, MediumLMArena text style control, instruction followingRating 1,465.5Different settings
Creative writing, MediumLiveBench story generation75.2Different settings
Creative writing, MediumLMArena text style control, creative writingRating 1,469.1Different settings
Summarizing, MediumLiveBench summarize82.6Different settings
Pulling out data, MediumLiveBench Data Analysis64.9Different settings
Pulling out data, MediumLiveBench tablereformat96.1Different settings
Pulling out data, MediumLMArena text style control, instruction followingRating 1,465.5Different settings
Sorting into categories, MediumLiveBench IF75.6Different settings
Conversation, MediumSimpleQA Verified66.2Different settings
Conversation, MediumLMArena text style control, multi turnRating 1,480.8Different settings
Analysis and reasoning, MediumSimpleBench76.7Same settings as ours
Analysis and reasoning, HardGPQA diamond88.9Close to ours
Analysis and reasoning, HardLiveBench Reasoning82.0Different settings
Analysis and reasoning, HardLMArena text style control, hard promptsRating 1,494.8Different settings
Analysis and reasoning, HardestARC-AGI-28.9Close to ours
Analysis and reasoning, HardestARC-AGI48.8Close to ours
Analysis and reasoning, HardestLMArena text style control, expertRating 1,492.3Different settings
Math, MediumLMArena text style control, mathRating 1,496.3Different settings
Math, HardLiveBench Mathematics88.2Different settings
Math, HardestFrontierMath-2025-02-28-Private39.0Different settings
Math, HardestFrontierMath-Tiers-1-3-v2-Private62.8Different settings
Math, HardestOTIS Mock AIME 2024-202588.9Close to ours
Writing code, HardLiveBench Agentic Coding49.0Different settings
Writing code, HardLiveBench Coding78.2Different settings
Writing code, HardLMArena text style control, codingRating 1,506.9Different settings
Writing code, HardLMArena webdev, overallRating 1,499.1Different settings
Fixing code, HardSWE-Bench verified79.3Different settings
Fixing code, HardLiveBench Agentic Coding49.0Different settings
SQL and data queries, MediumLiveBench tablejoin50.5Different settings
Reading images, MediumLMArena vision, overallRating 1,310.5Different settings
Questions about a document, MediumLMArena text style control, longer queryRating 1,481.5Different settings
Questions about a document, HardLMArena document, overallRating 1,463.3Different settings
Price against score in each kind of job (18)

Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.

Writing, Medium
5060708090100$1$10 gpt-5.4-nano: 64.6 +/- 3.9, $0.66 deepseek-flash: 66.3 +/- 3.9, $0.72 deepseek-flash@together: 66.3 +/- 3.9, $0.72 gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24 gemini-3.8-flash: 83.9 +/- 3.1, $2.10 gpt-5.4-mini: 64.9 +/- 3.9, $2.40 deepseek-v4-pro: 67.2 +/- 3.9, $2.64 deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64 grok-4.7: 79.1 +/- 3.4, $4.40 gemini-3.5-flash: 80.1 +/- 3.4, $4.80 claude-sonnet-5: 68.1 +/- 3.9, $5.60 gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40 gpt-5.4: 73.0 +/- 3.7, $8.00 claude-sonnet-4-6: 67.8 +/- 3.9, $8.40 claude-opus-5-5: 75.3 +/- 3.7, $11.20 claude-opus-5: 77.2 +/- 3.6, $14.00 claude-opus-4-8: 73.8 +/- 3.7, $14.00 gpt-5.5: 79.6 +/- 3.5, $16.00 claude-fable-5-1: 80.5 +/- 3.4, $28.00 gpt-6-astra: 83.8 +/- 3.2, $28.00 gemini-3.8-flash Best, Balanced, Cheapgemini-3.5-flash
Creative writing, Medium
5060708090100$1$10 gpt-5.4-nano: 67.7 +/- 6.6, $0.66 deepseek-flash: 67.3 +/- 6.6, $0.72 deepseek-flash@together: 67.3 +/- 6.6, $0.72 gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24 gemini-3.8-flash: 82.8 +/- 5.3, $2.10 gpt-5.4-mini: 66.6 +/- 6.7, $2.40 deepseek-v4-pro: 67.8 +/- 6.6, $2.64 deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64 grok-4.7: 74.9 +/- 6.1, $4.40 gemini-3.5-flash: 75.2 +/- 6.1, $4.80 claude-sonnet-5: 62.3 +/- 6.9, $5.60 gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40 gpt-5.4: 79.0 +/- 5.8, $8.00 claude-sonnet-4-6: 68.9 +/- 6.6, $8.40 claude-opus-5-5: 70.9 +/- 6.4, $11.20 claude-opus-4-8: 80.6 +/- 5.6, $14.00 claude-opus-5: 61.3 +/- 6.9, $14.00 gpt-5.5: 72.6 +/- 6.3, $16.00 claude-fable-5-1: 78.1 +/- 5.9, $28.00 gpt-6-astra: 74.0 +/- 6.2, $28.00 gemini-3.8-flash Best, Balanced, Cheapgemini-3.5-flash
Summarizing, Medium
5060708090100$1$10 gpt-5.4-nano: 71.8 +/- 6.4, $0.73 deepseek-flash: 68.9 +/- 6.6, $0.96 deepseek-flash@together: 68.9 +/- 6.6, $0.96 gemini-3.5-flash-lite: 68.5 +/- 6.6, $1.22 gemini-3.8-flash: 87.5 +/- 4.7, $2.55 gpt-5.4-mini: 59.6 +/- 6.9, $2.70 deepseek-v4-pro: 69.5 +/- 6.5, $3.96 deepseek-v4-pro@together: 69.5 +/- 6.5, $3.96 gemini-3.5-flash: 82.6 +/- 5.4, $5.40 grok-4.7: 78.2 +/- 5.8, $6.60 claude-sonnet-5: 71.2 +/- 6.4, $6.80 gemini-3.1-pro-preview: 88.2 +/- 4.6, $7.20 gpt-5.4: 68.4 +/- 6.6, $9.00 claude-sonnet-4-6: 61.7 +/- 6.9, $10.20 claude-opus-5-5: 69.1 +/- 6.5, $13.60 claude-opus-4-8: 77.7 +/- 5.9, $17.00 claude-opus-5: 66.4 +/- 6.7, $17.00 gpt-5.5: 72.3 +/- 6.3, $18.00 gpt-6-astra: 79.6 +/- 5.7, $34.00 claude-fable-5-1: 72.2 +/- 6.3, $34.00 gpt-5.4-nano Cheapgemini-3.8-flash Best, Balancedgemini-3.5-flash
Pulling out data, Medium
5060708090100$1$10 gpt-5.4-nano: 82.8 +/- 2.1, $0.41 deepseek-flash: 83.0 +/- 2.1, $0.48 deepseek-flash@together: 83.0 +/- 2.1, $0.48 gemini-3.5-flash-lite: 75.7 +/- 2.3, $0.74 gemini-3.8-flash: 75.1 +/- 2.5, $1.35 gpt-5.4-mini: 85.4 +/- 1.9, $1.50 deepseek-v4-pro: 87.3 +/- 1.9, $1.85 deepseek-v4-pro@together: 87.3 +/- 1.9, $1.85 gemini-3.5-flash: 80.5 +/- 2.4, $3.00 grok-4.7: 88.4 +/- 1.8, $3.08 claude-sonnet-5: 84.9 +/- 2.1, $3.60 gemini-3.1-pro-preview: 88.3 +/- 1.9, $4.00 gpt-5.4: 89.7 +/- 1.7, $5.00 claude-sonnet-4-6: 87.0 +/- 2.2, $5.40 claude-opus-5-5: 89.2 +/- 1.9, $7.20 claude-opus-5: 84.3 +/- 2.4, $9.00 claude-opus-4-8: 80.1 +/- 2.5, $9.00 gpt-5.5: 90.8 +/- 1.7, $10.00 gpt-6-astra: 91.5 +/- 1.6, $18.00 claude-fable-5-1: 87.2 +/- 2.3, $18.00 gpt-5.4-mini Cheapdeepseek-v4-pro Balancedgemini-3.5-flashclaude-opus-5-5 Best
Sorting into categories, Medium
5060708090100$1$10 gpt-5.4-nano: 67.2 +/- 3.3, $0.41 deepseek-flash: 63.1 +/- 3.4, $0.48 deepseek-flash@together: 63.1 +/- 3.4, $0.48 gemini-3.5-flash-lite: 67.2 +/- 3.3, $0.74 gemini-3.8-flash: 81.4 +/- 2.8, $1.35 gpt-5.4-mini: 59.8 +/- 3.5, $1.50 deepseek-v4-pro: 62.4 +/- 3.4, $1.85 deepseek-v4-pro@together: 62.4 +/- 3.4, $1.85 gemini-3.5-flash: 75.6 +/- 3.0, $3.00 grok-4.7: 75.3 +/- 3.1, $3.08 claude-sonnet-5: 63.9 +/- 3.4, $3.60 gemini-3.1-pro-preview: 79.1 +/- 2.9, $4.00 gpt-5.4: 70.2 +/- 3.2, $5.00 claude-sonnet-4-6: 63.2 +/- 3.4, $5.40 claude-opus-5-5: 65.7 +/- 3.4, $7.20 claude-opus-4-8: 72.0 +/- 3.2, $9.00 claude-opus-5: 63.8 +/- 3.4, $9.00 gpt-5.5: 70.7 +/- 3.2, $10.00 gpt-6-astra: 75.6 +/- 3.0, $18.00 claude-fable-5-1: 73.0 +/- 3.1, $18.00 gemini-3.8-flash Best, Balanced, Cheapgemini-3.5-flash
Conversation, Medium
0255075100$1$10 gpt-4o-mini: 8.3 +/- 0.9, $0.48 gpt-5.4-nano: 11.7 +/- 1.0, $0.82 deepseek-flash: 33.6 +/- 1.5, $0.96 deepseek-flash@together: 33.6 +/- 1.5, $0.96 gpt-5-mini: 21.6 +/- 1.3, $1.20 gpt-4.1-mini: 12.7 +/- 1.1, $1.28 gemini-3.8-flash: 69.7 +/- 1.5, $2.70 gpt-5.4-mini: 29.4 +/- 1.4, $3.00 claude-haiku-4-5: 13.2 +/- 1.1, $3.60 deepseek-v4-pro: 52.9 +/- 1.6, $3.70 deepseek-v4-pro@together: 52.9 +/- 1.6, $3.70 gemini-3.5-flash: 66.2 +/- 1.5, $6.00 gpt-4.1: 31.1 +/- 1.5, $6.40 claude-sonnet-5: 33.7 +/- 1.5, $7.20 gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00 gpt-5.4: 45.1 +/- 1.6, $10.00 claude-sonnet-4-6: 32.8 +/- 1.5, $10.80 claude-opus-5-5: 72.2 +/- 1.4, $14.40 claude-opus-4-8: 53.0 +/- 1.6, $18.00 claude-opus-5: 59.9 +/- 1.6, $18.00 gpt-5.5: 63.0 +/- 1.5, $20.00 gpt-6-astra: 75.6 +/- 1.4, $36.00 claude-fable-5-1: 70.8 +/- 1.4, $36.00 gemini-3.8-flash Cheapgemini-3.5-flashgemini-3.1-pro-preview Best, Balanced
Analysis and reasoning, Medium
0255075100$1$10 gpt-oss-120b: 22.1 +/- 2.9, $0.36 gpt-4o-mini: 10.7 +/- 2.2, $0.36 deepseek-flash: 61.1 +/- 3.5, $0.72 deepseek-flash@together: 61.1 +/- 3.5, $0.72 mistral-large-2512: 22.5 +/- 3.0, $1.00 llama-3.3-70b-instruct-turbo: 19.9 +/- 2.8, $1.25 gemini-3.5-flash: 76.7 +/- 3.0, $4.80 gpt-4.1: 27.0 +/- 3.1, $4.80 claude-sonnet-5: 60.6 +/- 3.5, $5.60 gemini-3.1-pro-preview: 79.6 +/- 2.9, $6.40 claude-opus-5: 80.6 +/- 2.8, $14.00 claude-opus-4-8: 64.8 +/- 3.4, $14.00 gpt-5.5: 69.0 +/- 3.3, $16.00 gemini-3.5-flash Best, Balanced, Cheap
Analysis and reasoning, Hard
0255075100$1$10$100 qwen3.5-9b: 79.0 +/- 2.4, $0.57 gpt-oss-120b: 75.8 +/- 2.7, $0.84 mistral-small-2603: 47.5 +/- 2.8, $0.84 gpt-4o-mini: 37.7 +/- 2.4, $0.84 gpt-5.4-nano: 66.1 +/- 2.4, $1.53 deepseek-flash: 79.0 +/- 2.0, $1.68 deepseek-flash@together: 79.0 +/- 2.0, $1.68 gpt-4.1-mini: 65.9 +/- 2.7, $2.24 gpt-5-mini: 71.7 +/- 3.2, $2.30 mistral-large-2512: 51.3 +/- 2.7, $2.35 gemini-3.5-flash-lite: 68.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02 gemini-3.8-flash: 92.3 +/- 1.3, $4.88 gpt-5.4-mini: 67.1 +/- 2.4, $5.55 deepseek-v4-pro: 78.0 +/- 2.1, $6.20 deepseek-v4-pro@together: 78.0 +/- 2.1, $6.20 claude-haiku-4-5: 60.5 +/- 2.8, $6.50 mistral-medium-3-5: 59.5 +/- 2.8, $9.75 grok-4.7: 82.7 +/- 2.7, $10.34 gemini-3.5-flash: 85.8 +/- 1.7, $11.10 gpt-4.1: 66.9 +/- 2.8, $11.20 claude-sonnet-5: 84.5 +/- 1.8, $13.00 gemini-3.1-pro-preview: 89.5 +/- 1.5, $14.80 gpt-5.4: 80.3 +/- 2.1, $18.50 claude-sonnet-4-6: 81.8 +/- 1.9, $19.50 claude-opus-5-5: 91.4 +/- 1.4, $26.00 claude-opus-5: 92.2 +/- 1.4, $32.50 claude-opus-4-8: 86.9 +/- 1.7, $32.50 gpt-5.5: 82.4 +/- 2.0, $37.00 gpt-6-astra: 94.2 +/- 1.2, $65.00 claude-fable-5-1: 91.7 +/- 1.9, $65.00 gemini-3.8-flash Cheapgemini-3.5-flashclaude-opus-5-5 Best, Balanced
Analysis and reasoning, Hardest
0255075100$1$10$100 gpt-4o-mini: 0.0 +/- 0.4, $1.35 gpt-5.4-nano: 9.9 +/- 1.1, $2.48 deepseek-flash: 7.0 +/- 1.0, $2.70 deepseek-flash@together: 7.0 +/- 1.0, $2.70 gpt-4.1-mini: 1.8 +/- 0.5, $3.60 gpt-5-mini: 15.8 +/- 0.9, $3.75 gemini-3.5-flash-lite: 9.3 +/- 1.1, $4.65 gemini-3.8-flash: 44.5 +/- 2.0, $7.88 gpt-5.4-mini: 7.1 +/- 1.0, $9.00 deepseek-v4-pro: 6.9 +/- 0.9, $9.90 deepseek-v4-pro@together: 6.9 +/- 0.9, $9.90 claude-haiku-4-5: 7.8 +/- 1.0, $10.50 mistral-medium-3-5: 4.5 +/- 0.8, $15.75 gemini-3.5-flash: 28.9 +/- 1.8, $18.00 gpt-4.1: 3.8 +/- 0.5, $18.00 gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00 gpt-5.4: 45.0 +/- 1.8, $30.00 claude-sonnet-4-6: 73.5 +/- 2.4, $31.50 claude-opus-5-5: 94.6 +/- 1.3, $42.00 claude-opus-5: 92.9 +/- 1.5, $52.50 claude-opus-4-8: 75.1 +/- 2.4, $52.50 gpt-5.5: 54.8 +/- 2.4, $60.00 claude-fable-5-1: 73.2 +/- 1.6, $105 gpt-6-astra: 66.8 +/- 1.7, $105 gemini-3.8-flash Cheapgemini-3.5-flashclaude-opus-5-5 Best, Balanced
Math, Hard
0255075100$1$10$100 gpt-4o-mini: 52.6 +/- 1.1, $0.84 mistral-small-2603: 46.8 +/- 1.2, $0.84 gpt-5.4-nano: 91.0 +/- 1.5, $1.53 deepseek-flash: 79.7 +/- 2.1, $1.68 deepseek-flash@together: 79.7 +/- 2.1, $1.68 gpt-4.1-mini: 87.3 +/- 0.7, $2.24 gpt-5-mini: 97.9 +/- 0.3, $2.30 mistral-large-2512: 50.3 +/- 1.2, $2.35 gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85 llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02 gemini-3.8-flash: 91.6 +/- 1.4, $4.88 gpt-5.4-mini: 78.5 +/- 2.1, $5.55 deepseek-v4-pro: 90.7 +/- 1.5, $6.20 deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20 claude-haiku-4-5: 86.9 +/- 0.8, $6.50 mistral-medium-3-5: 81.6 +/- 0.9, $9.75 grok-4.7: 95.7 +/- 1.0, $10.34 gemini-3.5-flash: 88.2 +/- 1.7, $11.10 gpt-4.1: 83.0 +/- 0.9, $11.20 claude-sonnet-5: 92.9 +/- 1.3, $13.00 gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80 gpt-5.4: 94.2 +/- 1.2, $18.50 claude-sonnet-4-6: 87.0 +/- 1.7, $19.50 claude-opus-5-5: 97.1 +/- 0.9, $26.00 claude-opus-5: 95.7 +/- 1.0, $32.50 claude-opus-4-8: 94.3 +/- 1.2, $32.50 gpt-5.5: 95.9 +/- 1.0, $37.00 claude-fable-5-1: 97.0 +/- 0.9, $65.00 gpt-6-astra: 96.8 +/- 0.9, $65.00 gpt-5-mini Best, Balanced, Cheapgemini-3.5-flash
Math, Hardest
0255075100$1$10$100 qwen3.5-9b: 42.8 +/- 6.5, $0.89 gpt-oss-120b: 88.9 +/- 4.5, $1.35 mistral-small-2603: 5.3 +/- 2.6, $1.35 gpt-4o-mini: 3.8 +/- 1.4, $1.35 gpt-5.4-nano: 25.7 +/- 2.9, $2.48 deepseek-flash: 76.0 +/- 1.8, $2.70 deepseek-flash@together: 76.0 +/- 1.8, $2.70 gpt-4.1-mini: 18.6 +/- 2.2, $3.60 gpt-5-mini: 34.1 +/- 2.9, $3.75 mistral-large-2512: 1.1 +/- 0.5, $3.75 gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65 llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68 gemini-3.8-flash: 83.7 +/- 1.5, $7.88 gpt-5.4-mini: 23.6 +/- 2.7, $9.00 deepseek-v4-pro: 54.0 +/- 4.6, $9.90 deepseek-v4-pro@together: 54.0 +/- 4.6, $9.90 claude-haiku-4-5: 20.0 +/- 3.2, $10.50 mistral-medium-3-5: 16.3 +/- 3.0, $15.75 gemini-3.5-flash: 65.3 +/- 2.2, $18.00 gpt-4.1: 16.6 +/- 2.2, $18.00 claude-sonnet-5: 72.8 +/- 3.3, $21.00 gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00 gpt-5.4: 60.9 +/- 3.3, $30.00 claude-sonnet-4-6: 51.8 +/- 3.7, $31.50 claude-opus-5-5: 95.6 +/- 2.6, $42.00 claude-opus-5: 92.8 +/- 1.6, $52.50 claude-opus-4-8: 72.3 +/- 2.5, $52.50 gpt-5.5: 69.1 +/- 4.5, $60.00 gpt-6-astra: 96.8 +/- 2.6, $105 claude-fable-5-1: 95.1 +/- 2.7, $105 gpt-oss-120b Cheapdeepseek-flash Balancedgemini-3.5-flashclaude-opus-5-5 Best
Writing code, Hard
0255075100$2$5$10$20$50$100 gpt-5.4-nano: 58.8 +/- 3.6, $2.09 deepseek-flash: 53.4 +/- 3.6, $2.22 deepseek-flash@together: 53.4 +/- 3.6, $2.22 gemini-3.5-flash-lite: 60.7 +/- 3.5, $3.98 gemini-3.8-flash: 63.4 +/- 3.6, $6.56 gpt-5.4-mini: 56.7 +/- 3.6, $7.58 deepseek-v4-pro: 56.3 +/- 3.6, $7.99 deepseek-v4-pro@together: 56.3 +/- 3.6, $7.99 grok-4.7: 65.6 +/- 3.5, $13.31 gemini-3.5-flash: 63.6 +/- 3.5, $15.15 claude-sonnet-5: 70.0 +/- 3.4, $17.50 gemini-3.1-pro-preview: 43.2 +/- 2.8, $20.20 gpt-5.4: 49.0 +/- 3.5, $25.25 claude-sonnet-4-6: 61.0 +/- 3.5, $26.25 claude-opus-5-5: 79.5 +/- 3.4, $35.00 claude-opus-5: 73.3 +/- 3.3, $43.75 claude-opus-4-8: 66.2 +/- 3.4, $43.75 gpt-5.5: 48.7 +/- 3.1, $50.50 claude-fable-5-1: 75.3 +/- 3.7, $87.50 gpt-6-astra: 61.5 +/- 4.6, $87.50 gemini-3.5-flashclaude-opus-5-5 Best, Balanced, Cheap
Fixing code, Hard
0255075100$1$10$100 qwen3.5-9b: 9.2 +/- 2.4, $0.68 gpt-oss-120b: 25.6 +/- 1.9, $1.11 gpt-4o-mini: 3.6 +/- 1.2, $1.11 codestral-2508: 11.1 +/- 2.1, $1.82 gpt-5.4-nano: 46.8 +/- 5.9, $2.09 deepseek-flash: 37.6 +/- 5.7, $2.22 deepseek-flash@together: 37.6 +/- 5.7, $2.22 gpt-4.1-mini: 32.4 +/- 3.1, $2.96 gpt-5-mini: 47.0 +/- 1.7, $3.20 gemini-3.5-flash-lite: 45.3 +/- 5.9, $3.98 gemini-3.8-flash: 54.2 +/- 5.9, $6.56 gpt-5.4-mini: 41.7 +/- 5.8, $7.58 deepseek-v4-pro: 57.1 +/- 3.5, $7.99 deepseek-v4-pro@together: 57.1 +/- 3.5, $7.99 claude-haiku-4-5: 13.9 +/- 2.7, $8.75 grok-4.7: 54.0 +/- 5.9, $13.31 gpt-4.1: 49.8 +/- 1.9, $14.80 gemini-3.5-flash: 64.2 +/- 3.1, $15.15 claude-sonnet-5: 59.4 +/- 5.8, $17.50 gemini-3.1-pro-preview: 49.8 +/- 4.0, $20.20 gpt-5.4: 69.7 +/- 2.3, $25.25 claude-sonnet-4-6: 59.9 +/- 2.2, $26.25 claude-opus-5-5: 71.7 +/- 5.3, $35.00 claude-opus-5: 65.2 +/- 5.6, $43.75 claude-opus-4-8: 50.5 +/- 5.9, $43.75 gpt-5.5: 59.0 +/- 3.6, $50.50 claude-fable-5-1: 66.1 +/- 5.6, $87.50 gpt-6-astra: 57.3 +/- 5.8, $87.50 gemini-3.5-flashclaude-sonnet-5 Cheapclaude-opus-5-5 Best, Balanced
SQL and data queries, Medium
0255075100$1$10 gpt-5.4-nano: 50.6 +/- 6.9, $0.41 deepseek-flash: 46.1 +/- 6.9, $0.48 deepseek-flash@together: 46.1 +/- 6.9, $0.48 gemini-3.5-flash-lite: 40.6 +/- 6.8, $0.74 gemini-3.8-flash: 45.3 +/- 6.9, $1.35 gpt-5.4-mini: 49.4 +/- 6.9, $1.50 deepseek-v4-pro: 48.2 +/- 6.9, $1.85 deepseek-v4-pro@together: 48.2 +/- 6.9, $1.85 gemini-3.5-flash: 50.5 +/- 6.9, $3.00 grok-4.7: 44.9 +/- 6.9, $3.08 claude-sonnet-5: 42.4 +/- 6.9, $3.60 gemini-3.1-pro-preview: 52.4 +/- 6.9, $4.00 gpt-5.4: 51.8 +/- 6.9, $5.00 claude-sonnet-4-6: 45.1 +/- 6.9, $5.40 claude-opus-5-5: 52.3 +/- 6.9, $7.20 claude-opus-5: 52.0 +/- 6.9, $9.00 claude-opus-4-8: 51.1 +/- 6.9, $9.00 gpt-5.5: 55.9 +/- 6.9, $10.00 gpt-6-astra: 58.4 +/- 6.8, $18.00 claude-fable-5-1: 56.2 +/- 6.9, $18.00 gpt-5.4-nano Best, Balanced, Cheapgemini-3.5-flash
Questions about a document, Hard
0255075100$1$2$5$10$20$50 gpt-oss-120b: 36.1 +/- 5.0, $1.74 gpt-5-mini: 69.4 +/- 5.0, $3.80 gpt-4.1-mini: 44.4 +/- 5.0, $4.64 llama-3.3-70b-instruct-turbo: 33.3 +/- 5.0, $9.26 gpt-4.1: 63.9 +/- 5.0, $23.20

Sources

What these numbers are

  • These numbers come from the sources named on each row, not from Zabio's own tests. Zabio's tests are shown separately and never combined with them into one figure.
  • Sources test models at their own settings. A settings mismatch against how Zabio calls the model is marked on the row and lowers its weight.
  • Scores are compared only inside one benchmark. Task scores combine benchmarks by evidence in standard errors, so a task score is a summary, not a benchmark result.
  • A model missing from every source is listed as such rather than left out.

Results from our own tests have their own section.

Where to next

How routing picks between these models

Model and company names are trademarks of their owners. Zabio uses these models through their vendors' APIs and is not affiliated with the vendors.