How we test models

Zabio tests the models it offers on its own test questions and publishes the results on zabio.ai, on the benchmark, model and comparison pages. Where we have not tested a model or a kind of job ourselves, the pages also show results aggregated from public benchmark publishers, always labelled as such and never mixed with our own numbers. This page explains how both work and what the numbers mean, so anyone can check them.

Methodology version 1.1, 27 September 2026.

What we test

Models are good at different things: one model writes excellent SQL and makes mistakes in math, another is the reverse. So we test each model per kind of job, not with one overall score, and at four difficulty levels: easy, medium, hard and frontier (problems where even the strongest models often fail).

The first published results cover two kinds of job, math and SQL, for DeepSeek Flash and DeepSeek V4 Pro. More models and kinds of job are added as they are tested; every result shows the date it was measured.

How requests are sent

Every test request goes through Zabio’s own API, exactly as a customer’s request would, naming the model and using the settings Zabio uses by default. The results therefore include everything that happens between your code and the vendor.

Many models can think before they answer. We test each thinking setting as its own option and show it next to every result:

Thinking off: Zabio’s default for these models.
Thinking low and thinking high: the same model when a request asks for more reasoning with reasoning_effort.

The test questions

Math: problems from public competitions: the MATH collection, with its own difficulty levels, and the AIME 2024 and 2025 contests, taken from public collections released under the MIT license. The hardest level uses the AIME problems.
SQL: questions written for Zabio on a test database with realistic data. A question is kept only if an independent model, not among the models being tested, returns exactly the same rows as the correct query. The models being tested never write or check questions.
Questions we write ourselves are not published, so that models cannot be trained on them. Public problems may already be in some models’ training data (see Limits).
Customer requests are never used as test questions.

How answers are graded

SQL: the model’s query runs on the test database. The answer is right when it returns exactly the rows the correct query returns.
Math: the final answer must match the correct one. Equivalent notation counts as right (for example 1/2 and 0.5); when the two differ in form, a small model checks whether they mean the same value.
Kinds of job that need judgment (translation, writing and similar) are graded by judge models from other vendors than the model being graded. They are published only after the judges agree with a person on at least 90 percent of a sample. Translation has been tested but is not published yet for this reason.

An answer cut off before it finished counts as wrong: that is what a customer would receive.

What the numbers mean

Number	Meaning
Right answers	the share of test questions answered right
Range	where the true share probably lies, given how many questions were asked (90 percent range)
Questions	how many questions the result rests on; a result needs at least 25 to be shown
Group	top: no other option is probably better; near: not clearly worse than the top; below: clearly worse
Cost per 1,000 tasks	what 1,000 such requests cost at the vendor’s list price, from the tokens each answer used, thinking included
Right answers per dollar	right answers divided by cost
Answer length and thinking	typical tokens in the answer, and estimated tokens spent thinking
Time	typical time to a complete answer, measured during the test, not on production traffic

“Probably better” means ahead by at least one standard error of the difference: about five chances in six that the lead is real. “Clearly worse” means behind by at least two: about 98 percent sure. A comparison page shows one model ahead of another only when it is probably better; otherwise it shows them as equal.

Best, balanced and cheap

For each kind of job and difficulty, the pages show three choices among the options tested:

Best: the cheapest option that no other option is probably better than. A more expensive option is chosen only when the results show it does better.
Cheap: the cheapest option that is not clearly worse than the top.
Balanced: the middle ground between the two.

These choices describe Zabio’s own test results. Automatic routing chooses by a skill score for the kind of work (see Models and routing) and does not read these results yet.

How often we test

Every result carries the date it was measured. We test new models and test again when a model changes. Automatic re-testing, with a weekly check of every model, is being built; until it runs, results are updated when we run a new test.

Limits

Public math problems may be in some models’ training data, so results at the hardest math level may be higher than a model’s real ability. The differences between thinking settings are the reliable part.
SQL questions were kept only when an independent model could answer them, which limits how hard the hardest SQL questions are.
Times are measured from our test environment, not from production traffic.
Results compare the models tested so far. The first round covers DeepSeek models only.
Vendors can change a model without notice between tests.

Neutrality and corrections

Zabio sells access to these models and earns a fee on the traffic it carries, so we publish the method and the data for anyone to check. No vendor pays for tests, results or placement. Our own tests and the public results we aggregate are always shown apart, each with its source, and are never combined into one number.

If you find an error, write to hello@zabio.ai. Corrections are published as dated changes on the benchmark pages.

Public results we aggregate

Our own tests cover a few kinds of job and models so far. For the rest, the pages show results published by three sources, refreshed every week, each row marked “Aggregated from public results” with its source, license and the date we read it:

Epoch AI (epoch.ai/benchmarks, CC BY 4.0): right-answer benchmarks. Epoch’s own runs come with a standard error; scores Epoch gathers from vendors and other leaderboards are marked as such and count for half as much, because they are self reported and usually at the vendor’s best settings.
LMArena (lmarena-ai/leaderboard-dataset on Hugging Face, CC BY 4.0): human preference ratings from blind pairwise votes, with confidence ranges. Preference is not correctness, so these count for 0.6 of a right-answer benchmark.
LiveBench (livebench.ai; code under Apache 2.0 and MIT, results used with attribution): right answers per task, with new questions every month, so it is the least affected by training data.

How the aggregation works:

Each source’s model names are mapped to Zabio’s catalog by hand. A result measured at a reasoning setting different from how Zabio calls the model is marked on its row and its weight is reduced (0.85 for a close setting, 0.7 for a different one).
Each benchmark is mapped to one kind of job and one difficulty level. Benchmarks that map to nothing Zabio routes on are still published.
Scores are compared only within one benchmark, never across sources. For two models on one benchmark, the gap is divided by its uncertainty, giving a distance in standard errors, capped at three so no single source decides alone. The distances from every benchmark that maps to a kind of job are combined with the weights above. Two models that never met on the same benchmark are compared on their combined right-answer scores at half weight.
The task score shown is the weighted mean of the right-answer benchmarks, in points, with its standard error. Preference ratings inform the order but never the points.
Best, balanced and cheap on aggregated data follow the same rules as on our own tests. A model that cannot be compared with every other model in a cell is shown but never picked, so missing evidence is never read as “unbeaten”. A model missing from every source is listed as such rather than left out.
Every Monday the sources are read again and the pages update. A run that maps fewer than 80 percent of the catalog, or moves a score by more than 15 points, is held for a person to review first.

Where we have our own results for a model and kind of job, both are shown side by side. Our own results are marked “Tested by Zabio” and take precedence in routing.

Using the data

The results are available as one file, https://zabio.ai/benchmarks.json, under the Creative Commons Attribution 4.0 license (CC BY 4.0): you can reuse them if you credit the source. The aggregated section of that file carries each source’s own license and citation, which apply to those rows. Suggested citation: “Zabio, Model benchmarks, zabio.ai/benchmarks”, with the date of the results.

Model and company names are trademarks of their owners. Zabio uses these models through their vendors’ APIs and is not affiliated with the vendors.

Changes to this method

1.1 (27 September 2026): results aggregated from Epoch AI, LMArena and LiveBench are shown beside our own tests, labelled and weighted as described in “Public results we aggregate”; weekly refresh.
1.0 (25 September 2026): first version.