Batching
For work nobody is waiting on: evaluations, backfills, classification, content pipelines. Send one file, and Zabio routes every line like a normal call, at the vendors' batch prices.
The video loads from YouTube only when you press play.
- About 50%off the list price, at the batch tiers of OpenAI, Anthropic, Google, Mistral and Together AI
- 2 min to 7 hfor our test batches to come back: minutes at OpenAI and Google, about an hour for Claude Sonnet, almost 7 hours for Claude Opus; the promise is within 24 hours
- 50,000lines per batch, in a file of up to 100 MiB
How it works
- Upload a JSONL file with purpose batch, in the same format as OpenAI's Batch API, so existing SDK code works unchanged.
- Create the batch with completion_window 24h. Zabio checks every line and sets the estimated cost aside on your balance.
- Every line is classified and routed like a normal call, then grouped by vendor and model and sent to the vendors' batch services.
- Poll the batch and download the output file. Each line carries its custom_id and the zabio object, with what it cost and what it saved.
Without a file
- A single request can get the batch price too: send it to the Responses API with background: true and a zabio.deadline of 24 hours or more.
- Zabio gathers your organization's background requests for up to 15 minutes, or until 500 arrive, and sends them as vendor batches. Each answer lands on its own response, which you poll.
- When the model has no batch tier, the request still runs in the background, at the flex or standard price, and zabio.tier says which one served it.
Which models
- Inside a batch, auto only picks models whose vendor has a batch tier, so an auto line always gets the batch price.
- Batches keep Zabio's routing features: Jev answers classification lines, and Smart Splitting runs its cheap pre-steps before the batched answer.
- A line that names a model without a batch tier stops the batch when it is created, before anything runs or is billed, unless the batch opts in to run such lines at the normal price; every line says which price it got.
- Today the batch tiers of OpenAI, Anthropic, Google, Mistral and Together AI; the feature support page shows which models have one.
What it costs
- Each line is billed when its result arrives, at the vendor's batch price plus Zabio's usual fees: the service fee, and the fixed fee for automatic model choice once per auto line.
- The saving is measured against your usual model's batch price, so the batch discount itself stays yours.
- A running batch is never cut off for money: what it needs is set aside when it starts.
Questions
- How does batching work on Zabio?
- Upload a file of chat requests in the same format as OpenAI's Batch API and create a batch. Zabio routes every line like a normal call, at the vendors' batch prices, about half the list price at OpenAI, Anthropic, Google, Mistral and Together AI, and returns the answers within 24 hours, often much sooner. Inside a batch, auto only picks models with a batch tier; a line that names a model without one stops the batch before anything runs or is billed.
Other features
Smart Splitting
One request, several models, a smaller bill, decided for you on every call. When we can save, we save.
Smart routing
The right model and reasoning setting for every request, picked from evidence, with the reason on every response.
Jev for classification
Tagging, routing, moderation and sentiment for a fraction of the price of a chat model, named or picked by auto.
Caching
Your history cached for you, the 1 hour cache when turns are slow, and never a cut of your cache discount.
Flex
Half price when you can wait, with one deadline field.
Teams and keys
Organizations, member quotas, keys with spend caps and model lists, and your own vendor keys with no service fee.
Start free, upgrade when it's real
Change two lines in your code, send model auto, and every response shows what it cost and what you saved.