Batches

/v1/files and /v1/batches follow OpenAI’s shape. A batch is a file of chat completion requests, one per line, that Zabio answers within 24 hours. Every line is classified and routed as a normal call, but priced at the vendors’ batch rates, about half of the list price: OpenAI’s Batch API, Anthropic’s Message Batches, Google’s Batch Mode, Mistral’s batch jobs and Together’s batch API for Llama 3.3 70B. The two savings stack: the routing choice, and the batch price of whatever model routing picked. Code written for the OpenAI batch SDK works unchanged with Zabio’s base_url and a Zabio key, auto included.

Batches are for work nobody is waiting on: evaluation runs, backfills, classification, content pipelines. Each line gets the full chat completion a synchronous call would return, zabio object included, so what every line cost and saved is in the result. The key needs the chat scope.

How it works

1. Upload a JSONL file with purpose: batch. Each line is {"custom_id": "...", "method": "POST", "url": "/v1/chat/completions", "body": {...}} and body is a normal chat completion request: auto, a line (auto/best, auto/balanced, auto/cheap) or a named model, messages, tools, response_format, max_tokens. Lines are checked on upload; a bad file is refused with the line number.
2. Create the batch with the file id, endpoint: "/v1/chat/completions" and completion_window: "24h". Zabio checks every line, sets aside the estimated cost on your balance, and starts routing.
3. Poll the batch until its status is completed, expired, cancelled or failed.
4. Download the output file and, if there is one, the error file. Results come back in the order of your input and carry your custom_id.

Upload

curl

cat > requests.jsonl <<'EOF2'
{"custom_id": "r1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "auto", "messages": [{"role": "user", "content": "Summarize: the quarterly report shows revenue up 12 percent."}], "max_tokens": 200}}
{"custom_id": "r2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "claude-haiku-4-5", "messages": [{"role": "user", "content": "Translate to Spanish: good morning."}], "max_tokens": 50}}
EOF2

curl https://api.zabio.ai/v1/files \
  -H "Authorization: Bearer insert your zabio.ai key here" \
  -F purpose=batch \
  -F file=@requests.jsonl

Python

from openai import OpenAI
client = OpenAI(base_url="https://api.zabio.ai/v1", api_key="insert your zabio.ai key here")

f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
print(f.id, f.bytes)   # file_..., the plaintext size

The answer is a file object: id (file_...), bytes, created_at, expires_at, filename, purpose, status (processed). An optional expires_after[anchor]=created_at and expires_after[seconds] (3600 to 2592000) shortens how long the file is kept.

Create

curl

curl https://api.zabio.ai/v1/batches \
  -H "Authorization: Bearer insert your zabio.ai key here" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id": "file_...", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": {"job": "nightly-eval"}}'

Python

b = client.batches.create(input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h", metadata={"job": "nightly-eval"})
print(b.id, b.status)   # batch_..., validating

metadata is optional: up to 16 pairs, keys of at most 64 characters, values of at most 512. The answer is the batch object with status validating.

Poll

curl

curl https://api.zabio.ai/v1/batches/batch_... \
  -H "Authorization: Bearer insert your zabio.ai key here"

Python

import time
while True:
    b = client.batches.retrieve(b.id)
    if b.status in ("completed", "expired", "cancelled", "failed"):
        break
    time.sleep(60)
print(b.status, b.request_counts, b.model_extra["zabio"]["cost"])

Status	Meaning
validating	the lines are being checked and the estimate set aside on your balance
failed	the batch did not run: errors lists why, line by line where it can; nothing was billed
in_progress	the lines are being routed and sent to the vendors
finalizing	every line has ended; the output and error files are being written
completed	done; output_file_id and, when some lines failed, error_file_id are set
expired	done, but some lines could not be answered before the window ended; they are in the error file with batch_expired
cancelling	you cancelled; the vendors are being asked to stop
cancelled	done after a cancel; what the vendors finished before stopping is in the output file

The batch object also carries request_counts (total, completed, failed: failed, expired and cancelled lines together) and a zabio object with tier: "batch" and cost, the running totals of the lines billed so far: vendor_micros, billed_micros, direct_micros (the same lines on your baseline model at its batch price), saved_micros (the lines’ own savings added up: an auto line on a key whose baseline is pending adds none) and classifier_micros (always 0: the classifier is Zabio’s cost).

Download

curl

curl https://api.zabio.ai/v1/files/file_.../content \
  -H "Authorization: Bearer insert your zabio.ai key here" -o output.jsonl

Python

out = client.files.content(b.output_file_id).text
for line in out.splitlines():
    r = json.loads(line)
    print(r["custom_id"], r["response"]["body"]["choices"][0]["message"]["content"], r["response"]["body"]["zabio"]["cost_micros"])
if b.error_file_id:
    print(client.files.content(b.error_file_id).text)

Each output line is {"id": "batch_req_...", "custom_id": "...", "response": {"status_code": 200, "request_id": "req_...", "body": {...}}, "error": null}, and body is the full chat completion the same request would have returned synchronously, with its zabio object: model_served, routing, cost (vendor_micros, the fee, direct_micros, saved_micros), tier ({"requested": "batch", "applied": "batch"}). Each error line is {"id": "batch_req_...", "custom_id": "...", "response": null, "error": {"code": "...", "message": "..."}}; a line the vendor answered with an error also carries response.status_code and an error envelope in response.body. Lines come in the order of your input; match them to your requests by custom_id, never by position.

Batches without files

A single request can reach the batch queue without a file: send it to the Responses API with background: true and a zabio.deadline of 24 hours or more. Zabio gathers your organization’s background requests for up to 15 minutes, or until 500 have arrived, and sends them together as one vendor batch per model, at the batch prices, routed line by line like a file batch. Each request is billed as a batch line, the result arrives within the vendors’ 24-hour window from the moment it was sent, and you read it with GET /v1/responses/{id}. There is no file to upload and no output file to download; these requests do not appear on GET /v1/batches. A background request on a model whose vendor has no batch tier runs as its own job at flex or standard instead, and its zabio.tier says so with fallback: "no_batch_tier".

Which models

Every line is routed as a normal call and priced at the batch rates. Inside a batch, auto and the lines pick only among models whose vendor has a batch API, so an auto line always gets the batch price. The models with a batch tier today are every OpenAI chat model except chat-latest, every Anthropic chat model, every Google Gemini model, every Mistral chat model (the Mistral, Codestral and Ministral rows) and Together’s llama-3.3-70b-instruct-turbo. Prices are on the Models page.

Some models have no batch tier because their vendor publishes no batch price for them: Together’s other models (qwen3.5-9b, gpt-oss-120b and the DeepSeek copies on Together) run at standard rates in Together’s batch API, xAI lists no batch discount for grok-4.7, DeepSeek has no batch API at all. A model gets a batch tier the day its vendor publishes a number, never before: Google printed none for gemini-3.5-flash-lite until October 2026, and the model has had one since 2026-10-04.

A line that names a model without a batch tier would not get the batch price, so the batch fails when it is created, before anything runs or is billed: errors names the model, the line numbers and the fixes, use auto, one of the batch-capable models, or the opt-in below. A discount on thousands of calls must never disappear silently.

Not in this release, inside a batch: streaming (stream: true is refused) and sessions. Each line is its own request: every auto line is classified and pays the fee for automatic model choice once, as any auto request does.

A batch line writes no prompt cache of its own unless the line asks for one. On a normal call Zabio’s automatic caching places cache markers on Anthropic models when a repeat is likely, so that each turn of a conversation reads the previous one back; inside a batch every line is one request and nothing reads its entry back, while the write itself costs more than fresh input (1.25 times the input price on the 5 minute cache, 2 times on the 1 hour cache, at the batch rates). So a batch line goes to the vendor uncached, and Zabio’s automatic caching in a batch is the shared prefix below. When several lines of one batch share a long prefix on an Anthropic model (the same system prompt, tools and opening messages, byte for byte, with a different question at the end of each line), Zabio caches the shared part for you: the first such line goes to the vendor alone and writes the prefix to the 1 hour cache, and the rest go once it has finished, each reading the prefix at the cache rate instead of paying for it as fresh input. Nothing is written for a line’s own question. This happens from three lines sharing at least 1,024 tokens (more on models whose minimum cacheable prefix is higher, 4,096 on claude-haiku-4-5) and costs the batch a few minutes at the start. Measured on real batches: every line after the first read the prefix, so a batch of 1,000 lines sharing 30,000 tokens on claude-sonnet-5 pays for the shared part about one write plus 999 reads, roughly a tenth of the uncached price. A line that sets zabio.cache.ttl itself keeps its own choice and is left out of the group. The output line’s zabio.cache shows what happened on each line, and the fee rule is unchanged: Zabio never takes a cut of the cache discount.

Lines Zabio answers directly

Most lines travel to a vendor’s batch API and come back within the window. Three kinds of lines are served by Zabio itself instead, and each one says so in its zabio.tier: requested is always batch inside a batch, applied is what served the line, and fallback says why they differ.

Classification lines answered by Jev. A line under auto whose response_format is a json_schema of fixed choices, booleans and bounded scores is a classification job, the kind Jev answers on auto. Inside a batch Zabio asks Jev at planning time when Jev’s estimated price is below the batch price of the text model routing picked; a kept answer ends the line at once, without waiting for a vendor batch. The line is billed like any auto call: Jev’s price plus the service fee and the fee for automatic model choice, with the saving measured against your key’s baseline model at its batch price. Its zabio.tier is {"requested": "batch", "applied": "standard", "fallback": "decision"}, model_served is jev-1.13.0 and zabio.decision carries the probabilities. When Jev’s answer is not sure enough, or Jev fails, the line goes to the vendor batch as any other and the Jev call is our cost.

Smart Splitting. A line under auto that carries a big source and a question gets the same treatment as a normal call: Zabio asks Jev whether the request can be distilled, prices both paths using the batch price of the answer, extracts the relevant lines with a cheap model at the standard price, checks the excerpt with Jev, and then batches the answer on the excerpt at the batch price. The output line’s zabio.split lists every step (questions, extract, check, answer) with its model and cost; the extract is billed at provider cost and the request pays one fee. If the model answers with the reserved line asking for the full source, the excerpt answer and the extract are our cost, nothing is billed for them, and the line runs again on the whole source in a second round when at least two hours remain before the window ends; otherwise it ends in the error file with batch_expired. zabio.split: "off" in a line’s body turns Smart Splitting off for that line, as on a normal call.

Lines at the standard price, when you ask for it. Set zabio.allow_standard_price: true on the batch and a line that names a model without a batch tier no longer fails the batch: Zabio runs it as a normal synchronous call from the batch worker at the standard price, with the same-vendor fallback and retries any call gets, and the line says {"requested": "batch", "applied": "standard", "fallback": "no_batch_tier"}. It is always your choice, never the default. A line naming deepseek-flash or deepseek-v4-pro under this opt-in waits for DeepSeek’s off-peak hours, at most four hours and always inside the 24-hour window, and then runs at the off-peak price, half of the list price, which DeepSeek applies on its own; if the wait would not fit before the window ends, the line runs at once. Lines at the standard price run a few at a time from the worker, so a batch of thousands of them takes longer than a vendor batch of the same size; the 24-hour window still holds.

Money

When a batch is accepted, Zabio estimates its cost and sets that amount aside on your balance: the hold. The estimate uses each line’s input, its max_tokens (or the default) and the price of the model it will run on; after routing it becomes the planned cost. Other calls cannot spend what a running batch needs. If the estimate exceeds your available balance, the batch fails at creation with insufficient_credits and the estimate, so you can add credits or send a smaller batch; nothing is billed.

Each line is billed when its result arrives, at the vendor’s batch price for the tokens it used, plus the usual fees: the service fee and, on an auto line, the fee for automatic model choice (see Billing). The saving is measured against your key’s baseline model at its batch price (or its standard price when that model has no batch tier), because that is what you would have paid there: we never take a cut of the vendor’s discount. As lines are billed the hold shrinks, and whatever is left is released when the batch ends. The bill is the real tokens; the hold was only the estimate. Lines Zabio answers directly follow the same rule: a line at the standard price is estimated at the standard price, and a line Jev answered or that Smart Splitting handled gives its share of the hold back when it ends. A running batch is never stopped for money: if the real cost runs past the estimate, the lines are still answered and billed, and your balance may end slightly below zero, like a single call can. A worked example is in Billing.

Limits

Limit	Value
File size	100 MiB
Lines per batch	50,000
Open batches per organization	20 at once
Line size	10 MiB
Endpoint	/v1/chat/completions only
Window	24h only

Lines are validated when the file is uploaded (JSON, a unique custom_id, method POST, a url, a body object) and again when the batch is created (the request fields, the model, the key’s rules).

Cancel

POST /v1/batches/{id}/cancel cancels the lines not yet sent to a vendor at once and asks the vendors to stop the rest. What they finish before stopping is kept, billed and written to the output file; the batch ends cancelled. A batch that has already ended answers batch_not_cancellable.

Python

client.batches.cancel(b.id)

Which key sees what

A key that belongs to a project lists, reads, cancels and deletes only the files and batches made with that project’s keys, and starts a batch only from one of those files; a key at the organization’s level reaches the whole organization. A file or batch outside answers 404 not_found, as if it did not exist. See Organizations, members and keys.

What is kept, and for how long

The input file, the output file and the error file are stored encrypted per organization, like stored conversations, for your organization’s retention period (30 days by default, 1 to 365), counted from the upload for inputs and from the end of the batch for results, or until you delete them with DELETE /v1/files/{id}, or until the expiry you set on upload with expires_after, whichever comes first. A file a running batch still needs is never deleted; deleting it answers file_in_use until the batch ends. Zabio keeps no copy of the requests or results outside those files, and deletes the copies the vendor holds as soon as the results are collected. Each line goes to the vendor it is routed to under that vendor’s API terms, like a normal request. See Privacy.

Batch objects and their counts stay listed on GET /v1/batches after the files are gone.

Errors

Status	code	When
400	invalid_file	the uploaded file is not valid batch JSONL; the message names the line
400	invalid_value	endpoint is not /v1/chat/completions, completion_window is not 24h, metadata is too big, or the file is not a processed upload with purpose batch
400	batch_too_large	the file has more than 50,000 lines
400	batch_not_cancellable	the batch has already ended
404	not_found	no file or batch with that id
409	file_in_use	a running batch still needs the file you tried to delete
413	file_too_large	the upload is over 100 MiB
429	too_many_open_batches	your organization has 20 batches open

A batch can also fail after creation, with the reason in its errors list and nothing billed:

code	When
model_not_batchable	a line names a model without a batch tier and the batch did not set zabio.allow_standard_price (the message lists the batch-capable models and the opt-in)
model_not_found	a line names a model that does not exist or the key cannot use
no_batch_model	no batch-capable model is enabled for your organization
insufficient_credits	the estimate exceeds your available balance; the message carries both numbers
invalid_url, invalid_request, missing_messages, invalid_json and the other request codes	a line is not a valid chat completion request
api_key_revoked	the key that created the batch was revoked
empty_file, input_file_missing	the file has no lines, or was deleted before the batch ran

And a single line can end in the error file with one of these codes:

code	When
batch_expired	the vendor could not answer the line before the window ended
batch_cancelled	you cancelled before the line ran
vendor_rejected	the vendor refused the request (a bad parameter for that model, for example)
vendor_batch_failed	the vendor could not run the group the line was in before the window ended
no_eligible_model	no batch-capable model could take the line under the key’s rules
byok_key_rejected	a line at the standard price ran on your organization’s own vendor key and the vendor rejected that key
transient_rate_limit, transient_5xx, network_failure, persistent_billing, schema_violation	the vendor answered the line with that error class and the window ended before a retry