Batch pricing: when it fits and when it does not

Batch fits any call that can wait

Batch pricing fits LLM calls whose answer nobody needs right away. You submit many requests together, the seller runs them when it has spare capacity and returns the results within a stated window, and in exchange the per-token rate is lower than the standard one. It does not fit a chat reply, a request a user is watching, or a step another step is blocked on.

Not every seller offers it. Listings with a batch tier: 121, out of input-price listings on record: 750. They appear on the token price table as their own rows beside the standard rate for the same model.

The trade is time for money. The lower rate pays for the seller's freedom to schedule your requests within the window, so the answer can take hours and the window is a maximum rather than a target.

Sorting calls by who is waiting

Go through your traffic and ask, for each kind of call, who is waiting on the answer. A person waiting means it is interactive and stays on the standard tier. A scheduled job waiting means asking whether the job could start later. Nobody waiting means it belongs in a batch.

Nightly jobs are the clearest fit: tagging the day's support tickets, summarising call transcripts, writing product descriptions for a catalogue update, re-embedding or re-labelling a dataset. Evaluation runs fit too, since a test suite of thousands of prompts is rarely needed within minutes.

Backfills are the case people miss. When a new feature needs a summary for every old record, that is a large one-off job with no deadline measured in seconds, and its size makes the batch rate count.

Where batch goes wrong

The window is a maximum, not a promise of speed. A job submitted at the end of the working day may come back the next morning. Where a downstream report or a customer email depends on it, the delay moves into somebody else's schedule.

Failures also arrive late. A malformed prompt in an interactive call fails in seconds and gets fixed. The same mistake in a large batch is found hours later, after every request has been billed. Run a small sample on the standard tier first and read the outputs before sending the full set.

Agents and multi-step chains rarely fit, because each step needs the previous answer. Batching breaks the loop into one step per window.

Mixed pipelines can still use it in part. A product that drafts replies live but scores them for quality afterwards can send the scoring to a batch tier and keep the drafting interactive, splitting one feature across both rates.

Pricing the split

Divide the month's tokens into an interactive part and a deferrable part. The interactive part is priced at the standard input and output rates. The deferrable part is priced at the batch rates, where the seller publishes them. The month is the sum.

A product that classifies incoming documents for a dashboard refreshed each morning has almost no interactive share. Moving all of it to batch can change the bill as much as a change of model, provided the seller of the model offers a batch tier. Where none is offered, the comparison is between that model on standard and another model that can be batched.

The LLM cost calculator has a serving tier filter with standard, batch and free. Select batch to see only the batch listings for the models you picked, then compare the monthly figure with the standard view at the same token totals.

Checking that the work really can wait

Ask the people who consume the output, not the people who produce it. An engineer will say a report can wait until morning. The account manager who sends that report at nine may disagree, and the batch window decides who is right.

Then check limits. Batch endpoints often have their own caps on file size and requests per job, separate from the interactive limits. Those caps are set by each seller and are not something this site records, so read them on the seller's documentation before planning a large run.

Write the decision down for each job: which tier it runs on, who consumes the output and by when. When a deadline changes later, the note shows which jobs need to come back to the standard tier, and nobody has to rediscover the reasoning from the code.

A sample job before the real one

The first batch run on any new prompt should be small. A few dozen requests show whether the output format parses, whether the instructions are followed and whether the token counts per request look like the estimate. A mistake caught here costs a few dozen requests instead of the whole job.

Then scale in steps. A first full-size job tells you how long the seller's queue actually took, which is the figure that decides whether the work can move into the overnight window or needs to start earlier in the day.

Keep the standard tier as a fallback in the code. When a batch job fails or comes back late on a day the output is needed, the same requests can be resent interactively at the standard rate. That costs more on that day and protects everyone downstream from a missed deadline.