Release notes that say what changed and what to retest

A buyer reads release notes as a test plan

Good model release notes tell a buyer exactly which of their own tests to run again. That means listing every change by kind (weights and behaviour, prompt or chat format, tokenizer, default settings, limits, price) and, for each, what in a buyer's system it can break. A note that says improved reasoning and better instruction following gives a buyer nothing to check and leaves them to rerun everything or nothing, and choosing nothing means finding out in production.

The model maker is the only party who knows what changed. Everyone downstream is guessing from outputs, slowly and at their own cost.

Behaviour changes need examples, not adjectives

New weights change behaviour, and the change is rarely uniform. Answers get longer or shorter. Refusals move. A model that used to return bare JSON starts wrapping it in prose. A buyer's parser breaks on Tuesday, and nothing in the notes mentioned format.

Write the behaviour changes you observed in your own evaluation, with the direction of each: replies are longer on average, the model refuses fewer medical questions, it now asks clarifying questions where it used to guess. You do not have to publish your evaluation set to say which way things moved.

Then say what a buyer should rerun. Their evaluation suite, if they have one. Any parser that reads model output. Any prompt that relied on a quirk, which a prompt tuned over months may well do.

Prompt format and tokenizer changes are silent breakers

A change to the chat template, the system prompt handling, the special tokens or the tool-call format can leave a model working perfectly through your own API and producing nonsense through a host that still applies the old template. It is an expensive change to leave out of release notes, because the failure shows up at a third party and gets blamed on them.

A tokenizer change matters for cost as well as behaviour. The same text can become a different number of tokens, so the same workload bills differently at the same rate per million. A buyer who estimated spend in the LLM cost calculator last quarter needs to know their token counts moved. Say whether the tokenizer changed, and if it did, roughly which way counts go for English, code and other languages you tested.

Special tokens and stop sequences deserve their own line. A change to the token that ends a turn can make a model run on past where it used to stop, which shows up as longer replies and higher bills before anyone finds the cause.

Limits, defaults and price belong in the same note

Context window, maximum output, rate limits and default sampling settings all change between versions, and each one changes results without any change to the buyer's code. A new default temperature alone can make an old prompt look worse. Put the before and after values in a small table, in the same units as the context and modality facts on the model page.

Rate limits deserve the same treatment. A new version launched with lower limits than the one it replaces can break a batch job that ran fine for months, and the buyer will look for the cause in their own code first.

Price is part of the release too. If the new version is billed differently, put the rates in the note, or link the dated pricing page. Sellers update their pages on their own schedule, and the token price table shows rates from 16 providers as each is read, so for a short while buyers may see the old and new versions at different prices in different places.

A shape that works for every release

Use the same headings every time, in the same order: identifier and date, what changed in behaviour, what changed in format, tokenizer, limits and defaults, price, what is deprecated, what to retest. Write none where nothing changed. A heading with none under it is information, because it tells a buyer they can skip that test.

Keep old notes on the same page or linked from it. Anyone pinned to a dated version needs the history to decide when to move, and a release-note page that only shows the latest entry forces them to reconstruct it. Retirement dates belong in the same history, and the identifier in each entry should match what the API accepts, character for character.

Put the notes where the model page links to them, and give each entry its own anchor, so a buyer can send a colleague straight to the change that matters to them.

Dated identifiers let buyers choose when to move

Release notes only help if a buyer can stay on the old version long enough to run the tests. Publish a dated or versioned identifier for every release alongside any moving alias, and say how long each dated version will keep answering. A buyer pinned to a fixed identifier can read the notes, run the checks and move when ready.

A moving alias that silently points to new weights is the opposite. Buyers who used it because it was in the quickstart get the change without the notes, and the first sign is a complaint from their own customers. If you keep such an alias, say in every release note that it moved and when.