Benchmark claims: publish the method with the result

The result is the last line, not the first

A GPU benchmark claim is checkable only when the method is published beside the number: the exact hardware and how many cards, the driver, library and framework versions, the model and its precision, the batch size, the input and output sequence lengths, what was measured, how many runs, and the date. Without those, a buyer cannot reproduce it, compare it, or tell whether it applies to the workload they have.

Vendors publish bare numbers because the number fits on a slide. A buyer who tries to reproduce a bare number and cannot has no way to tell whether the number or the attempt was wrong.

The list is long because every item on it can move a result on its own. It is also short enough to fit in a paragraph under a chart, which is where most of it belongs.

Hardware, named the way a buyer would rent it

Name the card in full, with form factor and memory: H100 SXM 80GB, not H100. An SXM module on a board with a fast GPU-to-GPU link and a PCIe card in an ordinary slot are different purchases, and a multi-GPU result on one says little about the other. The H100 SXM page and the PCIe page are separate for that reason.

Give the count, the node layout and the interconnect between nodes if there is more than one. Then the CPU, the system memory and the storage the data was read from, because a data-loading bottleneck shows up as a slow GPU. If the system is one a buyer can rent from you, say which instance type it is.

Memory belongs in the hardware line too, since it decides what fits. A 70 billion parameter model in 16-bit needs about 140 GB for its weights alone, before the key-value cache, so a result on that model came from more than one 80 GB card or from a smaller precision. A reader who can do that arithmetic will check it, and a method that answers the question first looks like it has nothing to hide.

Software, configuration and the workload

List the driver version, the CUDA or ROCm version, the framework and the inference engine with exact version numbers. A benchmark rerun a few months later on newer software can move a long way, and without versions nobody can tell whether a difference comes from the hardware or the stack.

Name the model exactly, with its size and precision: 16-bit, 8-bit floating point, or a 4-bit quantisation with the method named. Hopper and later cards support 8-bit floating point arithmetic, so precision is a variable a result has to state. Give the batch size and the input and output sequence lengths too. Throughput at a large batch and latency at batch one are different measurements of different things.

Say what was measured and how. Tokens per second per what: per GPU, per node, per request? Latency at which percentile? Time to first token or time per output token? A number with no definition beside it cannot be compared with anything.

Runs, spread and what went wrong

State how many runs were made and how much they varied. A single run is an anecdote. Report the median and the spread, and say whether any runs were thrown away and why.

Then say what did not work. A configuration that failed, a model that would not fit, a setting that made things worse: nobody includes these for marketing reasons, which is why they lend weight to the rest of the page.

Publish the scripts if you can. A repository with the exact commands and configuration turns a claim into something a buyer can run on the instance they are about to rent.

Where the method goes on the page

Directly under the result, not in a linked PDF and not in a footnote. A reader who sees the number and has to click away to learn what it means will often quote the number and skip the click. If the method is long, put a short version under the chart, naming card, count, model, precision, batch size and sequence lengths in one sentence, and link the full version from there.

Charts need a table beside them with the same figures, so a reader can copy the values without guessing from the bar heights. Give the page a date and a version: benchmark pages get updated as software improves, and a reader should be able to tell which version of the claim they quoted.

Compare like with like or say plainly that you did not. A result on your newest card set beside a rival result from an older round, on different software, is two measurements, not a comparison, and a sentence saying so costs less than the correction a reader will otherwise post.

What this site holds, and what it does not

We do not run benchmarks and hold none of our own. The one set of performance results on the site is the MLPerf page, which shows MLCommons submissions as submitted, each with the accelerator count of the system that produced it and a link to its source. We do not divide a system result down to a per-GPU figure, because no submitter claims one.

What we hold is prices, read from provider pages with the source and date. A benchmark gives a buyer the other half of a cost question, and that half is only usable when the method is on the page with it. A claimed listing can add what only the vendor knows, with a source, through the vendor workspace.