What an MLPerf result does and does not tell a buyer

What one result is

An MLPerf result is a measurement of one whole system, with a stated number of accelerators, on one benchmark, in one scenario, reported in one unit, as submitted to MLCommons by whoever built or sold that system. It tells a buyer what that configuration achieved under those rules. It says nothing about price, about whether that system can be rented, or about how a different system with the same cards would do.

That is narrower than the way results are often quoted, and the gap between the two is where buyers get misled. For a vendor, the narrowness is also the reason a result is worth citing: it is specific, sourced and dated by its round.

MLPerf is run by MLCommons, and results are published in rounds. A result belongs to its round, so a figure quoted without the round could be from any year and any software generation, and a careful buyer will ask which.

Why we never show a per-GPU figure

Our MLPerf page shows every figure for the whole system that produced it, beside its accelerator count, and does not divide a system result down to a per-GPU number. No submitter claims one, and scaling does not support the arithmetic: a system with many accelerators is not the same as that many single-accelerator systems added up, which is a large part of what is being measured.

Buyers do the division anyway, because a per-card number is what they want for comparing rental quotes. A vendor quoting its own result should resist doing it for them. A result divided down and set beside a rival invites a correction from anyone who knows the rules, and the correction is what gets remembered.

Comparable only within a benchmark and scenario

The page groups results by benchmark, scenario and unit, because only within one of those groups are two figures measuring the same thing. An offline result and a server result for the same model answer different questions: how much work the system gets through when it can batch freely, against how it holds up under a stream of requests with a latency bound.

So a vendor page that quotes "our MLPerf result" without naming the benchmark, the scenario and the round has quoted a number with its meaning removed. Name all three, link the submission, and give the accelerator count in the same sentence.

The same care applies across rounds. Software improves between rounds, so a system submitted again later can post a different figure on the same hardware. Comparing your result from one round with a rival result from another tells a buyer about two software stacks as much as about two machines.

What the page leaves out, and why

Submissions marked as research hardware are left out. They cannot be bought, and listing them beside systems a buyer can order would mislead. Each round sits at its own address, and every row links to the source it came from, so a figure quoted from the page can be traced back to the submission itself.

The page holds no prices for the submitted systems and makes no claim about where they can be rented. Joining a result to a rental rate is the buyer's job, and it is harder than it looks, because the rented machine is rarely the submitted one.

What a buyer still has to find out

Whether the system they would rent matches the one submitted: the same card, the same count, the same interconnect, the same software stack. A provider renting the same card in a different chassis, with different networking or an older driver, is selling something related but not identical.

Whether their workload looks like the benchmark. A result on one model at one precision says little about a different model, a different context length or a fine-tuning job. And what it costs: every GPU rate we hold is per GPU-hour, read from the seller page, which is the other half of the question an MLPerf table cannot answer.

A vendor who answers those questions on its own page, next to the result it cites, saves the buyer the work and earns more trust than the result does on its own.

Citing your own result without overstating it

Give the round, the benchmark, the scenario, the unit, the system and the accelerator count in one sentence, and link the submission. A vendor that writes "we led MLPerf" with none of that has made a claim nobody can check, and a competitor with a result in a different scenario can make the same claim on the same day.

Say whether the system you submitted is the one a buyer can rent from you, and if not, how it differs. If you rent the same card in a different node, with different networking, say that the result describes the submitted system. Buyers find this out eventually, and it is better coming from you on the page than from them in a support ticket.

Do not turn a result into a price claim. A result says nothing about cost, and pairing it with your hourly rate to produce a cost per unit of work invents a figure the benchmark never measured. If you want to make a cost argument, publish the method and the arithmetic separately, so a buyer can redo both.