Weights size and the GPU it fits: publish the arithmetic

The sum a buyer needs, done for them

The memory a model's weights need is about the parameter count times the bytes per parameter: 2 bytes in 16-bit, 1 in 8-bit and half a byte in 4-bit. So a 7 billion parameter model needs about 14 GB in 16-bit and about 3.5 GB in 4-bit. A model page that prints this for each precision it supports, and names the smallest common card that holds it, has answered the question a buyer was about to search for. The cache and activations come on top and must be stated as such.

The arithmetic is not hard. It may feel too obvious to print, and it is not obvious to the person in procurement holding a GPU price list.

Each step of the sum is something a buyer could do alone. The value of doing it on the model page is that the maker knows the exact count and the exact precision of each published build, and the buyer otherwise has to guess both.

The worked figures for common sizes

At 7 billion parameters, the weights are about 14 GB in 16-bit, 7 GB in 8-bit and 3.5 GB in 4-bit. The 16-bit build fits a 24 GB card such as the L4 or the RTX 4090 with room left over, and the 4-bit build fits a 16 GB T4 comfortably.

At 13 billion, the figures are about 26 GB, 13 GB and 6.5 GB. The 16-bit build no longer fits any 24 GB card and needs a 48 GB card such as the L40S, whose lowest listed hourly rate is $0.38. In 8-bit it is back on a 24 GB card.

At 70 billion, they are about 140 GB, 70 GB and 35 GB. In 16-bit that is beyond an 80 GB H100 and, on paper, one gigabyte under the 141 GB of an H200, which leaves nothing for the cache and so does not work in practice. A 192 GB B200 or MI300X holds it with room, and the lowest listed hourly B200 rate is $3.74. In 4-bit the same model fits on a single 48 GB card.

The cache is the part no single figure covers

The weights are the floor. The key-value cache grows with the length of each request's context and with the number of requests served at once, and activations need room too. A model that fits on a card with a short prompt and one user can run out of memory with long documents and a full batch.

That is why the page should state the weights figure as weights, never as the memory the model needs. If you have measured the cache for your architecture at a given context and batch, publish that too, with the settings. If you have not, say the weights figure is a minimum and stop there rather than guess.

Context length is where the gap gets large. A model advertised with a very long window, run at that full length with many requests at once, can need more memory for the cache than for the weights. A page that states the window and the weights figure side by side, without that caveat, invites a buyer to rent a card that will not hold the work.

When the weights need more than one card

Weights larger than one card can be split across several in the same machine. The arithmetic still holds: 140 GB of 16-bit weights for a 70 billion parameter model fits across a pair of 80 GB H100 cards with about 20 GB left between them for the cache, which may or may not be enough depending on context and batch. Splitting adds traffic between cards at every step, which is where SXM modules on a board with a fast GPU-to-GPU link differ from PCIe cards in an ordinary slot.

A model page can say which splits it has been run on and with which serving engine. That is not a speed claim, only a statement that the configuration loads and answers, and it saves a buyer from renting a node to find out.

From a card to a monthly bill

Once a buyer knows the card, the price follows. The GPU rate table lists every on-demand rate we hold with the page it was read from, and the H200 page shows its lowest listed hourly rate as $2.09. The rental cost calculator turns an hourly rate into the cost of the hours a buyer expects to run.

A maker who prints the card beside each precision gives buyers a path from model page to invoice in two clicks. A maker who prints only the parameter count, as argued in putting the count on the model card, gives them the first step and leaves the rest to them.

How to lay it out on the page

One short table, as text: precision in the first column, weights in gigabytes in the second, smallest common card that holds the weights with room for a modest context in the third. A line under it saying that the cache and activations need more and grow with context and batch. If the model routes tokens through only part of its weights, use the total count, because all of it has to be loaded.

Use the card names buyers search for, with their memory: L4 24 GB, L40S 48 GB, H100 80 GB, H200 141 GB, B200 192 GB. A buyer can match those against a rate table directly. Specs beyond memory belong to the card's maker, not the model's.

Round consistently and say which way. A weights figure rounded down to make a model appear to fit a card is the kind of error that costs a buyer a day, and printing the unrounded sum next to the card avoids it.