H100 or H200 for inference: what the memory difference buys

The difference is 61 GB of memory

The H100 SXM has 80 GB of memory and the H200 has 141 GB. Both are Hopper parts, so the question for inference is mostly whether those extra 61 GB let you do something the H100 cannot: hold a larger model on one card, serve longer contexts, or run more requests at once. If your model and its cache already fit comfortably in 80 GB, the extra memory is capacity you pay for and leave idle.

That framing is deliberate. This site holds no throughput or latency figures for either card, and any claim that one serves a given model faster belongs to a benchmark you can read or a test you can run. What can be said from the record is how much each card holds and what each is listed at.

Both cards are the same generation, which keeps a side-by-side test simple: rent each for an hour, run the same container and script, and compare peak memory and your own timings alongside the seller and the rate.

Models that change card at 80 GB

A 70 billion parameter model in 8-bit needs about 70 GB for its weights. On an H100 that leaves around 10 GB for the key-value cache, the activations and the serving engine. On an H200 it leaves around 71 GB. For a chatbot with short turns the H100 may be enough. For long documents or many simultaneous users, the cache is what grows, and the H200 has room for it.

At 16-bit the same model needs about 140 GB, which is past one H100 and very nearly the whole of one H200. Neither card serves it alone with headroom; the usual answer is to split it across 2 GPUs, and then the comparison becomes a pair of H100s against a pair of H200s, with very different amounts of spare memory.

Smaller models make the choice easy in the other direction. An 8 billion parameter model in 16-bit is about 16 GB, and an H100 holds it with plenty to spare. Renting an H200 for it buys memory the workload never touches.

Reading the two rates side by side

For the H100 SXM 80GB, the cheapest hourly listing on record is $1.68 (seller: Vast.ai; sellers with a listed rate: 13). For the H200 141GB it is $2.09 (seller: Beam; sellers with a listed rate: 26). Both cards have their own pages, the H100 SXM page and the H200 page, with every seller and the date each rate was read.

A useful way to read the pair is cost per gigabyte of memory per hour. Divide each rate by the card's memory and you see what the extra capacity costs on today's listings. That number says nothing about speed, but for a memory-bound decision it is the right first comparison.

Look at the highest rates too. The gap between the cheapest and dearest seller of one card can be larger than the gap between the H100 and the H200, so a choice made on the card alone can be undone by the choice of seller.

Where one H200 replaces two H100s

The case that most often favours the H200 is a model that needs a little more than 80 GB. Splitting it across 2 H100s means two rates, a tensor-parallel setup in the serving engine and traffic between the cards. Holding it on one H200 avoids all of that. Whether the single card is cheaper depends on the two rates you can get, which is a sum worth doing before testing anything.

The opposite case is a model that fits in 80 GB with room left over and traffic that never stretches the cache. There, the H100 is the smaller bill for the same job, and the rental cost calculator will show the difference over a month of hours.

Long contexts are where the cache decides

For a fixed model, the cache is the part of memory that moves. Every token held in context for every active request takes space, and the amount per token is set by the model's layer count and attention layout. Double the context and the cache roughly doubles; double the concurrent requests and it doubles again.

So a retrieval system feeding whole documents into a model can outgrow an H100 when a chat assistant on the same model does not. The weights are identical. The difference is how much context sits in memory at once, and on an 80 GB card with a 70 GB model there is not much room for it.

Some serving engines can store the cache in 8-bit, which roughly halves it at some cost the engine documents. That can keep a workload on an H100 that would otherwise need an H200. Test it on your prompts before counting on it.

What to test before committing

Rent each card for a few hours with your actual model, your actual context lengths and a realistic number of concurrent requests. Record peak memory, tokens per second at your target latency and how the numbers change as the load rises. No table, including ours, holds those readings for your workload.

Ask the provider whether the H200 they list is the SXM part on a board with fast GPU-to-GPU links, and whether a single card or only a full node is rentable. Ask what the minimum term is. A cheap rate on a card only sold as an eight-GPU node is not the same offer as a single card by the hour.

Check the region as well. A rate listed for one region does not always apply in another, and data that has to cross regions to reach the card adds a transfer bill the hourly rate never shows.