A100 or H100: which to rent

Same memory, different generation

Rent the A100 when your model fits in its memory, your code runs in 16-bit or 32-bit, and the lower rate matters more than the newer architecture. Rent the H100 when you want 8-bit floating point arithmetic, which arrived with Hopper and which the Ampere-based A100 does not support, or when a test on your own workload shows the newer card finishing enough sooner to cost less per job. The 80 GB versions hold the same model; the difference is generation and price.

The A100 comes in 40 GB and 80 GB versions, and the H100 in 80 GB as SXM or PCIe. Compare like with like. An A100 40 GB against an H100 80 GB is a memory decision before it is anything else.

The 40 GB A100 holds a 13 billion parameter model in 16-bit at about 26 GB with room for a cache, but not a 34 billion parameter one in 8-bit with any comfort. If your model sits between those sizes, the 80 GB version is the one to price against the H100.

What FP8 changes, and when it does not

FP8 stores numbers in a single byte, so weights take half the memory they take in 16-bit and the card does the arithmetic in its 8-bit units. Serving engines and training frameworks that support it can use it for inference and, with care, for training. On an A100 that path is not available; 8-bit there means integer quantisation, which is a different method with its own trade-offs.

Without FP8 in your stack, a large part of the generational argument falls away for your job. Plenty of fine-tuning runs, embedding jobs and batch inference pipelines still run in bf16, and on those the question becomes how the A100 and the H100 compare on your code, which only a test answers. This site holds no benchmark for either.

The listed rates, card for card

Cheapest hourly listing on record for the A100 SXM 80GB: $0.41 (sellers with a listed rate: 9). For the H100 SXM 80GB: $1.68 (sellers with a listed rate: 13). Some sellers list the A100 without naming its form factor; those rates sit on the A100 page and should be checked before they are compared with an SXM rate.

These figures say nothing about stock. A rate on record means a seller published it on the date shown, not that a card is free today or that it can be rented singly rather than by the node.

The A100 SXM 80GB page and the H100 SXM page show every seller at both ends, with the date each figure was read. Read across both before deciding the older card is the cheap one: the rate a single seller charges for an A100 can sit above the rate another seller charges for an H100.

Cost per job, not cost per hour

A lower hourly rate is a lower bill only when the job takes the same number of hours. If the H100 finishes your run in fewer hours, the sum can go the other way. The comparison that matters is rate times hours for each card, and the hours have to come from your own test, because the ratio differs by model, precision, batch size and software version.

A cheap way to get the hours is to run a short, representative slice of the job on each card and time it. Multiply by the full job size, then by each rate. The rental cost calculator does the last step for any card and any number of hours.

For always-on serving where the card is mostly waiting for requests, hours are fixed by the calendar rather than the work, so the lower rate carries more weight.

Where each card tends to point

A fine-tune with adapters on a 13 billion parameter base in bf16 needs about 26 GB for the frozen weights plus a modest amount for the adapter and activations. Either 80 GB card holds it easily, nothing in it uses FP8, and the run is short. This is the kind of job where the lower rate usually decides, after a timed slice confirms the hours are close.

Batch embedding over a large corpus is similar. The model is small, the work runs for many hours, and what you want is the lowest cost per million documents. Time a fixed batch on each card, divide the rate by the documents per hour, and pick the smaller number.

Serving a 70 billion parameter model in 8-bit is where the H100 has an argument. On Hopper the 8-bit path is floating point and supported natively; on the A100 it means integer quantisation, which some engines support for some models. If your engine and model only offer an FP8 path, the A100 drops out of the comparison before any rate is read.

Questions to put to the provider

Ask which A100 it is: 40 GB or 80 GB, SXM or PCIe. Ask the same of the H100. Ask whether multi-GPU instances have a fast link between the cards, since that matters for any model split across them. Ask the driver and CUDA versions on the image, because older images can hold back newer frameworks on either card.

Then ask what happens at the end of the hour you pay for: the billing increment, whether storage on a stopped instance is billed, and whether the instance can be reclaimed. The generation of the card has no bearing on these terms, and each of them changes the bill.