The L40S for inference: when 48 GB is enough

What fits in 48 GB

The L40S is an Ada Lovelace card with 48 GB of memory, and for inference it is enough when your model's weights plus its cache stay under that line with room to spare. By arithmetic, a 7 billion parameter model in 16-bit needs about 14 GB and a 13 billion one about 26 GB, both comfortable. A 34 billion parameter model fits in 8-bit at about 34 GB. A 70 billion parameter model fits only at 4-bit, about 35 GB, leaving little for the cache.

Whether the L40S is good for your inference therefore starts with one question: how much memory does your traffic need on top of the weights? Short prompts and a modest number of users leave the card room. Long contexts and heavy concurrency fill it quickly.

This site holds no speed figures for the L40S or any other card, so nothing here says how quickly it serves a given model. What follows is memory arithmetic, the listed rates, and the test that settles the rest.

Graphics memory, not stacked memory

Data-centre cards such as the H100 and the H200 use stacked high-bandwidth memory. The L40S uses graphics memory, the same family as workstation cards, and sits in an ordinary PCIe slot. It is a different purchase from an H100, not a cheap version of the same thing.

This site holds no bandwidth or speed figures, so it cannot tell you how much the memory type matters for your model. Rent one, serve your model at your usual load, and measure tokens per second against what you need.

Ask each seller whether a single card can be rented by the hour rather than only as part of a multi-GPU machine. A single card keeps a first test small, and the answer varies by seller.

Three serving cases, by arithmetic

An internal assistant on an 8 billion parameter model in 16-bit needs about 16 GB for weights. On a 48 GB card that leaves around 32 GB for the cache, which is generous for a team of users with ordinary prompts. This is the case where the L40S is most clearly enough.

A retrieval pipeline that stuffs long documents into a 13 billion parameter model in 16-bit starts at about 26 GB and has around 22 GB left. Long contexts eat that quickly, so the test is how many simultaneous requests at your longest context fit before the engine starts queueing or refusing.

A 70 billion parameter model at 4-bit, about 35 GB, leaves around 13 GB. It will load. Whether it serves your traffic is uncertain, and whether 4-bit quality is acceptable for your task is a separate question for your own evaluation set.

The rate, and the cards next to it

For the L40S 48GB, the cheapest hourly listing on record is $0.38 (seller: Lium) and the dearest is $3.51; sellers with a listed rate: 20. Every seller is on the L40S page with the date each rate was read.

The L40 and the RTX 6000 Ada also carry 48 GB. Their cheapest hourly listings on record: L40, $0.45; RTX 6000 Ada, $0.60. For a memory-bound inference job these are the cards to price against the L40S, and the right one is whichever holds your model at the lowest rate you can actually get, after a test shows it serves your load.

Running the test on a rented card

Rent one L40S for an afternoon. Load the model in the precision you intend to serve, start the serving engine with the context length you need, and read the memory it reserves for the cache. Many engines report how many tokens of cache they could allocate; divide that by your typical request length to see how many requests fit at once.

Then send real traffic. Replay a sample of production prompts, or generate a load that matches your busiest hour, and watch latency as concurrency climbs. The point where latency turns up sharply, or the engine starts queueing requests, is the practical capacity of one card for your workload.

Repeat on whichever 48 GB card is next cheapest. If both hold the model and both serve the load, the rate decides. If neither serves it, the answer is a bigger card or more of them, and the test has told you which without a long commitment.

Signs you have outgrown it

A peak on test close to 48 GB means the card is the wrong size: one long request will push it over. If you are running a 70 billion parameter model at 4-bit only because 8-bit will not fit, the 80 GB tier is the next thing to price, and the GPU rate table lists what is available there.

Splitting a model across several L40S cards is possible, but these are PCIe cards without the board-level links an SXM node has, so the traffic between them travels a slower path. Ask the provider how the cards in a multi-GPU instance are connected before planning on it.

Starting on 48 GB, learning what your traffic needs and then moving up costs a few days of rent at the 48 GB rate. Committing to an 80 GB card first costs its higher rate for every hour, whether or not the load arrives.