Using an RTX 4090 for LLM work: where it stops

Where 24 GB runs out

You can run an LLM on an RTX 4090 as long as the weights and the cache fit in its 24 GB. A 7 billion parameter model in 16-bit needs about 14 GB, which fits with room for a working context. A 13 billion parameter model needs about 26 GB in 16-bit, which does not, but fits at 8-bit in about 13 GB. A 32 billion parameter model at 4-bit is about 16 GB. A 70 billion parameter model at 4-bit is about 35 GB, and that is where a single 4090 stops.

Those are weights only. A long context or several users at once adds cache on top, and on a card this size the cache is usually the thing that fails first, often with an out-of-memory error halfway through a long prompt rather than at load.

This site holds no benchmarks, so the speed of a 4090 on any model is not something it can tell you. The memory arithmetic is certain, and it is enough to know where the card stops being an option at all.

What people actually use it for

Prototyping is the obvious fit. Trying a prompt format, wiring a small model into an app, running an evaluation set against a 7 or 8 billion parameter model: all of that fits, and none of it needs a data-centre card. Small fine-tunes with adapters on a quantised base model fit too, as long as the sequences stay short.

Many image generation, speech and embedding models are small enough to sit inside 24 GB; the same parameter-count arithmetic tells you whether yours is. For a small project the 4090 can be the whole stack, and a larger card only makes sense once the model or the traffic grows past it.

Where it falls short is anything that needs the model in higher precision at a size above about 10 billion parameters, anything with very long contexts, and anything serving real concurrent traffic.

Rates for consumer cards

Cheapest hourly listing on record for the RTX 4090 24GB: $0.12 (seller: Vast.ai; sellers with a listed rate: 14). For the newer RTX 5090 with 32 GB: $0.25. For the older RTX 3090, also 24 GB: $0.09. The RTX 4090 page has every seller and the date each rate was read.

Consumer cards may be rented through a marketplace, where the machine belongs to someone else and the terms vary by host. Read what a listing says about whether the instance can be reclaimed, how storage is handled and what the network allows before putting anything you care about on it.

What you give up against a data-centre card

The 4090 uses graphics memory rather than stacked memory, and it was built for a desktop. It sits in a PCIe slot with nothing like the board-level links between GPUs that an SXM server has. Splitting a model across several of them works in software, but the traffic between cards takes a slower path, and how much that costs your workload is something to test, not assume.

Expect fewer guarantees. A consumer card in a rented machine may come with a home-grade network, a host that can take it back, and no promise about what else runs on the box. For a prototype that is fine. For a service other people depend on, those terms matter more than the rate.

Data-centre features such as error-correcting memory and remote management are things to ask about rather than assume on a consumer card. Whether they matter depends on how long your jobs run and what a silent fault would cost you.

Getting more out of 24 GB

Quantisation is the first lever. Moving a 13 billion parameter model from 16-bit to 8-bit halves its weights to about 13 GB, and 4-bit takes it to about 6.5 GB, leaving far more room for the cache. The quality cost depends on the model and method and should be checked on your own examples.

Shorter contexts are the second. If the application can trim its prompts, summarise history or retrieve fewer chunks, the cache shrinks in proportion. Some engines can also store the cache in 8-bit, which roughly halves it.

Offloading part of the model to system memory lets a larger model load, but every token then waits on data crossing from the host. It works for experiments. For anything interactive, a card with enough memory is the plainer fix: rent one for the hours you need.

The next step up

When the model or the cache outgrows 24 GB, the RTX 5090 adds 8 GB, and the 48 GB cards such as the L40S, the L40 and the RTX 6000 Ada double the room. The L40S post works through what 48 GB holds by the same arithmetic.

Price the step before taking it. The rental cost calculator shows a month of hours on any of these cards, and part-time use changes the answer a lot: a 4090 rented only while you work can cost less than a larger card left running all month, even when the larger card has the lower rate per gigabyte.

Keep the model files and environment portable while you are on the 4090. A container image and a script that loads the model at a set precision make the move to a larger card an afternoon of work instead of a rebuild.