Serverless or dedicated inference

The short answer is utilisation

Serverless inference charges for the calls you make, per token or per second of compute, and costs nothing while idle. Dedicated inference rents a GPU by the hour and charges for every hour whether a request arrives or not. Uneven or small traffic usually costs less on serverless. A GPU that would be busy most of the day usually costs less dedicated. The figure that decides it is how much of the rented hour you would actually use.

Everything else in the choice, from cold starts to model choice to who gets paged, follows from that one number, so estimate it before anyone opens a provider console.

How to estimate utilisation before you have traffic

Utilisation here means the share of rented hours in which the card is doing useful work at a batch size worth having. It is not the GPU utilisation figure a monitoring tool prints, which can read high while a single request trickles through. The question is simpler: across a week, how many hours would this card be serving enough requests to justify being on?

Draw the week. Mark the hours your users are awake, the hours a scheduled job runs, and the hours nothing happens. A consumer product with users across many timezones fills more of the week than a business tool used in one. A product launched last month fills almost none of it with confidence, because the traffic forecast is a guess.

A week that is more than half dark leaves a dedicated card spending most of its bill waiting. That usually settles it for serverless until the picture changes, and the week can be redrawn each quarter.

What you are paying for in each model

With a serverless endpoint you buy output. The provider owns the GPUs, packs many customers onto them, and sells you a slice measured in tokens or seconds. The price includes their idle time, their spare capacity for your spikes, and their margin. You never see the card, and you never pay for it standing still.

With a dedicated deployment you buy time on a card, and the work you get out of that time is your problem. The lowest listed on-demand rate for an L40S 48GB, run for a month of 730 hours, is $277.40. A rental bill arrives at the same size in a quiet week and a busy one, and the only lever on cost per request is to send it more requests.

Token prices for serverless APIs are on the LLM prices page (providers on record: 16), and hourly GPU rates are on the GPU rates page. They are different units, which is why the comparison needs a utilisation guess in the middle.

Traffic shapes that point to serverless

A product in its first months, where nobody knows whether next week brings ten requests or ten thousand, fits serverless. So does an internal tool used during office hours in one timezone, which leaves a dedicated GPU dark for most of every day and all of the weekend.

Spiky workloads also fit, with a caveat. A batch of documents processed once a night is cheap on a per-token bill because nothing is paid for between runs. The caveat is that serverless platforms impose rate limits and concurrency caps, and a large burst may queue on their side instead of yours.

Serverless also suits a team that does not want to run anything. There is no image to build, no driver to pin, and no replica count to reason about at three in the morning. The provider still has bad days, but their bad day is a status page you read, not an incident you lead.

Traffic shapes that point to dedicated

Steady, high volume is the clearest case. A feature that serves requests around the clock keeps a card busy, and a busy card spreads its fixed hourly cost across many requests. At some volume the per-request figure drops under what a token price would charge for the same work.

Control is the other reason. A dedicated deployment lets you pick the model weights, the precision, the serving engine and the batching policy. You can run a fine-tuned model a serverless catalogue does not carry, and keep data inside a network boundary you chose.

The cost of that control is the operating work: capacity planning, upgrades, monitoring, and owning the outage. Teams often undercount this, and it is the part no rate card shows.

A useful test is to name the person who would be woken when the dedicated endpoint fails at night. When nobody comes to mind, the team is not ready to run one yet, whatever the arithmetic says.

A decision rule you can run this week

Take last month of real or projected traffic and count tokens in and out. Price that volume on a serverless API with the LLM cost calculator. Then price one always-on card that could serve the same model with the GPU cost calculator. A token bill well under the rental month says stay serverless. One well over it says a dedicated card deserves a trial.

When the two land close together, stay serverless and revisit when traffic grows. Close numbers mean the rental would carry idle hours, which no amount of tuning recovers.

The inference layer lists live products across APIs, engines, gateways and edge runtimes (the count on record is 186), and it is the place to see which category a vendor sits in before comparing their prices.