API or self-host: where the break-even sits
Self-hosting beats an API when your monthly token bill exceeds the rental month of the GPUs needed to serve the same traffic, plus the cost of running them.
The formula, in words
Self-hosting a model pays off when the monthly token bill for the same traffic on an API is larger than the cost of the GPUs that would serve it, run for a month, plus the cost of the people who keep them running. Count the cards the model needs, multiply each hourly rate by the hours they must be up, add operations, and compare. Below that line the API is cheaper. Above it, self-hosting starts to earn its keep.
The rest of this post is about getting each side of that comparison honest, because both sides are easy to get wrong in a direction that flatters the answer you wanted.
The rental side: cards, hours and the always-on month
Start with memory. A model needs about its parameter count times the bytes per parameter just for weights: 2 bytes in 16-bit, 1 byte in 8-bit. A 7 billion parameter model in 16-bit is about 14 GB of weights and fits on an L4 24GB with room left for cache. A 34 billion parameter model in 16-bit is about 68 GB, which fits on one 80 GB card but leaves little room for the cache.
Then hours. A production endpoint that must answer at any time runs 730 hours a month. On that basis, the lowest listed rate per card for a month is $240.90 for the L4 and $1,226.40 for the H100 SXM 80GB. Multiply by the number of cards, and by the replicas you keep for failover.
The GPU cost calculator does this arithmetic from published rates for any card and hour count. Run it with the hours you will really pay for, not the hours you hope to use.
The token side: what the API would charge
Take a month of traffic and split it into input and output tokens, because providers price them separately and output is commonly the dearer of the two. Long prompts with short answers and short prompts with long answers can produce very different bills for the same request count. Retrieval pipelines are the usual surprise here, since they stuff retrieved passages into every prompt and the input side grows quietly.
Price that volume against a model of similar size and ability. Token prices by model are on the LLM prices page (models on record: 467), and the LLM cost calculator turns a token count into a monthly figure. Where a provider offers a cached input rate or a batch tier and your traffic qualifies, price with it, since that is the rate you would really pay.
Running the comparison once, step by step
Suppose a support assistant built on a 7 billion parameter model. In 16-bit its weights are about 14 GB, so one L4 24GB holds them with some memory left for the cache. The product must answer at any hour, so the card runs all 730 hours, and you want a second card so one failure does not take the feature down. The rental side is the L4 month, counted once for each card, before operations. At the lowest listed rate, one L4 month is $240.90.
Now the token side. Pull a month of logs, or a fortnight doubled, and total the input and output tokens. Enter them against a hosted model of similar size in the LLM cost calculator, which gives the figure the rental has to beat.
Then ask the capacity question honestly. Can that pair of L4s serve your peak hour at a latency you accept? We cannot answer that for you, because we hold no throughput figures. A day of load testing on a rented card can, and it costs a day of rental.
When the answer is no and the peak needs a second pair, the rental side doubles while the token side stays where it was. This is the step that most often reverses a spreadsheet that looked like a clear win for self-hosting.
What the comparison leaves out
Operations is the largest omission. Someone has to build the serving image, pin drivers, watch memory, handle a failed node and upgrade the model. A fraction of an engineer is still a salary, and it belongs on the rental side of the ledger.
Capacity is the second. A self-hosted endpoint serves only as many concurrent requests as its cards hold, and peak traffic decides how many cards you need. A bill sized for the average request rate will queue at the peak, so size for the peak and accept that the average hour carries spare capacity.
Quality is the third and the one this site cannot help with. A self-hosted model that is cheaper per token but worse at your task costs you in a different currency. Test it on your own prompts before the spreadsheet decides.
Then there is the exit. An API can be swapped for another with a change of endpoint and a round of testing. A self-hosted stack is images, scripts, dashboards and habits, and leaving it takes longer than building it did.
How to read the result
A token bill that is a small fraction of the rental month ends the exercise: the API is cheaper and stays cheaper until traffic grows a lot. A token bill of several rental months justifies a proper self-hosting trial, priced at peak capacity with operations included.
The uncomfortable middle is where most teams sit. There, the deciding factors are not price: data that cannot leave your network, a fine-tuned model no API serves, or a latency target you need to control. Those can justify self-hosting at a loss. Say so plainly in the decision rather than adjusting the arithmetic until it agrees.
For the serving choice that sits just before this one, see serverless or dedicated inference, which sets the utilisation question up in more detail.
What we could not answer
- ⚑We hold no throughput measurements, so this cannot say how many requests one card serves and therefore where the exact break-even volume falls for a given model.
- ⚑Engineering and on-call cost varies by team and is not in any figure here.
What this rests on
infratoapp, “API or self-host: where the break-even sits”, read 27 October 2026. lowest H100 SXM 80GB rate, 730 hours: $1,226.40. https://www.infratoapp.com/blog/api-or-self-host-the-break-even-point
Questions people ask about this
When is it cheaper to self-host an LLM than use an API?
When the monthly API bill for your traffic is larger than the cost of renting the GPUs that would serve the same traffic around the clock, plus the engineering time to run them. For light or uneven traffic the API is almost always cheaper, because idle rented hours are still billed.
How many GPUs do I need to self-host a model?
Start from memory. Weights need about the parameter count times the bytes per parameter, so a 34 billion parameter model in 16-bit needs about 68 GB before any cache. Add memory for the key-value cache, which grows with context length and concurrent requests and cannot be given as one figure.
Why do people self-host when the API is cheaper?
Usually for reasons that are not price: data that must stay inside their own network, a fine-tuned or custom model no provider serves, or a need to control the serving stack and its latency directly. Those are legitimate reasons, and they are separate from the cost arithmetic.
These render as FAQPage structured data · answers are filled from the same live figures
Read next
An inference engine is the software that loads model weights onto a GPU and turns requests into tokens efficiently. You need one the moment you self-host.
Quantization stores weights in fewer bits so a model fits on fewer or smaller GPUs. You give up some precision, and only your own tests say what that costs.
Run a model on the device or near it when data must stay local, the network is unreliable, or round trips are too slow. The price is a much smaller model.