Inference engines: what the category is for

The program between the weights and the request

An inference engine is the server software that loads a model's weights into GPU memory, accepts requests, schedules them into batches, manages the key-value cache and streams tokens back. A raw model file does none of this. A hosted API provider runs an engine for you. Renting a GPU and serving a model yourself makes choosing and running an engine your first real decision.

The category sits inside the inference layer of the directory, beside APIs, gateways and edge runtimes. Live products in that layer on record: 186.

What an engine actually does on each request

A request arrives as text. The engine tokenises it, runs the whole prompt through the model in one pass to fill the cache (the prefill), then generates output one token at a time (the decode), reading the cache at each step. It streams tokens back as they come and frees the cache when the reply ends.

The hard part is doing that for many requests at once. The engine decides which requests share a step, how cache memory is allocated so a long request does not strand space, when to refuse new work, and whether identical prompt prefixes can share cache. Those decisions are most of what separates one engine from another.

Around that core, engines add the practical parts: an HTTP endpoint, often shaped like a common chat API, support for particular precisions and quantized formats, and ways to split a model too large for one card across several.

Where the engine ends and the platform begins

An engine serves one model on one set of cards. It does not usually decide how many replicas run, route traffic between them, roll out a new model version, or bill a team for its share. Those jobs belong to whatever sits around it: a container orchestrator, a serving platform, a gateway, or a managed service that bundles all of them.

This boundary matters when reading product pages. A product described as an inference platform may include an engine, or may run someone else's engine underneath and add the scaling, routing and dashboards on top. A product described as an engine may ship a small server and nothing else.

Licensing follows the same line. An open engine can be inspected and patched; a commercial platform comes with someone to call. Which you want depends on your team, and the product pages say which is which.

A team that already runs containers in production may need only an engine plus its existing orchestration. For a team that does not, the platform around the engine is most of the work, and a managed option that hides it deserves a serious look. Read what each product page says it includes before comparing two that turn out to be different things.

Questions to ask of any engine

Does it run your model? Engines support architectures one by one, and a new model family may land in one engine weeks before another. Check the vendor documentation, not a blog post.

Does it run on your card? Some engines target one vendor's GPUs, some several, some include CPUs or edge chips. A Hopper card supports 8-bit floating point and an older Ampere card does not, so the same engine may offer different formats on each.

How does it split a model across cards, and across machines? A model whose weights alone exceed one card's memory has to be split, and engines differ in how they do it. Then the plain questions: licence, release cadence, who answers a bug report, and whether it exposes the metrics you will need at three in the morning.

Why this post names no engine

The site holds no latency, throughput or quality measurements, so it has no basis for saying one engine is faster or better. Published benchmarks from vendors are run on their chosen model, card, batch size and prompt length, which may look nothing like yours.

What the inference engine category page does hold is the list of products, each with the facts read from the vendor's own pages and the date they were read. A missing value there means no answer on record, not a no. Use it as the shortlist, then measure. The engine category is one part of the directory (live products across the whole directory on record: 1,116), sitting beside the hosted APIs that run an engine for you.

Running a fair trial between engines

Pick two or three candidates that support your model and card. Deploy each on the same rented GPU, with the same model weights and precision, and replay a sample of your real traffic at the concurrency you expect at peak. Record time to first token, tokens per second per request, tokens per GPU-hour, and memory at peak.

Include the boring checks in the trial. How long does each engine take to start with your model loaded? Does it recover cleanly when a request is cancelled mid-stream? Do its metrics show queue length and cache use? An engine that is fast and opaque is harder to run than one that is slightly slower and tells you what it is doing.

Then repeat at double the concurrency, because behaviour at average load does not predict behaviour under strain. Price the hours the trial needs with the GPU cost calculator; a day on one card is a small bill set against months of serving on the wrong engine.