Inference / Serving

Hosted inference sold per token or per request, where somebody else operates the hardware. Software you run yourself to serve the same weights is an inference engine, and a single endpoint spreading requests across several of these providers is a gateway or router. The price is the product here, which is why this category is priced and most are not.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

Beam8 ATTRIBUTES ANSWEREDbentoml7 ATTRIBUTES ANSWEREDFireworks AI7 ATTRIBUTES ANSWEREDHugging Face TGI6 ATTRIBUTES ANSWEREDKServe6 ATTRIBUTES ANSWEREDModal6 ATTRIBUTES ANSWEREDReplicate6 ATTRIBUTES ANSWEREDCerebras Inference4 ATTRIBUTES ANSWEREDGGladia4 ATTRIBUTES ANSWEREDRev AI4 ATTRIBUTES ANSWEREDAmazon Bedrock3 ATTRIBUTES ANSWEREDDeepInfra3 ATTRIBUTES ANSWERED

Software that actually executes a model and serves tokens: batching, KV cache, quantisation, streaming. Running it yourself is the point; paying somebody per token is an inference API.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

vllm10 ATTRIBUTES ANSWEREDgomodel9 ATTRIBUTES ANSWEREDOllama9 ATTRIBUTES ANSWEREDLocalAI8 ATTRIBUTES ANSWEREDLMDeploy7 ATTRIBUTES ANSWEREDSGLang7 ATTRIBUTES ANSWEREDBaseten6 ATTRIBUTES ANSWEREDdynamo6 ATTRIBUTES ANSWEREDFriendliAI6 ATTRIBUTES ANSWEREDllama.cpp6 ATTRIBUTES ANSWEREDMLC LLM6 ATTRIBUTES ANSWEREDopenvino6 ATTRIBUTES ANSWERED

Running models on the device — phone, browser, microcontroller or a box in a shop — where the constraint is memory and power rather than throughput.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

Pphoneclaw6 ATTRIBUTES ANSWEREDlitert-lm5 ATTRIBUTES ANSWEREDBbigmoeonedge4 ATTRIBUTES ANSWEREDCcarwatch4 ATTRIBUTES ANSWEREDcosmo-edge4 ATTRIBUTES ANSWEREDlemonade4 ATTRIBUTES ANSWEREDNVIDIA Triton Inference Server4 ATTRIBUTES ANSWEREDbrowserai3 ATTRIBUTES ANSWEREDcactus3 ATTRIBUTES ANSWEREDEefficientsam33 ATTRIBUTES ANSWEREDSSCMA3 ATTRIBUTES ANSWEREDCloudflare Workers AI2 ATTRIBUTES ANSWERED

Turning a model into something faster on particular hardware: graph compilation, kernel generation, quantisation toolchains. The runtime that then serves it is an inference engine.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

DeepSpeed7 ATTRIBUTES ANSWEREDflashinfer4 ATTRIBUTES ANSWEREDcactus3 ATTRIBUTES ANSWEREDkitops2 ATTRIBUTES ANSWEREDAannotaterb1 ATTRIBUTE ANSWEREDenvd1 ATTRIBUTE ANSWEREDjetstream1 ATTRIBUTE ANSWEREDtensorrt-llm1 ATTRIBUTE ANSWEREDBbrickNO ANSWER ON RECORDTurboOCRNO ANSWER ON RECORD