Skip to content
InfratoApp
Layers / Inference / Serving / Compilers and runtimes

Compilers and runtimes

COMPARE TWO OF THESE →

Turning a model into something faster on particular hardware: graph compilation, kernel generation, quantisation toolchains. The runtime that then serves it is an inference engine.

NOINDEX

This category is below the depth bar — it holds 10 listings where we would want 12 before trusting a shortlist, so it is excluded from search engines and says so here rather than quietly.

10 in this category
SORTED BY STARS

NARROW BY WHAT THIS CATEGORY ASKS

LICENCE · 4 NOT READ

PRODUCTUSED BYSTARSDOWNLOADS 30D
DeepSpeedDeepSpeed library1043,091tensorrt-llmTensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.114,686flashinferIntroduce Techniques to accelerate Large Language Model Deployment6,381cactusQuantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.6,003envd🏕️ Reproducible development environment for humans and agents2,229kitopsAn open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.1,413TurboOCRTurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC1,065AannotaterbA Ruby Gem that adds annotations to your Rails models and route files.617jetstreamJetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).457BbrickAuto-generate models, views, controllers, and routes in a Rails app based on database structure388

SORTED BY ONE STATED COLUMN · USED BY = LISTED PRODUCTS DEPENDING ON IT, IN THIS DIRECTORY — NOT A POPULARITY FIGURE · NO OVERALL SCORE AND NO RANK COLUMN, BECAUSE BLENDING ADOPTION, STARS AND DOWNLOADS NEEDS WEIGHTS, AND WEIGHTS ARE AN OPINION PRESENTED AS ARITHMETIC

Cite this page

The count and its date sit inside the citation, because “Compilers and runtimes” without a date is a claim that quietly goes wrong as the category fills.

infratoapp, “Compilers and runtimes — Inference / Serving”, read 28 October 2026. 10 listed products. https://www.infratoapp.com/layers/inference/compilers

No overall score, and no rank column

Blending adoption, stars and downloads needs weights, and weights are an opinion presented as arithmetic — so this table sorts by one stated column at a time and shows which. This view holds no billing column and joins no billing table: a sponsor cannot appear higher, and the signals meter is a row’s own confidence, never a score you can sort on.

Frequently asked

What is in the Compilers and runtimes category?
10 products in the Inference / Serving layer of the AI stack. Turning a model into something faster on particular hardware: graph compilation, kernel generation, quantisation toolchains. The runtime that then serves it is an inference engine. Listed members include DeepSpeed, tensorrt-llm and flashinfer. Every fact on each listing carries the page it was read from and the date it was read.
How are Compilers and runtimes compared here?
By the 6 attributes this category declares, asked of every member and answered only where a source states it. A product with no answer on record shows a gap rather than a guess, and no blended score is published — weights would be an opinion presented as arithmetic.
DeepSpeed or tensorrt-llm?
Both are listed under Compilers and runtimes. We publish no verdict between them — we hold no benchmark that would support one. What we do hold is the same attribute sheet for each, side by side with the source behind every value, at /compare/deepspeed-vs-tensorrt-llm.
What does a blank mean on a compilers and runtimes row?
That nobody has told us, not that the answer is no. A hollow ring is "no answer on record", which is a different state from a recorded no — and the two look different on purpose. A listing with a blank has not failed the question; it has not been read for it.
How often are compilers and runtimes figures re-read?
Every value carries the date it was read and a link to the page it came from. Prices move fastest and are re-read most often; an attribute read weeks ago says so rather than presenting itself as current. Anything past 21 days is hidden rather than shown stale.
Is this every compilers and runtime there is?
No. This is 10 we have read and can cite. A directory that claimed completeness would be making a promise it cannot keep, so the page states its own count instead and says when a category is too thin to shortlist from.
Can a vendor pay to rank higher here?
No. Nothing on this page can be bought: a sponsor cannot enter a table, reorder a row or change a mark, and there is no ranked position for sale. How the site is paid for, and the line each source of money does not cross, is set out under money and independence at /our-data#sec5.