Edge inference: when it makes sense

Three reasons, and one cost

Edge inference means running the model on the user's device, a gateway box on site, or a server close to them, instead of in a distant data centre. It makes sense for three reasons: the data must not leave the device or building, the network is unreliable or absent, or a round trip to a cloud region is too slow for the task. The cost is model size. An edge device usually has far less memory than a data-centre GPU, so the model must shrink to fit.

Without one of those three reasons, a hosted API or a rented GPU is simpler to run, simpler to update, and can serve a larger model than any device in a pocket or a cabinet.

What counts as the edge

The word covers a wide spread of hardware. At one end are phones and laptops with a neural accelerator built into the chip. In the middle are small boards and industrial PCs on a factory floor or in a shop. At the near end are servers in a regional point of presence, close to users but still in a rack.

Each has different memory, power and cooling limits, and each runs a different software stack. The edge inference category lists runtimes and platforms aimed at these targets; the silicon layer covers the chip itself (live products on record: 8).

Privacy is the strongest reason

A model that runs on the device never sends the prompt anywhere. For medical notes, legal documents, keyboard input or anything a contract says must stay on the premises, that removes a whole class of problem: no data processing agreement with a model provider, no question of where logs are kept, no transfer across a border.

It is a strong argument because it is binary. A cloud provider can promise not to retain data; a device that never transmits does not have to promise. Where the requirement is legal, not a preference, edge may be the only design that passes review.

Check the whole path, though. A local model that sends telemetry, crash reports or usage logs to a server has reopened the door it was meant to close. The privacy case holds only if nothing derived from the prompt leaves the device either.

Latency and connectivity, more carefully

A network round trip adds delay before the model even starts. For a voice assistant, a camera that must react to what it sees, or a control loop on a machine, that delay can matter more than how fast the model runs. Local inference removes the trip.

It does not make generation fast. Small hardware with a small model generates at whatever pace that device manages, so a long reply may finish later on the device than it would from the cloud, and only a measurement on your hardware says which. Work out whether your task is judged by time to start or time to finish: a wake word or an object detector is judged by the start, a drafted email by the finish. Offline operation is simpler: on a ship, a mine or a plane without signal, local is the only option.

Cost moves from a monthly bill to the device

Cloud inference is an operating cost: tokens or GPU-hours, every month, rising with use. Edge inference moves most of that onto hardware someone has already bought. When the model runs on the user's own phone or laptop, the compute is paid for by the user, and your marginal cost per request is close to nothing.

That changes the arithmetic for products with many free users. A feature that would cost a token bill for every free account can run locally for no per-request charge, which is a real reason some teams accept a smaller model. On hardware you supply, such as kiosks or factory boxes, the cost becomes a capital purchase spread over the device's life, plus the work of maintaining a fleet.

Compare it honestly. Price the hosted alternative with the LLM cost calculator at your expected volume, and set that against the engineering time to ship, update and support a model on every target device. For a small team the engineering side can be the larger number.

Fitting a model onto small hardware

Memory is the wall. Weights need about the parameter count times the bytes per parameter, so a 7 billion parameter model needs about 14 GB in 16-bit and about 3.5 GB in 4-bit. That is why edge deployments lean hard on quantization and on small models, and why the key-value cache, which grows with context length, often limits how long a conversation can run.

Plan for updates too. A cloud model changes when the provider ships; a model on a fleet of devices changes only when you push a new file to each of them. Version pinning, rollback and storage on the device all become your problem.

Download size is part of that problem. A model of a few gigabytes is a large file to push to a phone on a metered connection, and users notice when an update eats their data allowance.

Devices also differ from each other. A model tested on a recent flagship phone may not fit, or may run too slowly to be useful, on the older devices many users still carry. Test on the weakest hardware you intend to support and set a floor below which the feature falls back to the cloud.

A hybrid design is common: a small local model handles the private or instant part and hands harder requests to a larger hosted one. Compare the hosted side on the LLM prices page.