Edge AI and Local LLMs: Cheaper, Greener and More Private

The default way to use a language model is to send a request to a large remote data centre. It works, but it is not always the best fit. Latency, recurring cost, data privacy and energy use all improve when the right model runs closer to where the data lives.

Why small can be enough

Most business tasks, such as classification, extraction, summarisation, routing and question answering over your own documents, do not need the largest model available. Fine-tuned small and medium models often match large general models on a narrow task, at a fraction of the compute.

The toolbox for running models locally

• Quantisation: storing weights in 8-bit or 4-bit precision shrinks memory and speeds up inference with modest quality loss when done carefully.
• Efficient runtimes and formats: engines such as llama.cpp (GGUF), ONNX Runtime and TensorRT target CPUs, GPUs and embedded accelerators.
• Distillation and fine-tuning: teach a compact model the behaviour you need, using methods such as LoRA and QLoRA.
• Edge hardware: from workstation GPUs to embedded modules like NVIDIA Jetson and modern NPUs in laptops and phones.

The hybrid pattern

The practical architecture is rarely "all local" or "all cloud". A router sends routine, sensitive or latency-critical requests to a local model and escalates the rare hard ones to a larger model when policy allows. You keep privacy and cost under control without giving up capability.

What about carbon and water?

Data centres consume electricity and, for cooling, water. Running efficient models closer to the user can reduce data movement and avoid over-provisioned compute. The honest caveat: the saving depends on utilisation, hardware efficiency and the local energy mix. A badly utilised local GPU can be worse than a well-run shared one. That is why we measure instead of assuming; try the editable estimator in the Impact Lab on our homepage and swap in your own numbers.

A short checklist before you go local

1. Define the task and the quality bar with real examples.
2. Benchmark two or three candidate models, including quantised variants.
3. Measure latency, throughput and energy on the target hardware.
4. Plan updates, monitoring and fallback to a larger model.