Local AI

Why people want it

Three reasons, usually in this order. Confidentiality: text that never leaves the machine cannot be logged, retained or used for training by anyone else. Cost: once the hardware is paid for, generation is free, so you can run the same prompt fifty times while iterating. Independence: no rate limits, no deprecation notice, no price change.

There are two common shapes. Either the model runs in the browser through WebGPU, which requires no install from the user, or it runs as a separate process on the machine — Ollama, llama.cpp, LM Studio — that applications talk to over localhost.

The trade-off is real

Local models are smaller models. A 7-billion-parameter quantized model that fits in 5 GB of video memory is not competitive with a frontier hosted model on hard reasoning, long documents or obscure knowledge. It is often entirely adequate for classification, extraction, summarising, rewriting and drafting.

The other costs are practical: a multi-gigabyte first download, a fixed and usually small context window, noticeably lower tokens per second, and support questions from users whose hardware cannot do it.

The distinction people get wrong

Local is not the same as offline, and it is not automatically private end to end. Two things are worth separating:

1. Where the model runs versus whether the app needs the network. Weights are downloaded over the internet the first time. A page that runs a model locally can still send analytics, telemetry or your prompt somewhere else.
2. Local generation versus local everything. A product can advertise a local mode and still make one cloud call per session for planning, routing or safety filtering.

The question worth asking a vendor is not "do you support local models" but "which specific requests leave the device, and when". NorthernGo's answer is that local generation runs entirely on your own GPU on every plan and sends no request off the device, while its cloud mode uses hosted Google Gemini models and is metered — two different paths, not one with a privacy label.

A third path people mix in is bring-your-own-key. That is still a cloud call: the prompt leaves the device, and only the bill and the model family change. It is not local AI.