Ollama
Ollama is an open-source tool that runs language models on your own computer and exposes them over a local HTTP API. It handles downloading, quantized model files and GPU setup, so using a local model becomes one command instead of a build process.
How it is used
Install it, then pull and run a model:
ollama pull llama3.1
ollama run llama3.1A background service listens on http://localhost:11434. That is the part that matters for developers: any application on the machine can call it, and Ollama also exposes an OpenAI-compatible endpoint, so most existing client libraries work by changing the base URL and passing a dummy API key.
Models are distributed as quantized GGUF files, most commonly 4-bit, and are pulled from a registry by name and tag. A Modelfile lets you package a base model together with a system prompt and sampling parameters under your own name — useful for pinning a configuration your whole team uses.
Underneath it is llama.cpp, which is why it runs on Apple Silicon, NVIDIA and AMD GPUs and falls back to CPU. CPU-only works and is slow; expect a few tokens per second on a large model.
Where it fits
Ollama is the standard answer for local AI when the model should run on the machine but outside the browser. Compared with a browser runtime on WebGPU, it gives you larger models, a bigger context window and better speed, at the cost of requiring an install. NorthernGo can connect to a local Ollama instance for exactly that reason: it is the option for people who want a guarantee that nothing leaves the machine.
What to watch out for
- A web page cannot call localhost by default. Ollama's CORS policy blocks browser origins until you allow them, typically by setting
OLLAMA_ORIGINSbefore starting the service. This is the single most common reason a browser integration silently fails, and it is a deliberate protection rather than a bug. - Do not expose it to the network. The API has no authentication. Binding it to a public address hands anyone your GPU, and worse.
- Memory is the constraint. A model that does not fit in VRAM spills to system RAM and becomes dramatically slower, or fails outright.
- A model name is not a guarantee. The default tag is usually a 4-bit quantization of an instruction-tuned variant, which is not the same as the full-precision model of that name. Check the tag when comparing quality against a hosted provider.
Frequently asked questions
Is Ollama free?
Yes. The tool is open source and there is no charge per request, since everything runs on your own hardware. The costs are the disk space for model files, a few gigabytes each, and electricity. Individual models carry their own licences, which is worth checking before commercial use.
Why can my web app not reach Ollama on localhost?
Because Ollama rejects cross-origin requests from browsers unless the origin is allowed. Set the OLLAMA_ORIGINS environment variable to include your site and restart the service. If the request fails without any response at all, that is the cause; if you get an HTTP error instead, the problem is elsewhere.
Build it yourself
NorthernGo turns a plain-text description into a working web app with a database, login and a live URL. Local AI generation runs on your own GPU, is unlimited, and is free on every plan.