Inference
Inference is the act of running an already-trained model to get an answer, as opposed to training, which produces the model in the first place. Every prompt you send is one inference request, and it is where essentially all of the ongoing cost of using AI sits.
Two phases, with different bottlenecks
An inference request splits in two. Prefill processes your whole prompt at once; it is compute-heavy and parallel, and it determines how long you wait for the first word. Decode then generates one token at a time, each pass depending on the previous one, so it cannot be parallelised within a single response. Decode is limited by memory bandwidth rather than raw compute — the weights have to be read for every token produced.
This is why the two numbers people quote measure different things. Time to first token is mostly about prompt length and queueing. Tokens per second is about bandwidth and model size. A long prompt makes the first metric worse without touching the second.
To avoid recomputing attention over the whole prompt at every step, runtimes keep a KV cache. It is fast and it is large, and its size grows with the context window — which is a large part of why long contexts get expensive.
Where it runs
Hosted inference means an API call to a provider running the model on server GPUs. Local inference means the same computation on your own hardware, usually with a quantized model so it fits in available memory. The arithmetic is identical; only the economics and the data path change.
What to watch out for
- Cost scales with tokens, not requests. A conversation that re-sends its full history pays for that history again on every turn. Long system prompts are charged every single call.
- Latency is not linear in output length only. Doubling the prompt raises prefill time; doubling the answer raises decode time. Diagnose which one you have before optimising.
- Batching helps the server, not you. Providers batch requests to raise total throughput, which can make your individual response slower under load.
- Fixed weights, variable answers. Inference does not change the model. Identical prompts can still give different outputs because sampling is random above temperature zero.
Caching is the underrated lever: repeated prefixes, embeddings and full responses to common questions are all cheaper to store than to regenerate.
Frequently asked questions
Why is the first response slow and the rest fast?
Usually because the model has to be loaded into memory before the first request, which can take from seconds to minutes depending on its size. On hosted APIs it is more often queueing or a cold container. After that the weights are resident and only the actual generation costs time.
Build it yourself
NorthernGo turns a plain-text description into a working web app with a database, login and a live URL. Local AI generation runs on your own GPU, is unlimited, and is free on every plan.