# Inference

> Inference is the act of running an already-trained model to get an answer, as opposed to training, which produces the model in the first place. Every prompt…

Source: https://northerngo.com/glossary/inference/
Language: en
Updated: 2026-08-28

---
**Inference is the act of running an already-trained model to get an answer, as opposed to training, which produces the model in the first place. Every prompt you send is one inference request, and it is where essentially all of the ongoing cost of using AI sits.**

### Two phases, with different bottlenecks

An inference request splits in two. **Prefill** processes your whole prompt at once; it is compute-heavy and parallel, and it determines how long you wait for the first word. **Decode** then generates one [token](/glossary/token/) at a time, each pass depending on the previous one, so it cannot be parallelised within a single response. Decode is limited by memory bandwidth rather than raw compute — the weights have to be read for every token produced.

This is why the two numbers people quote measure different things. *Time to first token* is mostly about prompt length and queueing. *Tokens per second* is about bandwidth and model size. A long prompt makes the first metric worse without touching the second.

To avoid recomputing attention over the whole prompt at every step, runtimes keep a **KV cache**. It is fast and it is large, and its size grows with the [context window](/glossary/context-window/) — which is a large part of why long contexts get expensive.

### Where it runs

Hosted inference means an API call to a provider running the model on server GPUs. [Local inference](/glossary/local-ai/) means the same computation on your own hardware, usually with a quantized model so it fits in available memory. The arithmetic is identical; only the economics and the data path change.

### What to watch out for

- **Cost scales with tokens, not requests.** A conversation that re-sends its full history pays for that history again on every turn. Long system prompts are charged every single call.
- **Latency is not linear in output length only.** Doubling the prompt raises prefill time; doubling the answer raises decode time. Diagnose which one you have before optimising.
- **Batching helps the server, not you.** Providers batch requests to raise total throughput, which can make your individual response slower under load.
- **Fixed weights, variable answers.** Inference does not change the model. Identical prompts can still give different outputs because sampling is random above temperature zero.

Caching is the underrated lever: repeated prefixes, embeddings and full responses to common questions are all cheaper to store than to regenerate.

---

NorthernGo is an AI-powered platform for building production-ready web apps with zero coding. Local AI generation via WebGPU is unlimited and free, and you own all generated source code. https://northerngo.com/