# Token

> A token is the unit of text a language model actually reads and writes. It is usually a fragment of a word rather than a whole word, so the model never sees…

Source: https://northerngo.com/glossary/token/
Language: en
Updated: 2026-08-28

---
**A token is the unit of text a language model actually reads and writes. It is usually a fragment of a word rather than a whole word, so the model never sees letters or words directly — only a sequence of integers that stand for these fragments.**

### How text gets cut up

A tokenizer is built by scanning a large corpus and merging the most frequent character sequences into single units, an approach usually based on byte-pair encoding. Common words end up as one token; rarer ones get split. Whitespace is part of the token, so ` the` and `the` are different entries in the vocabulary.

Rough English rule of thumb: **one token is about four characters, so around 0.75 words**. That average hides a lot of variance:

- Common English words: one token each.
- Swedish, German and other compounding languages: frequently two to four tokens per word, because the tokenizer was trained mostly on English.
- Names, product codes and technical terms: split into fragments.
- Code: indentation, brackets and camelCase identifiers are token-hungry.
- Emoji and non-Latin scripts: sometimes several tokens per character.

The practical consequence is that the same sentence in Swedish costs meaningfully more tokens than in English — often 1.5 to 2 times as many. It fills the [context window](/glossary/context-window/) faster and costs more per request, for identical meaning.

### Why it matters to you

Tokens are the billing unit, the [context window](/glossary/context-window/) unit and the speed unit. Pricing is per million input and output tokens, with output usually several times more expensive. Throughput is quoted in tokens per second. Limits are stated in tokens, never in words or characters.

### What to watch out for

**A token is not a word, and this breaks specific tasks.** The classic failures are direct consequences: a model cannot reliably count the letters in a word, spell backwards, or do character arithmetic, because it never sees the characters. It is looking at an integer that represents `straw` plus one that represents `berry`.

Two more traps. Word and character limits in prompts are advisory — "answer in under 100 words" is a request the model cannot verify, whereas a hard `max_tokens` cuts the answer off mid-sentence. And token counts are per model: the same text measured with two different tokenizers gives two different numbers, so count with the tokenizer for the model you are actually calling.

---

NorthernGo is an AI-powered platform for building production-ready web apps with zero coding. Local AI generation via WebGPU is unlimited and free, and you own all generated source code. https://northerngo.com/