Token
A token is the unit of text a language model actually reads and writes. It is usually a fragment of a word rather than a whole word, so the model never sees letters or words directly — only a sequence of integers that stand for these fragments.
How text gets cut up
A tokenizer is built by scanning a large corpus and merging the most frequent character sequences into single units, an approach usually based on byte-pair encoding. Common words end up as one token; rarer ones get split. Whitespace is part of the token, so the and the are different entries in the vocabulary.
Rough English rule of thumb: one token is about four characters, so around 0.75 words. That average hides a lot of variance:
- Common English words: one token each.
- Swedish, German and other compounding languages: frequently two to four tokens per word, because the tokenizer was trained mostly on English.
- Names, product codes and technical terms: split into fragments.
- Code: indentation, brackets and camelCase identifiers are token-hungry.
- Emoji and non-Latin scripts: sometimes several tokens per character.
The practical consequence is that the same sentence in Swedish costs meaningfully more tokens than in English — often 1.5 to 2 times as many. It fills the context window faster and costs more per request, for identical meaning.
Why it matters to you
Tokens are the billing unit, the context window unit and the speed unit. Pricing is per million input and output tokens, with output usually several times more expensive. Throughput is quoted in tokens per second. Limits are stated in tokens, never in words or characters.
What to watch out for
A token is not a word, and this breaks specific tasks. The classic failures are direct consequences: a model cannot reliably count the letters in a word, spell backwards, or do character arithmetic, because it never sees the characters. It is looking at an integer that represents straw plus one that represents berry.
Two more traps. Word and character limits in prompts are advisory — "answer in under 100 words" is a request the model cannot verify, whereas a hard max_tokens cuts the answer off mid-sentence. And token counts are per model: the same text measured with two different tokenizers gives two different numbers, so count with the tokenizer for the model you are actually calling.
Frequently asked questions
How many tokens is a page of text?
An A4 page of ordinary English prose is roughly 500 to 700 tokens. The same page in Swedish is commonly 800 to 1,100, because compound words and inflected forms get split into more pieces. Code and tables run higher still. Count with the model’s own tokenizer when the number matters.
Why does Swedish cost more than English to run through a model?
Because tokenizers are trained on corpora dominated by English, so English words are single tokens while Swedish compounds and inflections are split into several. The same meaning therefore consumes more tokens, which means higher cost, more of the context window used and slower generation.
Build it yourself
NorthernGo turns a plain-text description into a working web app with a database, login and a live URL. Local AI generation runs on your own GPU, is unlimited, and is free on every plan.