How tokenization works, and why every model counts your text differently

Tokens are the unit you are billed in, but nobody agrees on what one is. Understanding why is the difference between a cost estimate that holds and one that is 30% out.

Tokenization · 10 min read · Last reviewed 2026-07-28

Key takeaways

Tokens are subword fragments produced by byte-pair encoding, learned statistically from training data rather than defined by any rule.
Every provider trains its own tokenizer, so identical text yields different token counts — differences of 10–30% between model families are normal.
English prose runs about 4 characters per token. Code, JSON, non-Latin scripts and emoji are far denser and can cost several times more per character.
Only OpenAI publishes a tokenizer that runs client-side, which is why every browser-based counter estimates the rest.
Formatting choices — JSON vs prose, verbose keys, deep indentation — have a measurable and often large effect on token count.

You are billed per token, so it is worth knowing what one is. The short answer is that a token is a fragment of text, usually between one character and one word, drawn from a fixed vocabulary that the model learned during training.

The longer answer explains why the same paragraph costs different amounts on different models, why your cost estimate can be 30% out if you work in Chinese or Python, and why a few formatting decisions can save more money than switching models.

Why not just use words?

A model needs a fixed-size vocabulary of symbols it can map to numbers. Words look like the obvious unit, and they fail badly.

English has hundreds of thousands of words in common use, plus names, plus typos, plus technical terms, plus every other language you want to support. A word-level vocabulary would either be impossibly large or would hit unknown words constantly — and unknown words are catastrophic, because the model has no representation for them at all. "Antidisestablishmentarianism" and a novel product name would both become the same useless placeholder.

Characters solve that problem and create a worse one. A 26-letter vocabulary never encounters anything unknown, but it makes sequences enormously long, and transformer attention cost grows quadratically with sequence length. It also forces the model to learn spelling from scratch before it can learn meaning.

Subword tokenization is the compromise, and it is what everyone uses. Common words get their own single token. Rare words get decomposed into meaningful fragments. Nothing is ever unknown, because the vocabulary bottoms out at individual bytes.

Byte-pair encoding

The dominant algorithm is byte-pair encoding, or BPE. Training it is mechanically simple:

Start with a vocabulary of individual bytes.
Scan a large text corpus and count every adjacent pair of symbols.
Find the most frequent pair and merge it into a single new symbol, added to the vocabulary.
Repeat until the vocabulary reaches the target size.

Run this on English and the early merges are unsurprising: t and h become th; th and e become the. Thousands of merges later, common words, frequent prefixes and suffixes, and even multi-word phrases have their own tokens.

Two properties follow from this that explain most of the surprising behaviour you will encounter.

The vocabulary is statistical, not linguistic. Nobody decided that -ing should be a token. It became one because it appeared frequently. This means token boundaries often do not align with anything a linguist would recognise as a morpheme, and it means a tokenizer trained mostly on English is genuinely worse at everything else.

Leading spaces are part of the token. In most modern tokenizers " the" (with the space) and "the" are different tokens. This is why token counts do not decompose neatly by word, and why concatenating strings without spaces can change the count in unintuitive ways.

Why models disagree on the count

Each provider trains its own tokenizer on its own data mixture, at its own vocabulary size. Nothing forces them to agree, and they do not.

OpenAI's current models use an encoding called o200k_base, with a vocabulary of roughly 200,000 tokens — a substantial increase over the ~100,000 of the previous generation, and part of why newer OpenAI models handle non-English text more efficiently. Anthropic, Google, Meta, xAI and DeepSeek all use different tokenizers with different vocabularies.

Larger vocabularies mean fewer tokens for the same text, because more multi-character sequences have dedicated entries. So a model with a 200,000-token vocabulary will typically report a lower count than one with 32,000 for the same input — which does not make it cheaper, because the per-token price differs too. It only means token counts are not comparable across model families and per-token prices cannot be compared without them.

Differences of 10–30% between families are routine. Tokenizer changes within a family happen too: a newer model in the same lineup can produce noticeably more tokens for identical text if its tokenizer was retrained, which is a real and easily missed cost increase when upgrading.

What makes text expensive

The rule of thumb for English prose is about four characters per token, or roughly 0.75 tokens per word. Here is where that breaks down, in rough order of severity.

Approximate token density by content type, relative to English prose. Directional rather than exact — actual ratios vary by tokenizer.
Content type	Chars / token	Cost vs. prose
English prose	~4.0	baseline
Western European languages	~3.0–3.5	slightly higher
Source code	~2.5–3.5	higher
JSON / XML	~2.5–3.0	notably higher
CJK (Chinese, Japanese, Korean)	~1.0–2.0	much higher per character
Base64, hashes, UUIDs	~2.0	very high
Emoji	<1.0	extreme — several tokens each

Code is dense because indentation, punctuation and camelCase or snake_case identifiers all fragment. A variable named userAuthenticationTokenExpiry may be five or six tokens. Deeply indented code pays for whitespace on every line.

Structured data is worse than it looks. Every brace, bracket, quote, colon and comma is a token or part of one, and keys repeat on every object in an array. Converting a table from JSON to CSV or a compact markdown table routinely cuts token count by 30–50% with no loss of information — one of the easiest savings available if you pass tabular data to a model.

Non-Latin scripts are the harshest case. Tokenizers trained predominantly on English text have relatively few dedicated tokens for Chinese, Japanese, Korean, Arabic, Hindi, Thai and others, so text in those languages fragments toward individual characters or even bytes. A Chinese character may cost a token or more on its own, against four English characters per token. This is a real and well-documented equity problem in LLM pricing: the same semantic content costs several times more to process in some languages than in English. Newer, larger vocabularies have improved this considerably but not eliminated it.

Emoji and unusual Unicode often decompose into multiple bytes and therefore multiple tokens. A single emoji can cost three or four tokens; a composed emoji with skin-tone and gender modifiers can cost substantially more.

Why browser-based counters estimate

OpenAI open-sourced its tokenizer, so tiktoken and its JavaScript port can run entirely in your browser and produce exact counts for OpenAI models. That is why this site's OpenAI figures are labelled Exact.

No other major provider publishes a tokenizer that can run client-side. Anthropic offers a token-counting API endpoint, which is accurate but requires sending your text to a server — unacceptable for a tool whose main guarantee is that your prompt never leaves your machine. So non-OpenAI counts are estimated from a characters-per-token ratio tuned per provider, and labelled ~Approx.

For English prose this lands within a few percent. For the dense content types above it will run low, sometimes considerably. The methodology page lists the exact ratios used and where they are least reliable. If you need an exact non-OpenAI count, the provider's own endpoint is the only real answer.

Practical implications

Do not compare per-token prices across families without accounting for tokenizer differences. A model priced 10% lower per token that produces 20% more tokens for your content is more expensive.

Measure with your real content. If your workload is Python, or Japanese, or JSON payloads, generic estimates will mislead you. Paste the actual thing.

Choose compact formats. CSV over JSON for tabular data. Markdown tables over nested objects. Terse field names in schemas you send on every request. These are free savings that compound across every call.

Strip what the model does not need. Boilerplate headers, license blocks, redundant instructions, and retrieved chunks that turned out not to be relevant are all billed at full rate.

Re-measure after a model upgrade. Tokenizer changes within a model family are a silent cost increase. Check the count on a representative prompt before and after.

Try it

Paste your real prompt into the calculator and compare token counts across all thirteen models at once. The differences between families on your specific content are usually larger than people expect — and they are the differences that decide which model is cheapest for you.