September 17, 2026

What Is Tokenization in LLMs (and why it actually matters)

A short sentence might use 15 tokens in one model and 18 in another. An uncommon word can produce an even bigger difference.

That happens because LLMs don’t all split text in the same way. Their tokenizers can break the same text into different words, subwords, or characters. These differences affect how models process text and how much they cost to use.

This guide explains what tokens are, how tokenizers work, and why token counts vary between models.

What is a token in AI?

Language models don’t see words the way we do. Before any text reaches the model, it’s broken into smaller units called tokens. According to OpenAI, those units might be whole words, parts of words, single characters, or even punctuation marks. This process is called tokenization.

For example, the sentence:

"Tokenization is fascinating."

might get split into something like this:

["Tokenization", " is", " fascinat", "ing", "."]

Notice that "fascinating" didn’t stay whole — it got split into two pieces. That’s completely normal, and it’s the key to understanding how these models actually work under the hood.

Types of Tokenization Methods

There are three main strategies for breaking text into tokens, and each one makes a different trade-off:

Word-level

Every word is its own token. For example, "unbelievable" stays whole: ["unbelievable"]. It’s simple, and every token is a whole word instead of a fragment, but it may require a huge vocabulary and can’t process new or rare words — a typo or new slang term has no matching token.

Character-level

Every single letter is its own token. "unbelievable" becomes ["u", "n", "b", "e", "l", "i", "e", "v", "a", "b", "l", "e"]. It can process a wider range of inputs — unknown words, typos, complex syntax — and may allow the vocabulary to be reduced. But it produces far more tokens per piece of text, which costs more compute.

Subword

Subword tokenization sits between word-level and character-level tokenization. It keeps common words or character sequences as single tokens and splits less common words into smaller units from the tokenizer’s vocabulary. "unbelievable" splits into ["un", "believ", "able"].

How does the tokenizer work?

At a high level, turning raw text into something a model can actually use follows the same predictable routine every time:

1. Take the input. This is just the raw text you’re starting with — a prompt, a sentence, whatever needs processing.
2. Split it into units. The tokenizer breaks that string into pieces — for example, words, subwords, or characters. How it chooses those tokens depends on the method it uses, as mentioned above.
3. Map each unit to a token ID. Every token in the tokenizer’s vocabulary has its own numerical ID. Once the text has been split, the tokenizer replaces each token with its corresponding number.

The model doesn’t receive the original sentence. It receives an ordered sequence of token IDs representing that sentence.

4. Package the IDs into sequences. The model uses the sequence to predict the next token. It adds that token to the sequence and repeats the process until it reaches a stopping point, such as an end-of-sequence token or its maximum output length.

The tokenizer then converts the generated token IDs back into text you can read.

Strung together on our example sentence, it looks like this:

Text → Tokens → IDs
"Tokenization is fascinating."
→ ["Tokenization", " is", " fascinat", "ing", "."]
→ [18413, 318, 8871, 278, 13]

Tokens, cost: the part that actually costs you money

Tokens aren’t just how LLMs process text. They’re also used to calculate cost.

Say your prompt contains 50 tokens and the model’s response contains 100. Together, the prompt and response use 150 tokens.

The calculation sounds simple, but token counts can vary between models. The same prompt might become 100 tokens in one model and 120 in another because their tokenizers split the text differently. Since models can also charge different rates, the same prompt can end up costing a different amount.

A study by the University of Cambridge and The University of Hong Kong put it under the microscope. The number swings wildly depending on which model you’re using and how common the word is.

Claude and Gemini tend to overestimate token counts on everyday words. DeepSeek and Mistral undershoot.

Here’s what that looks like per model, pulled straight from the study’s data.

First, the uncommon Hawaiian word humuhumunukunukuāpuaʻa:

Next, let’s look at the ordinary English sentence "A fox knows many things, but a hedgehog knows one big thing.":

Same text, same length, meaningfully different token counts depending on whose tokenizer is doing the counting — nearly 50% more tokens for the Hawaiian word between Grok and the highest count, and 20% more for the plain English sentence between the lowest and highest.

Start with the tokens behind the text

Token counts get confusing because different tokenizers handle text differently. One might keep a word whole, while another splits it into two or three pieces. The LLM then uses the token IDs for those pieces to generate its response.

That’s why word count isn’t much help when you’re trying to estimate cost. You need the token count for the LLM you’re using, plus the tokens in its response. A lower price per token can still work out more expensive if that LLM uses more tokens for the same text.

If you’re comparing LLMs, test them with a few of your real prompts and compare the total token usage. You’ll get a much clearer idea of what each one will actually cost.