Tokens: how text is fed to a model

A language model reads text as short pieces called tokens. This lesson shows how sentences are cut up and what that means for accuracy and cost.

You ask an AI chatbot how many times the letter r appears in "strawberry". The answer comes back in a moment, and the number can be wrong. The same chatbot can write a clear paragraph about the topic. It reads text as short pieces called tokens, and the pieces decide what it can see.

Your questionTyped text
The text, cut into piecesCut by a tokenizer
The modelReads the tokens
Think first. What would the model need to receive as input to count letters reliably? Write down your answer, then compare it with what the model receives in the next step.

Select each example and compare the colored pieces with the text you typed. Then compare the Tokens count for English and Tagalog.

As you typed it
As the model receives it
Characters
—
Tokens
—
Characters per token
—
More detail

How the pieces are chosen

The tokenizer chooses its pieces from text that turned up often in the material it was built from. Each token is then replaced by a number, and the model works with those numbers. It does not see the individual letters inside a token.

The model cannot see inside a token, so counting letters is unreliable. The context window is measured in tokens. Providers charge per token, so the same sentence costs more in a language that the tokenizer splits into more pieces, such as Filipino.

You can now explain why a model miscounts letters and why some languages cost more.

More detail

Why it miscounts letters

The model receives strawberry as three chunks, and the letters inside each chunk are not separate pieces of input. It has to work out the spelling indirectly, which makes counting unreliable.

Cost differs by language

English averages about four characters per token because most of the text used to build the tokenizer was English. Filipino and Tagalog are split into more pieces, so the same meaning takes more tokens. It costs more to send and fills the context window sooner.