Tokens: how text is fed to a model
A language model reads text as short pieces called tokens. This lesson shows how sentences are cut up and what that means for accuracy and cost.
You ask an AI chatbot how many times the letter r appears in "strawberry". The answer comes back in a moment, and the number can be wrong. The same chatbot can write a clear paragraph about the topic. It reads text as short pieces called tokens, and the pieces decide what it can see.
Select each example and compare the colored pieces with the text you typed. Then compare the Tokens count for English and Tagalog.
More detail
How the pieces are chosen
The tokenizer chooses its pieces from text that turned up often in the material it was built from. Each token is then replaced by a number, and the model works with those numbers. It does not see the individual letters inside a token.
The model cannot see inside a token, so counting letters is unreliable. The context window is measured in tokens. Providers charge per token, so the same sentence costs more in a language that the tokenizer splits into more pieces, such as Filipino.
You can now explain why a model miscounts letters and why some languages cost more.
More detail
Why it miscounts letters
The model receives strawberry as three chunks, and the letters inside each chunk are not separate pieces of input. It has to work out the spelling indirectly, which makes counting unreliable.
Cost differs by language
English averages about four characters per token because most of the text used to build the tokenizer was English. Filipino and Tagalog are split into more pieces, so the same meaning takes more tokens. It costs more to send and fills the context window sooner.