How a language model writes: predicting the next word

A language model writes by choosing one word at a time. It repeats the same step for every word it adds.

You type a message on your phone. Three words appear above the keyboard. You tap one, and three new words appear. The keyboard picks them from the words you have typed so far. The language model behind an AI chatbot also picks each next word from the text so far.

Your message so farContext
Suggested next wordsProbability for each
Think first. A model has read "The capital of France is". Does it give its top word about 30, 60 or 90 percent probability? You will check in the next step.
Collect guesses before students open the widget. Most pick 90 percent or more, and the widget shows 89 percent. The distinction between recalling a fact and continuing a pattern comes up again in the code example under "Try other prompts".

The model gives every candidate word a probability, then picks one. Press Pick one a few times. Then set the temperature to 0, press Do it ten times and compare the two tallies.

More detail

What temperature does

Temperature controls how much randomness goes into the choice. At 0 the model takes the top word every time. At 1 it picks at random, in proportion to the percentages shown. Picking at random in proportion to the percentages is called sampling.

The top word can still be wrong

At temperature 0 the model takes the most likely word each time. If that top word is wrong, you get the same wrong answer every time you ask, worded the same way and equally confident. That is harder to spot than an answer that changes from run to run.

Try other prompts

The first two prompts are sentences. The third is the start of a line of Python code that looks up a user in a database, where the model has to choose between two database commands, get and filter. The fourth shows what a flat list of probabilities looks like.

A full answer is the same step repeated. The model adds the word it picked to the prompt and asks again, and it never goes back. A model that works this way is called generative. Press Next word and watch the list change each time.

You can now explain why a model writes one word at a time and why an error carries forward.

More detail

What changes in the loop

After the first word, the model reads its own output as part of the prompt. By the time the next word is chosen, the words it wrote are treated the same way as the words you wrote. A long answer of perhaps eight hundred words is built this way, one chosen word at a time. The model does not write a draft and revise it, and no separate step checks whether the sentence is true.

Why one early error can run through a whole answer

If the model picks a slightly wrong word early, every word after it is chosen to fit that word. The result reads smoothly to the end, because each word is chosen to be a likely continuation of the text before it.

Generative models and classifiers

In Training and test data, the boundary answered a closed question: which side is this student on? The answer was one choice from a set that existed before the model ran, and nothing new was made. A program that sorts inputs into fixed categories like this is called a classifier.

The model in this lesson answers an open question. The set of possible answers is every sequence of words there is. A generative model's output can be fluent, new and wrong at the same time. A classifier can only be wrong in the ways its fixed set of answers allows.

Image generators work differently

Image generators mostly start from random dots, called noise, and remove the noise in steps until a picture is left. That process is different from predicting words, and it fails in different ways. What you learn here about words does not explain why a generated hand has six fingers.

Students tend to assume that one mechanism sits behind every AI tool. Say out loud that this lesson applies to language models only.