How a language model writes: predicting the next word
A language model writes by choosing one word at a time. It repeats the same step for every word it adds.
You type a message on your phone. Three words appear above the keyboard. You tap one, and three new words appear. The keyboard picks them from the words you have typed so far. The language model behind an AI chatbot also picks each next word from the text so far.
The model gives every candidate word a probability, then picks one. Press Pick one a few times. Then set the temperature to 0, press Do it ten times and compare the two tallies.
More detail
What temperature does
Temperature controls how much randomness goes into the choice. At 0 the model takes the top word every time. At 1 it picks at random, in proportion to the percentages shown. Picking at random in proportion to the percentages is called sampling.
The top word can still be wrong
At temperature 0 the model takes the most likely word each time. If that top word is wrong, you get the same wrong answer every time you ask, worded the same way and equally confident. That is harder to spot than an answer that changes from run to run.
Try other prompts
The first two prompts are sentences. The third is the start of a line of Python code that looks up a user in a database, where the model has to choose between two database commands, get and filter. The fourth shows what a flat list of probabilities looks like.
A full answer is the same step repeated. The model adds the word it picked to the prompt and asks again, and it never goes back. A model that works this way is called generative. Press Next word and watch the list change each time.
You can now explain why a model writes one word at a time and why an error carries forward.
More detail
What changes in the loop
After the first word, the model reads its own output as part of the prompt. By the time the next word is chosen, the words it wrote are treated the same way as the words you wrote. A long answer of perhaps eight hundred words is built this way, one chosen word at a time. The model does not write a draft and revise it, and no separate step checks whether the sentence is true.
Why one early error can run through a whole answer
If the model picks a slightly wrong word early, every word after it is chosen to fit that word. The result reads smoothly to the end, because each word is chosen to be a likely continuation of the text before it.
Generative models and classifiers
In Training and test data, the boundary answered a closed question: which side is this student on? The answer was one choice from a set that existed before the model ran, and nothing new was made. A program that sorts inputs into fixed categories like this is called a classifier.
The model in this lesson answers an open question. The set of possible answers is every sequence of words there is. A generative model's output can be fluent, new and wrong at the same time. A classifier can only be wrong in the ways its fixed set of answers allows.
Image generators work differently
Image generators mostly start from random dots, called noise, and remove the noise in steps until a picture is left. That process is different from predicting words, and it fails in different ways. What you learn here about words does not explain why a generated hand has six fingers.