Recurrent networks: the approach before transformers

A recurrent network reads text one word at a time and keeps a fixed-size summary of everything read so far. Until about 2017, language models worked this way.

You paste a long message into an AI chatbot and ask what "it" in the last sentence refers to. The chatbot names a noun from near the start of the message. To do that, it had to connect one word to another many words earlier.

Your long messageMany words in order
The language modelReads the message
Its answerA noun from the message
Think first. In The animal didn't cross the street because it was too tired, what does it refer to? What must the program still have when it reaches it?
Ask which words a program would still have if it kept only a short summary. Most students say the most recent ones, which the widget confirms.

After each word, the network rewrites its hidden state and discards the word. Press Read the next word and watch the hidden state. Each new word goes in at full strength and everything older fades. When the sentence is finished, press the button once more to ask what it refers to.

What is left in the hidden state
More detail

The hidden state has one fixed size

The hidden state stays the same size however long the sentence is. A short sentence fits comfortably. A paragraph does not, and nothing tells you which parts were lost.

Why the network is called recurrent

The same small step is applied over and over. Each time, it takes the previous hidden state and the next word and produces a new hidden state.

Why the answer is street

The two candidates for it are the nouns animal and street, because it stands for a thing. The hidden state holds street more strongly in both sentences, so the model answers street whichever last word it read.

Distance. Early words fade, so anything that depends on a connection across a long gap is lost. Order. Word fifty cannot be read until word forty-nine has been read, so training cannot be spread across many machines. A fifty-page contract shows both problems.

You can now name the two problems that led to transformers.

More detail

One fix for both problems

Keep every word, and let each word look directly at all the others, weighing them by how well they fit. Every word is then one step from every other word, and all the comparisons can happen at once. This approach is covered in Attention and transformers.