Recurrent networks: the approach before transformers
A recurrent network reads text one word at a time and keeps a fixed-size summary of everything read so far. Until about 2017, language models worked this way.
You paste a long message into an AI chatbot and ask what "it" in the last sentence refers to. The chatbot names a noun from near the start of the message. To do that, it had to connect one word to another many words earlier.
After each word, the network rewrites its hidden state and discards the word. Press Read the next word and watch the hidden state. Each new word goes in at full strength and everything older fades. When the sentence is finished, press the button once more to ask what it refers to.
More detail
The hidden state has one fixed size
The hidden state stays the same size however long the sentence is. A short sentence fits comfortably. A paragraph does not, and nothing tells you which parts were lost.
Why the network is called recurrent
The same small step is applied over and over. Each time, it takes the previous hidden state and the next word and produces a new hidden state.
Why the answer is street
The two candidates for it are the nouns animal and street, because it stands for a thing. The hidden state holds street more strongly in both sentences, so the model answers street whichever last word it read.
Distance. Early words fade, so anything that depends on a connection across a long gap is lost. Order. Word fifty cannot be read until word forty-nine has been read, so training cannot be spread across many machines. A fifty-page contract shows both problems.
You can now name the two problems that led to transformers.
More detail
One fix for both problems
Keep every word, and let each word look directly at all the others, weighing them by how well they fit. Every word is then one step from every other word, and all the comparisons can happen at once. This approach is covered in Attention and transformers.