Word embeddings: how words get positions

A word's embedding is a list of numbers that gives it a position on a plot. The positions come from reading sentences, and words that appear together end up close together.

You upload a reading to an AI chatbot and ask, "Where does it talk about cutting costs?" The chatbot points to a paragraph about reducing expenses. That paragraph contains neither cutting nor costs.

Your questionTyped in the chat
Your uploaded readingThe text it searches
A position for each wordIts embedding
Think first. What would the program need to know about cutting and reducing, and where could it learn that?
Collect guesses before anyone presses Run the training. Most students say a dictionary or a person who labeled the words. The next steps show that sentences alone are enough.

Every word starts at a random spot. Each sentence moves its words a little closer together and pushes all other words slightly apart. Press Run the training and watch the groups form.

More detail

The two gauges

Sentences read counts the sentences used so far. Groups apart by compares two averages: the distance between words from different groups, and the distance between words in the same group.

Sentences read
0
Groups apart by
—

Each step is small and fixed

Each sentence moves the words by the same small step, so no word chooses where it goes. The plot has two directions, called dimensions, so each embedding is two numbers. Real models use hundreds of numbers per word.

Look at the list. These fourteen sentences, called the corpus, were the only input. There is no dictionary and there are no labels. Nobody marked which words are about royalty. The colors on the plot were added afterward to show the groups.

The whole corpus

You can now explain where word positions come from.

More detail

How this differs from real training

This is the idea behind a method called word2vec, run small enough to watch. Real training uses billions of words and hundreds of dimensions, where this demo uses fourteen sentences and two. In real text each word also appears in far more varied sentences.

A corpus this small produces the clusters. It cannot produce the finer structure behind the word arithmetic in Vectors and similarity, because that needs the same word in thousands of different contexts.