Vectors and similarity of meaning

A vector is a list of numbers that gives a word a position on a plot. Words used in similar ways get nearby positions, so small distance stands for similar meaning.

You type "how do I sign up for a class" into the help search of the learning management system. The top result is a page called "Course enlistment process". The page title shares no word with your question.

Your questionTyped in the search box
The help pagesEach one has a title and text
The results listBest match first
Think first. No word matches, yet the search found the page. What could the computer store for each text so that it can tell the two are related?
Students tend to say a list of synonyms. Ask what in ordinary text could place two phrases close together without anyone writing a synonym list.

Each word here has two numbers, which give its position on the plot. Real models use hundreds of numbers per word. Each button adds the first word's position, subtracts the second and adds the third. The circle marked result is where the sum ends up, and the nearest word is the answer. These positions were placed by hand.

More detail

The rule that is often left out

The well-known answer appears only when the three input words are removed from the candidates. Turn the option off to see which word is nearest to the result.

Why the bias example is disputed

A 2016 paper by Bolukbasi and colleagues reported the programmer analogy using word2vec trained on Google News, and it became a standard illustration of bias in embeddings. Nissim and colleagues showed in 2020 that the analogy method produces part of the result. The search must exclude the input words, so it has to return something other than programmer, and a gendered-looking answer is partly a product of that rule.

What holds up is smaller. An embedding is built from a corpus, and patterns in that corpus become geometry. A prompt cannot remove that, because it is built into the positions the model uses. Anyone who claims one tidy number for how biased a model is should be asked how it was measured.

One vector per use of a word

On this plot, bank would get one point, placed between its river sense and its money sense. Newer models give a word a different vector depending on the sentence around it. That change is a main reason retrieval works as well as it does today.

Both statements hold, and students tend to keep only one of them: embeddings do absorb the statistics of their corpus, and the famous analogies overstate how cleanly that shows up.

A sentence also gets a position, worked out from its words. How alike two sentences are depends on the angle between their arrows. Pick two sentences and compare the similarity score with the number of words they share.

Similarity
—
Words in common
—
Pick two sentences
More detail

What the score is

The score is the cosine of the angle between the two arrows. It is 1 when the arrows point the same way and 0 when they are at right angles. It measures direction only, so a long sentence and a short one can score highly when they are about the same thing.

The help search turns your question and every help page into positions. A keyword search would find nothing useful, because the question and the title share no words. The two positions point almost the same way. Finding the text with the nearest position, called retrieval, returns the enlistment page.

You can now explain how a search by meaning finds a page that shares no words.

More detail

Retrieval in AI assistants

Retrieval happens when you point an AI assistant at your documents. A student asking how to stop a double enlistment can be matched to a page about idempotency keys, which stop a repeated request from being processed twice, without the two sharing a single word.

The reverse case

How do I drop a class shares five words with the sign-up question and means the opposite, and the score reflects that. Two sentences can share many words and mean opposite things.