The math behind it: vectors, slopes and probabilities

Three calculations appear in every lecture on language models: a weighted sum, a slope and a probability distribution. Each one is school arithmetic.

A chatbot's model does three calculations when it answers. It adds weighted numbers. For example, scores of 80 and 90 with weights 0.6 and 0.4 give a course grade of 80 × 0.6 + 90 × 0.4 = 84. It finds a slope, which says which way to adjust a number. It turns scores into probabilities that add to 1.

Weighted sumDot product
Which way to adjustSlope
Chance of each outcomeProbability distribution
Think first. A course weights exams at 0.7 and projects at 0.3. You score 60 on exams and 100 on projects. What is your grade?
The answer is 72. Students who answer 80 averaged the two scores and ignored the weights. Ask why 72 is closer to the exam score.

Any list of numbers can be drawn as an arrow. Arrow a stays fixed and you swing arrow b. The dot product is a weighted sum of the two lists. Drag the slider and watch it change.

Dot product
—
More detail

Hundreds of numbers

Two numbers can mark a position on a page. Three hundred numbers mark a position in a space you cannot draw, which is where words are placed in Vectors and similarity.

Where you have met it

The attention weight in Attention and transformers is a dot product between what one word is asking for and what another word offers. The similarity score in Vectors and similarity is the same operation with the lengths divided out, which is called cosine similarity. A unit in Neural networks multiplies each input by a weight and adds the results, then adds a fixed number called a bias.

Matrices and the notation Wx

A matrix is a grid of numbers, which you can read as a stack of vectors. Multiplying by a matrix performs many dot products at once. This is the meaning of Wx: W is the matrix of weights and x is the input vector.

A weight is a number the model can adjust. The curve shows how wrong the model is at each weight. The slope at your position says which way lowers the error. Drag the slider along the curve and watch the slope.

How wrong it is
—
Slope
—
More detail

The weight

The weight
—

Slope and derivative

A derivative is the formal name for the slope at one point. The slope is negative on the left of the dip, positive on the right and zero at the bottom. To lower how wrong the model is, move in the direction opposite to the slope.

Gradient

A gradient is the slope worked out for every number at once. With three numbers it is three slopes. With a billion numbers it is a billion slopes, and the arithmetic for each one is the same.

Where you have met it

The Step button in How training works does exactly this. It works out the slope for each of the three numbers, moves each one against its slope and repeats. The learning-rate slider sets how far each move goes.

Given "The capital of France is", a model gives every possible next word a probability. Together those probabilities form a probability distribution, and they add up to 100%. Raise the probability of Paris and watch the other words shrink.

Adds up to
—

You can now read a weighted sum, a slope and a probability distribution.

More detail

Loss

A loss is one number that says how wrong the model was, given what turned out to be true. The true answer here is Paris. The loss is small when the model put most of its probability on Paris and large when it put little there.

Loss
—

The formula

The formula is -log(p), where p is the probability given to the right answer and log is the logarithm function on a calculator. This loss is called cross-entropy. It rises steeply as the probability approaches zero.

Where you have met it

This is the number that falls in How training works. Temperature, in Predicting the next word, changes the shape of the distribution the model samples from. The loss measures how much probability sits on the right answer.

What you can skip for now

This lesson leaves out matrix decompositions, the mechanics of the chain rule (how slopes combine through a chain of steps), probability theory beyond distributions, and proofs. The three ideas here are enough to read a diagram in a lecture and recognize which symbol is doing the work.