Backpropagation
A network has to find out how much each of its weights contributed to a wrong answer. Backpropagation does this by tracing the error backward through the network.
A photo app puts a picture of your friend in someone else's album. You move it to the right album. The network that made the mistake has a huge number of weights in several layers. Each weight contributed a different amount to the wrong answer.
A slope tells a weight which way to move. Press Step to run the network forward, then follow the error backward until every weight has a slope. Play runs all the steps.
More detail
What the name means
In the forward pass, each layer hands its answer to the next layer. In the backward pass, each layer hands its share of the error to the layer behind it. Backpropagation means propagating, or passing along, the error backward.
The chain rule
The rule underneath is called the chain rule. You can use the idea without the formula: to know how much a weight mattered, multiply how much its output mattered by how much the weight affected that output.
Why a middle weight needs the layers after it
In How training works, each of the three numbers touched the answer directly, so finding its slope took one step of arithmetic. With hidden units that shortcut stops working, because a weight in the middle affects the answer only through the units after it.
Each number on the trace is built from a number already worked out one step earlier, so nothing is computed twice. Backpropagation gets every slope in one backward pass, which costs about the same as one forward pass. Nudging each weight and rerunning the network would need a run per weight, a billion for a real model.
You can now say why training a large network is feasible.
More detail
History
The idea was published in 1986 and the arithmetic has not changed since. Hardware improved later, and then the amount of text available to train on grew.
What you can skip
You can skip the derivation, the notation and the proof that the method is correct. You will not implement this, because every machine learning library does it for you. One common cause of a training run that produces nothing is that the error stopped passing backward somewhere, and people describe that problem in these words.