Neural networks: combining several lines
A neural network combines several simple cuts into one decision. This lesson uses students who drop a course, a case where no single straight line works.
You open a photo app and tap a person in its People album. Every picture of that person appears, in different light, from different angles, with and without glasses. The app works from the pixels of each picture, which are only numbers for color. A neural network combines many measurements of those numbers into one decision.
Students drop when they study very little and when they carry far too many other commitments, so the ones who stay sit in the middle. Move the slider to add units and watch straight lines mark out a curved region. Then switch off the nonlinearity and move the slider again.
More detail
The diagram and the plot
Most people meet neural networks as a diagram of circles and arrows. The diagram shows the wiring, meaning which number is multiplied by which weight and passed on to where. The plot shows the effect, meaning what that wiring does to the space of possible students. Both are correct, and seeing only the diagram can make a network look more mysterious than it is.
How the layers combine
A unit is one line. A layer is a set of units. A deeper network has more layers, and each layer works on the answers of the layer before it. Nobody draws the shape on the plot by hand. Training moves each line until the combined shape makes fewer mistakes.
Why the nonlinearity matters
With the nonlinearity off, a layer can only add its units together, and a sum of straight lines is another straight line. A deeper network without a nonlinearity gains nothing, because eight units build the same kind of line as two. That small step between the layers is the reason any of this works.
When more units stop helping
Pushing the slider higher stops helping once the plot is separated. Extra capacity that you do not need is how you get the bent line in Training data and test data.
Each straight line is a unit. The set of units is a layer, and the output unit says "stays" only when every line agrees. The layer is called a hidden layer because nobody shows you what its units decided. Training moved their lines.
More detail
Where the terms come from
One line fitted to data is logistic regression, which the lesson Training data and test data covered. Once you use several lines and combine their answers, you have a neural network with one hidden layer. A unit is sometimes called a neuron.
Units on the plot
One circle in the middle row of the diagram is one line on the plot. The weights on the two arrows entering a circle set that line's angle, and its bias sets how far from the center it sits. That is three numbers for each unit, as in How training works.
Deeper networks
Deeper networks stack more layers. Each layer works on what the previous layer found, so the shapes become more elaborate while every piece is still a straight cut. A language model does this with billions of numbers. Attention, covered in Attention and transformers, is one particular kind of layer.
The features are the measurements a model works from.
A person picks them, so a wrong one can be pointed at.
Early layers work out their own from raw input.
The cost is that a real model has billions of numbers, so it can be accurate and still impossible to explain.
You can now explain what a hidden layer does and why deep models are hard to explain.
More detail
Before deep learning
To spot a likely dropout, an analyst would decide what to measure: hours studied, attendance, or whether the student had missed two submissions in a row. Getting those right was the main job, and it needed someone who knew the subject area.
After deep learning
You give the network raw input and enough units, and its early layers work out their own features. In an image model, the first layers end up finding edges, later layers find shapes, and later still layers find faces. Nobody told the network to look for edges.
The cost: nobody can say what each unit means
The few numbers on this screen can still be read. A real model has billions of numbers, and when it goes wrong there is often nothing to point at. The same property that lets a model find features nobody named also lets it pick up patterns nobody intended. The lesson Word embeddings returns to where meaning comes from.