Training data and test data

A model is built from examples and then judged on examples it has not seen. This lesson fits a line to some students and tests it on others.

You open a streaming app and a row of recommended titles appears. You watch one, close the app and come back later. The row has changed. A model built from the viewing records of many people chose those titles. The team that built it has to say how well it works.

Records used to build itTraining set
The recommendation modelThe fitted model
A viewer it has not seenTest set
Think first. The builders must report how accurate the picks are. Should they score the model on the viewing records it was built from, or on other records? Why?
Let students propose a fair test before they open the next step. Most will suggest records the model has not seen. Name that idea as a held-out test set and connect it to the five dashed students in the widget.

The widget follows 17 students who either finished a course or did not. Press Next to set five aside, fit a line to the other twelve and test it on the five. Then switch between the straight line and the bent line and compare the scores. A line that fits its examples too closely shows overfitting.

Training set, the twelve
—
Test set, the five
—
Gap
—
What is happening
More detail

Jump to a stage

What this kind of model is called

A straight boundary fitted to labeled examples is called logistic regression. The name confuses many people, because regression usually means predicting a number, and this model predicts a category.

Linear regression draws a line through the points to predict a value, such as the mark a student will get from the hours studied. Logistic regression draws a line between the points to predict a group, such as pass or fail. On screen both are a line. The first answers "how much?" and the second answers "which group?". The later lessons use the second kind.

How a model becomes a boundary

Nobody writes the line down. It starts in a poor position and is adjusted until it makes fewer mistakes. Adjusting a model to match examples is called fitting it, and the final line is the model.

Split the data before fitting, and only once. If you look at the test set, adjust something and test again, you have trained on it. That mistake is called leakage.

Training set

Data the model sees while it is fitted. A good score here means little.

Test set

Data set aside first and never looked at. Report this score.

You can now say why a score on training data proves little.

More detail

Why a good training score means little

A model that has seen every answer can remember them, and from the inside remembering looks the same as learning. Every adjustment is made to reduce mistakes on the training examples, so a good score there is expected.

Why leakage is easy to miss

People make this mistake often without noticing, and results that looked strong during development then fall apart in front of real users. Information from the test set leaks into the model, which is where the name comes from.

Reading the gap

A small gap between the two scores means the model found the general shape of the data. A large gap means it memorized its examples. The bent line in "Fit a line and test it" shows this directly. A test set is commonly 20 to 30 percent of the data.