How a model learns
A real training run drives the film: the model's error falls from 0.680 to 0.0059 in 60 steps.
The question
How does a machine-learning model learn? I wanted to show it in a film that works with the sound off.
A tiny model fits a line through 12 made-up dots, and a ball rolling into a valley shows each correction.
- The first guess0.680loss
The line starts at a slope of −0.60, tilted the wrong way. Its error (its loss) is 0.680.
- The swing, step 71.42slope
With momentum, the line swings past the best slope to 1.42, then back, like a weight on a spring.
- After 60 steps0.0059loss
The lowest possible for a straight line through these dots. Never zero: the dots don't sit on one line.
The numbers drive the film
Facts first, from Google's Machine Learning Crash Course. A real training run drives the film, so nothing moves that the numbers don't. It took 17 versions.
- Google, Machine Learning Crash Course: loss and gradient descent
- Gabriel Goh, "Why Momentum Really Works", Distill, 2017
- A small training script whose output drives every frame
- Designed and animated in code (Remotion)
One code base, two palettes
The film shipped on 4 November. A day later it was recoloured for the blog from the same code: one render setting switches the palette. Timing, copy, numbers and sound are unchanged.


It became a post and a tool

Self-initiated work. Sources: Google, Machine Learning Crash Course; Gabriel Goh, "Why Momentum Really Works", Distill, 2017. The dots are made up for the film; every number comes from the training run.
Next project