How models actually learn, from vanilla gradient descent to Adam
Momentum gives gradient descent memory. Instead of using only the current gradient, it keeps a moving average of recent gradients and steps in that accumulated direction.
This helps in two ways: it smooths noisy gradients, and it builds speed along directions where gradients keep agreeing. Across a narrow ravine, alternating sideways gradients cancel; along the useful direction, repeated gradients add.
A bowling ball does not forget the last push. One push starts it moving, and repeated pushes in the same direction build speed. Small side nudges do not instantly reverse it. Momentum makes optimization behave less like separate steps and more like motion with inertia. See it happen below: run plain descent first with β = 0, then raise β and run again. The side-to-side ricochet fades and the path gathers speed along the valley.