Learning-Rate Finder

How models actually learn, from vanilla gradient descent to Adam

A learning-rate finder is a short diagnostic run. Start with a tiny learning rate, increase it over many mini-batches, and watch how the loss responds.

You are not trying to finish training here; you are hunting for the range where the model starts learning before the loss becomes unstable.

Camera exposure bracketing uses the same idea. You take a sequence of shots from too dark to too bright, then pick the range where detail is clear. The finder does the same with training steps, sweeping η from timid to reckless and marking where learning turns crisp. You can act out a manual finder in the figure below: run with a small η, nudge it up, run again. Somewhere the smooth slide turns into overshooting and bouncing, and that is the blow-up edge the sweep looks for.

Where this lives in MLLearning-rate finders are useful when starting a new model or batch size. They turn “try random learning rates” into a quick measurement of where learning begins and where instability starts.
▶ Learning-Rate Finder
← Adam vs AdamWEarly Stopping →