How models actually learn, from vanilla gradient descent to Adam
Adam and AdamW differ in how they handle weight decay. Adam mixes an L2 penalty into the adaptive gradient update. AdamW applies weight decay as a separate shrinkage step.
That separation matters because Adam rescales gradients per parameter. If weight decay is mixed into those gradients, the regularization is also rescaled in a parameter-dependent way.
Think of a library that wants books returned on time. A flat fine per late day is easy to reason about: it applies the same way to every borrower. Folding the fine into each borrower's personalized membership formula makes the penalty uneven and hard to predict. AdamW is the flat separate fine; Adam with coupled L2 folds the penalty into its per-parameter machinery first. And why penalize at all? The figure has the answer: growing flexibility keeps lowering training error while validation error eventually turns upward. Weight decay is one of the main tools for staying near the sweet spot.