How models actually learn, from vanilla gradient descent to Adam
Regularization is often introduced as a penalty added to the loss. Geometrically, it changes which parameter vectors are considered cheap or expensive. That changes the shape of the optimization problem. Two symbols recur below: R(θ) names the penalty term, and λ (lambda) sets how strongly it counts.
The two most common penalties behave differently: L2 discourages large weights smoothly, while L1 has corners that can push some weights exactly to zero.
Packing a suitcase with a strict weight limit has the same shape. Every item may help, but heavy items use up the budget quickly. Regularization makes large parameter choices use up budget, so the model keeps them only when they help enough. The figure shows why the budget is worth having: as model flexibility grows, training error keeps falling while validation error eventually turns back up. Regularization is the knob that reins flexibility in before that upturn arrives.