How models actually learn, from vanilla gradient descent to Adam
Sometimes parameters are not allowed to move anywhere. They must satisfy constraints: nonnegative weights, bounded norms, probabilities that must stay nonnegative and sum to 1 (a set called the probability simplex), fairness limits, safety limits, or physical feasibility.
Constrained optimization means minimizing the loss while staying inside the allowed set. One practical method is projected gradient descent: take a normal step, then project back to the feasible set.
A robot vacuum with boundary strips can try to move through a wall, but the boundary forces it back into the allowed room. Projected optimization works the same way. A gradient step may point outside, then projection clips the result back into the feasible region. The figure below shows the geometric heart of the operation: dragging a point down to its closest representative on an allowed set (there, a line). Projecting onto a box or a probability simplex uses the same closest-point principle with a different allowed set.