Matrix Norms

Geometry and algebra of linear maps, vectors, and matrices

Just as a vector has a length, a matrix has a "size." Two measures dominate, and they answer different questions: how big are the entries, versus how much can the matrix stretch a vector?

The Frobenius norm treats the matrix as one long list of numbers and takes the Euclidean length: square every entry, add, square-root. The spectral norm instead measures the maximum stretch, the largest factor by which A can lengthen any unit vector, which turns out to be the largest singular value.

Think of a matrix as a guitar amplifier: you feed in a signal and it comes out louder. The spectral norm is the amplifier's maximum gain, the biggest factor by which it can boost any input you send through. Turn the knob to its loudest setting and the loudest a unit signal can come out is exactly that norm.

Where this lives in MLThe Frobenius norm is L2 weight regularization for a whole matrix: penalizing ‖W‖_F² keeps the weights small and the model smooth. The spectral norm drives spectral normalization, which divides a weight matrix by its largest singular value to cap how much it can amplify. That makes it a key stabilizer in GANs and a tool for enforcing Lipschitz bounds.
▶ Matrix Norms
← Least SquaresProjections →