Symmetric Matrices

Geometry and algebra of linear maps, vectors, and matrices

Symmetric matrices (A = Aᵀ) are unusually well-behaved, and they happen to be the ones that show up most in ML. Covariance matrices, Hessians, Gram matrices: all symmetric. They come with a guarantee clean enough to have a name.

The spectral theorem: every real symmetric matrix has real eigenvalues and a full set of orthogonal eigenvectors. No complex numbers, no defective cases, and the eigen-directions meet at perfect right angles. You can always diagonalize it with an orthogonal matrix.

Because Q is orthogonal, Q⁻¹ = Qᵀ, so the decomposition is built from a rotation, a scaling, and the reverse rotation. The eigenvectors give you a perfect orthonormal coordinate system, handed to you for free.

Where this lives in MLThe Hessian of a loss is symmetric (mixed partials commute), so its eigenvalues are real and tell you the curvature in each direction: all positive ⇒ a local minimum (a bowl), mixed signs ⇒ a saddle. Covariance matrices are symmetric and positive semi-definite, which is exactly why PCA's eigen-decomposition always yields real, orthogonal principal directions with non-negative variances.

▶ Symmetric Matrices

← Diagonalization SVD →