Linear Independence & Basis

Geometry and algebra of linear maps, vectors, and matrices

A set of vectors is linearly independent when no one of them is a combination of the others. Each pulls in a genuinely new direction, none is redundant. If you can write one as a combination of the rest, the set is dependent and contains slack.

The crisp test: the only way to make the zero vector from a combination is to use all-zero weights.

Think of a minimal Lego toolkit. A set of building blocks is linearly independent when every block adds a shape you could not have built from the others — none is redundant. If one block is really just a couple of the others snapped together, it is dead weight, and you could toss it without losing a single buildable shape. A basis is the leanest kit that still builds everything.

Where this lives in MLThis is the meaning of rank: the number of independent directions a matrix actually uses. If the rows of a weight matrix are dependent, some neurons are redundant. They compute combinations of the others and add no representational power. Low rank means a compressible layer (the idea behind LoRA), and a full-rank embedding table means every feature direction is genuinely distinct.
▶ Linear Independence & Basis
← Linear Combinations & SpanMatrices as Linear Maps →