Brief: Vector Spaces of Functions

Single-variable calculus from first principles

Functions behave like vectors. You already know you can add two arrows and stretch an arrow by a number. You can do the very same two things to functions, and almost everything you know about vectors carries straight over.

To add two functions, you add them pointwise: at every input x, the new function's output is just the sum of the two outputs. To scale a function by a number c, you multiply every output by c. Those two operations are exactly what makes something a "vector space."

Think of two audio tracks playing at once: a bassline and a melody. To mix them you add the two waveforms moment by moment, exactly like adding functions pointwise. And turning one track's volume knob to 70% is just scaling that function by 0.7 at every instant. Mixing and volume are addition and scaling, the two moves that make functions behave like vectors.

Where this lives in MLA linear layer outputs a weighted sum of basis features: exactly "c₁·f₁ + c₂·f₂ + …" with learned weights. Fourier features, polynomial features, and the hidden units of a network are all bases you combine to span a space of functions. When people say a network is a "universal approximator," they mean its building blocks span a rich enough function space to get close to almost anything.
▶ Brief: Vector Spaces of Functions
← Trigonometric FunctionsTransformations →