Limits

Single-variable calculus from first principles

A limit answers a careful question: as the input gets closer and closer to some value a, what number does the output home in on? Crucially, it doesn't matter what happens at a; maybe the function isn't even defined there. The limit is about the approach, not the destination itself.

Drag the input toward a in the figure and watch the output settle onto a value L, even across a little hole where the function has no value of its own.

You can approach a from the left (inputs a little below a) or from the right (a little above). These are the two one-sided limits. The full (two-sided) limit exists only when both sides agree on the same number. If the left side heads to one value and the right to another, there's a jump, and the limit doesn't exist.

Where this lives in MLLimits are the bedrock under derivatives (a limit of slopes) and integrals (a limit of sums), the two engines of training. They also formalise what "converges" means: a training loss converging to its floor is a limit. And the 0/0 traps you learn to defuse here are exactly the numerical-stability issues that bite in practice (e.g. computing log-of-softmax safely).
▶ Limits
← Reading GraphsContinuity →