Differentiability

Single-variable calculus from first principles

A function is differentiable at a point if it has a single, well-defined slope there: one tangent line, no ambiguity. Most smooth curves are differentiable everywhere. But some functions, while perfectly continuous, have a spot where the slope simply can't be pinned down. Understanding where derivatives fail is as important as computing them.

If a function has a slope at a point, it can't have a jump there, so differentiable ⇒ continuous. The reverse is false: a function can be continuous (drawable without lifting your pen) yet still fail to have a slope at a point. The gap between "continuous" and "differentiable" is exactly the interesting part.

The absolute value |x| is the standard example. It's continuous everywhere, with no break at 0. But right at the corner, the slope coming in from the left is −1 and the slope leaving to the right is +1. Two different slopes meet at a sharp point, so there's no single tangent. The derivative does not exist at x = 0.

Where this lives in MLReLU, the most common activation, is literally max(0, x): a corner at 0, just like |x|. Its derivative is undefined right at 0, so frameworks simply pick a value (usually 0), called a "subgradient." The corners of ReLU, the kinks of L1 regularisation, and the non-smoothness of the hinge loss are all places where this exact issue shows up and gets handled with subgradients.
▶ Differentiability
← The DerivativeBasic Derivative Rules →