How a network computes, why gradients vanish, and what makes depth trainable
ReLU is the shortest activation there is: keep the input if it is positive, otherwise return 0. Written out, ReLU(x) = max(0, x). No exponential, no division, and no ceiling on its derivative. Swapping it in for the sigmoid is most of the reason deep networks became trainable.
🔒 This is a Pro lesson — the interactive figure, worked examples, quiz and practice open with Pro access.
▶ ReLU and the Dead-Unit Problem