How a network computes, why gradients vanish, and what makes depth trainable
ReLU makes a hard decision about every number that reaches it: positive, keep it exactly; negative, delete it. GELU and SiLU replace that switch with a dial. Rather than keeping or deleting the input, they keep a fraction of it, and the fraction is how confidently positive the input is.
🔒 This is a Pro lesson — the interactive figure, worked examples, quiz and practice open with Pro access.
▶ GELU, SiLU and Smooth Gates