How a network computes, why gradients vanish, and what makes depth trainable
dl.21 balanced the weights against the layer's shape and quietly assumed the activation passed the signal through untouched. ReLU does not. optim.24 states the correction as Var(w) ≈ 2/fan-in and leaves the 2 unexplained. The 2 has a one-line reason, and it is worth having.
🔒 This is a Pro lesson — the interactive figure, worked examples, quiz and practice open with Pro access.
▶ Initialization II: He, Gain and Orthogonal