How a network computes, why gradients vanish, and what makes depth trainable
Two complaints you will hear from people training deep networks. The first: the loss drops for a while, then flattens, and when they print the first layer's weights after an hour those weights are still the numbers they were initialized with. The second: it trains normally for a few hundred steps and then every number in the model is NaN.
🔒 This is a Pro lesson — the interactive figure, worked examples, quiz and practice open with Pro access.
▶ Vanishing and Exploding Gradients