How a network computes, why gradients vanish, and what makes depth trainable
The initialization you chose in dl.21 and dl.22 sets the scale of every layer's signal at step zero. It has no say after that. The optimizer moves the weights, the pre-activations move with them, and nothing in the network is watching. Train a small tanh net for 3000 steps and its third hidden layer's activation spread moves from 0.37 to 0.46, on a run that goes perfectly well. Nothing was holding that number anywhere. It moved because the weights did.
🔒 This is a Pro lesson — the interactive figure, worked examples, quiz and practice open with Pro access.
▶ Batch Normalization