Depth Is a Product of Jacobians

How a network computes, why gradients vanish, and what makes depth trainable

A gradient does not travel through a deep network. It is multiplied by every layer it passes, once per layer, and what arrives at the front is the product of all those multiplications. Nothing is added along the way.

๐Ÿ”’ This is a Pro lesson โ€” the interactive figure, worked examples, quiz and practice open with Pro access.

โ–ถ Depth Is a Product of Jacobians
โ† The Tape: Memory, Recomputation and CheckpointingVanishing and Exploding Gradients โ†’