How a network computes, why gradients vanish, and what makes depth trainable
Everything is now in place. dl.12 gives the number the backward pass starts from. dl.14 gives one block, from the gradient at its output to the gradient at its input. A network is blocks stacked, so backpropagation is dl.14 run once per layer, from the top down. This lesson is the whole algorithm.
Think of a complaint travelling back down a supply chain. A customer reports a fault in the finished product. The final assembler works out how much of the fault its own machines caused, keeps that, and passes the remainder back to the supplier that shipped it the parts. That supplier does exactly the same, and so on down the chain. Nobody needs to understand the whole chain. Each stage needs its own local answer and the message it was handed.
Number the layers 1 to L, with zˡ = Wˡaˡ⁻¹ + bˡ and aˡ = φ(zˡ), where a⁰ is the input and aᴸ = ŷ. The backward pass starts at the top: