How a network computes, why gradients vanish, and what makes depth trainable
optim.3 introduces warmup as part of a recipe: start with a small learning rate, ramp it up over a few thousand steps, then decay. Written like that it sounds like a preference, the kind of thing one lab does and another does not.
๐ This is a Pro lesson โ the interactive figure, worked examples, quiz and practice open with Pro access.
โถ Why Warmup Is Not Optional With Adam