How models actually learn, from vanilla gradient descent to Adam
Gradient accumulation simulates a larger batch when memory is limited. Instead of stepping after every micro-batch, you add gradients from several micro-batches, then take one optimizer step.
The effective batch size is the micro-batch size times the number of accumulation steps. This lets a small GPU behave more like it trained on a larger batch.
A rain barrel captures the idea. A small cup cannot water the whole garden at once, so you pour several cups into a barrel and then use the barrel amount. Gradient accumulation collects several small gradient contributions before one update. The figure below is literally this process: each new term is one cup, and the climbing bars are the barrel filling toward its total. Gradient accumulation is a partial sum of gradients that you cash in as a single step.