How models actually learn, from vanilla gradient descent to Adam
Full-batch gradient descent uses every training example to compute each update; stochastic gradient descent goes to the other extreme and uses just one. Mini-batch gradient descent sits in between with a small batch, and that compromise is what deep learning actually runs on.
A mini-batch gradient is a noisy estimate of the full gradient. It is cheaper and often more useful than the exact full gradient because it gives many quick updates and its noise can help exploration.
Cereal quality checks use the same compromise. Opening every box is accurate but slow. Checking one box is noisy. Checking a tray of boxes gives a useful estimate quickly. Mini-batches are those trays. The figure below makes the statistics visible: press Run and watch a running average settle down as more samples arrive. A mini-batch gradient is the same kind of object, an average that steadies as the batch grows.