How models actually learn, from vanilla gradient descent to Adam
Mixed precision training uses smaller number formats for speed and memory. Instead of storing every calculation in full precision (standard 32-bit floats), many operations use float16 or bfloat16: 16-bit formats that take half the memory in exchange for less precision and, for float16, a narrower range of representable sizes.
The risk is numerical range. Some gradients are tiny. If a tiny number is rounded to zero, the optimizer loses information. Loss scaling protects those small gradients by multiplying the loss before backpropagation, then dividing the gradients back down.
A kitchen scale that rounds to whole grams can miss a tiny pinch of spice. If you weigh ten identical pinches together, the scale can see the total. Then you divide by ten to recover one pinch. Loss scaling uses the same trick: make the small value easier to represent, then scale it back. The figure below is a reminder of what is at stake. Descent only works if each step's gradient survives the arithmetic; precision does not change the loop, it decides whether the tiny slopes near the minimum are still visible to it.