Gradient accumulation
Gradient accumulation: Accumulating gradients over several mini-batches before updating weights, emulating a larger batch. It lowers memory requirements because each mini-batch is processed separately. The number of accumulation steps changes the effective batch size and requires consistent loss scaling, learning rate and scheduling.
In italiano: Gradient AccumulationLegge IV · Loss, gradiente e ottimizzatoriIt lowers memory requirements because each mini-batch is processed separately. The number of accumulation steps changes the effective batch size and requires consistent loss scaling, learning rate and scheduling.
Formula
Effective batch = device batch × accumulation steps × devices
Examples
- Eight mini-batches of 4 for an effective batch of 32
- Training LLMs with limited memory
- Loss scaling for accumulation