Mixed precision
Mixed precision: Training or inference that combines numerical formats of different precision to reduce memory and compute. Operations such as matrix multiplication often use FP16 or BF16, while accumulators and master weights retain higher precision. Loss scaling helps prevent gradient underflow in low-precision formats.
In italiano: Mixed PrecisionLegge IV · Loss, gradiente e ottimizzatoriOperations such as matrix multiplication often use FP16 or BF16, while accumulators and master weights retain higher precision. Loss scaling helps prevent gradient underflow in low-precision formats.
Examples
- BF16 on recent accelerators
- FP16 with a gradient scaler
- FP32 master weights