Weight decay
Weight decay: A regularization technique that penalizes large parameter values during model optimization. In the L2 formulation it adds a penalty proportional to squared weights; in decoupled AdamW it modifies the update directly. Its coefficient controls the trade-off between data fit and model capacity.
In italiano: Weight DecayLegge IV · Loss, gradiente e ottimizzatoriIn the L2 formulation it adds a penalty proportional to squared weights; in decoupled AdamW it modifies the update directly. Its coefficient controls the trade-off between data fit and model capacity.
Formula
L = L_data + (λ/2)‖w‖₂²
Examples
- AdamW with a 0.01 coefficient
- L2 regularization of a network
- Reducing overfitting