Exploding Gradient
/ɪkˈsploʊdɪŋ ˈɡreɪdiənt/
Exploding Gradient: A problem where gradients become extremely large during training, causing unstable updates and divergence. Exploding gradients occur when repeated multiplication of large derivatives ( 1) makes gradients exponentially larger. This causes massive weight updates that destabilize training. Solutions: gradient clipping, proper initialization, batch normalization.
In italiano: Exploding GradientLegge V · Overfitting, bias e collasso dei modelliExploding gradients occur when repeated multiplication of large derivatives (> 1) makes gradients exponentially larger. This causes massive weight updates that destabilize training. Solutions: gradient clipping, proper initialization, batch normalization.
Examples
- RNN training divergence
- NaN values in weights
- Oscillating loss