Vanishing Gradient
/ˈvænɪʃɪŋ ˈɡreɪdiənt/
Vanishing Gradient: A problem in deep networks where gradients become extremely small, preventing effective learning in early layers. Vanishing gradients occur when repeated multiplication of small derivatives (< 1) makes gradients exponentially smaller in backpropagation. Common in deep RNNs and networks with sigmoid activations.
In italiano: Vanishing GradientLegge V · Overfitting, bias e collasso dei modelliVanishing gradients occur when repeated multiplication of small derivatives (< 1) makes gradients exponentially smaller in backpropagation. Common in deep RNNs and networks with sigmoid activations. Solutions: ReLU, LSTM, ResNet.
Examples
- Deep RNNs failing to learn long dependencies
- Sigmoid activation in deep networks
- Pre-ResNet very deep networks
Autovalutazione
Tre domande per verificare la comprensione della voce. Le spiegazioni restano disponibili anche senza JavaScript.