Vanishing Gradient

/ˈvænɪʃɪŋ ˈɡreɪdiənt/

Vanishing Gradient: A problem in deep networks where gradients become extremely small, preventing effective learning in early layers. Vanishing gradients occur when repeated multiplication of small derivatives (< 1) makes gradients exponentially smaller in backpropagation. Common in deep RNNs and networks with sigmoid activations.

In italiano: Vanishing GradientLegge V · Overfitting, bias e collasso dei modelli

Vanishing gradients occur when repeated multiplication of small derivatives (< 1) makes gradients exponentially smaller in backpropagation. Common in deep RNNs and networks with sigmoid activations. Solutions: ReLU, LSTM, ResNet.

Examples

  • Deep RNNs failing to learn long dependencies
  • Sigmoid activation in deep networks
  • Pre-ResNet very deep networks

Autovalutazione

Tre domande per verificare la comprensione della voce. Le spiegazioni restano disponibili anche senza JavaScript.

1. Che cos’è Vanishing Gradient?
Spiegazione

Vanishing Gradient: A problem in deep networks where gradients become extremely small, preventing effective learning in early layers. Vanishing gradients occur when repeated multiplication of small derivatives (< 1) makes gradients exponentially smaller in backpropagation. Common in deep RNNs and networks with sigmoid activations.

2. Quale applicazione è coerente con Vanishing Gradient?
Spiegazione

Un esempio è Deep RNNs failing to learn long dependencies.

3. Quale voce è collegata a Vanishing Gradient?
Spiegazione

Le voci correlate sono scelte per collegamento tecnico, non per somiglianza casuale.