Weight Initialization

/weɪt ɪˌnɪʃəlaɪˈzeɪʃən/

Weight Initialization: Methods for setting initial values of neural network weights before training begins. Proper initialization is crucial for effective training. Random initialization breaks symmetry. Methods like Xavier/Glorot (for tanh/sigmoid) and He initialization (for ReLU) ensure gradients flow well initially.

In italiano: Inizializzazione PesiLegge IV · Loss, gradiente e ottimizzatori

Proper initialization is crucial for effective training. Random initialization breaks symmetry. Methods like Xavier/Glorot (for tanh/sigmoid) and He initialization (for ReLU) ensure gradients flow well initially.

Examples

  • Xavier initialization for tanh layers
  • He initialization for ReLU layers
  • Zero initialization for biases