Weight Initialization
/weɪt ɪˌnɪʃəlaɪˈzeɪʃən/
Weight Initialization: Methods for setting initial values of neural network weights before training begins. Proper initialization is crucial for effective training. Random initialization breaks symmetry. Methods like Xavier/Glorot (for tanh/sigmoid) and He initialization (for ReLU) ensure gradients flow well initially.
In italiano: Inizializzazione PesiLegge IV · Loss, gradiente e ottimizzatoriProper initialization is crucial for effective training. Random initialization breaks symmetry. Methods like Xavier/Glorot (for tanh/sigmoid) and He initialization (for ReLU) ensure gradients flow well initially.
Examples
- Xavier initialization for tanh layers
- He initialization for ReLU layers
- Zero initialization for biases