Learning rate warmup

Learning rate warmup: An initial phase in which learning rate gradually increases to the value intended by the subsequent schedule. It reduces excessive early updates, especially in Transformers. Duration and shape affect stability and convergence speed. This perspective connects the definition with practical behaviour in artificial intelligence systems.