Speculative decoding
Speculative decoding: Decoding that uses a smaller model to propose tokens and a main model to verify them in parallel. Accepted sequences preserve the main model’s distribution when the method is correctly applied. Gains depend on acceptance rate and draft cost.
In italiano: Speculative DecodingLegge IV · Loss, gradiente e ottimizzatoriAccepted sequences preserve the main model’s distribution when the method is correctly applied. Gains depend on acceptance rate and draft cost.
Examples
- Lightweight draft model
- Verifying multiple tokens in one step
- Latency reduction