KV cache
KV cache: A temporary store of key and value tensors that avoids recomputing them for previous tokens during generation. It accelerates autoregressive decoding because each new token attends to the stored past. The cache grows with sequence length, batch size, layers and heads, often becoming the main memory constraint.
In italiano: KV CacheLegge III · Come una rete impara dai datiIt accelerates autoregressive decoding because each new token attends to the stored past. The cache grows with sequence length, batch size, layers and heads, often becoming the main memory constraint.
Examples
- Long-context chat
- Reusing prompt tokens
- Batches of generative requests