Paged attention
Paged attention: Managing the KV cache in non-contiguous blocks to reduce fragmentation and memory waste. Like virtual memory, a table maps logical and physical blocks during generation. This facilitates sharing and scheduling of variable-length sequences. Examples include KV-cache blocks, Prefix sharing, Serving with continuous batching.
In italiano: Paged AttentionLegge IV · Loss, gradiente e ottimizzatoriLike virtual memory, a table maps logical and physical blocks during generation. This facilitates sharing and scheduling of variable-length sequences.
Examples
- KV-cache blocks
- Prefix sharing
- Serving with continuous batching