Continuous batching
Continuous batching: Scheduling that adds and removes requests during generation instead of waiting for the whole batch. Iterations combine sequences at different stages and reduce accelerator idle time. Priorities and memory limits prevent long requests from penalizing all others. Examples include LLM server with concurrent requests, Dynamic per-iteration batching, Throughput optimization.
In italiano: Continuous BatchingLegge IV · Loss, gradiente e ottimizzatoriIterations combine sequences at different stages and reduce accelerator idle time. Priorities and memory limits prevent long requests from penalizing all others.
Examples
- LLM server with concurrent requests
- Dynamic per-iteration batching
- Throughput optimization