Pipeline parallelism
Pipeline parallelism: Distribution of layer groups across devices with micro-batches flowing through the pipeline in sequence. It reduces per-device memory but introduces bubbles and complex scheduling. Stage balance affects utilisation and throughput. This perspective connects the definition with practical behaviour in artificial intelligence systems.
In italiano: Pipeline ParallelismLegge IV · Loss, gradiente e ottimizzatoriIt reduces per-device memory but introduces bubbles and complex scheduling. Stage balance affects utilisation and throughput.