Tensor parallelism
Tensor parallelism: Partitioning operations and tensors within one layer across devices to distribute memory and computation. It requires frequent communication inside each layer. It helps models too large for one accelerator and combines with other strategies. This perspective connects the definition with practical behaviour in artificial intelligence systems.
In italiano: Tensor ParallelismLegge IV · Loss, gradiente e ottimizzatoriIt requires frequent communication inside each layer. It helps models too large for one accelerator and combines with other strategies.