Data parallelism

Data parallelism: A distributed strategy replicating the model and assigning different batches to devices before synchronising gradients. It increases throughput and effective batch size. Communication, sharding and synchronisation become decisive as model or cluster scale grows. This perspective connects the definition with practical behaviour in artificial intelligence systems.