Performance Evaluation of Distributed Deep Learning Training on Low-Bandwidth Networks - Matos, Rayan Raddatz de; Gulart, Marcelo Cardoso Oliveira; Brumato, Kenichi; Schnorr, Lucas Mello (Anais da Escola Regional de Alto Desempenho da Região Sul (ERAD-RS), 2026)
The growing complexity of Deep Learning models makes training on single devices difficult, requiring distributed strategies like Data Parallelism. However, this approach introduces critical synchronization challenges between processing units. This work evaluates TensorFlow parallel strategies on an HPC cluster with low-bandwidth network to analyze how application factors contribute to time reduction. The study compares the standard synchronous method against a proposed Custom Training Loop with gradient accumulation designed to reduce the bottleneck by synchronizing gradients less frequently. Our results indicate that optimizing synchronization intervals and batch sizes allows for scalable performance gains even on low-bandwidth infrastructure.