Ember: Asynchronous Dynamic Data Serving for PyTorch Distributed Training - Araújo, Patrick O. C.; Ramos, Fábio T.; Costa, Jhonata M. da; Drumond, Mario; Nacif, José Augusto M. (Anais do Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos (SBRC), 2026)
Advancements in natural language processing (NLP) and computer vision (CV) have led to substantial growth in data and model sizes, often requiring the distribution of the model and data across multiple GPUs and machines to accelerate training. Existing tools such as PyTorch Distributed Data-Parallel (DDP) operate at a low level of abstraction, requiring vast knowledge of distributed training. Therefore, users must adapt workflows to rigid tools or rebuild the entire code for each model and implementation, which can lead to performance issues. We introduce Ember, a customizable distributed training framework with new data distribution mechanisms. Ember utilizes remote procedure calls to decouple data loading logic from model synchronization, ensuring efficient computational resource utilization while maintaining accessibility and customization. To evaluate Ember’s performance, we compare its core training features with Ray, a powerful distributed framework, to ensure that our functions can be competitive with state-of-the-art implementations. The results of testing Ember use on datasets and models show that the framework achieves its goals, reducing GPU idle time and optimizing data transfer, achieving competitive training times with our baseline while utilizing its new asynchronous services with up to 36% less memory usage.