A Collective Communication Library for Distributed Tensors
Resumo
Tensors, represented as multidimensional arrays, are a fundamental abstraction in scientific computing and deep learning. This paper presents NxColl, a collective communication library for distributed Nx tensors in the Elixir programming language. NxColl generalizes the semantics of collective communication to support multidimensional tensors distributed across processes arranged in a Cartesian topology, while offering a declarative interface tailored to functional programming. A case study based on the CG kernel from the NAS Parallel Benchmarks (NPB) evaluates the proposed approach.
Palavras-chave:
Tensor, Collective Communication, High-Performance Computing, Functional Programming Language, Elixir, and Nx
Referências
J. Armstrong. 2007. A history of Erlang. In Proceedings of the Third ACM SIGPLAN Conference on History of Programming Languages (San Diego, California) (HOPL III). Association for Computing Machinery, New York, NY, USA, 6–1–6–26. DOI: 10.1145/1238844.1238850
D. H. Bailey and et al. 1991. The NAS Parallel Benchmarks. International Journal of Supercomputing Applications 5, 3 (1991), 63–73.
F. H. de Carvalho Junior and J. M. de Alencar. 2024. Cloud-based parallel computing across multiple clusters in Julia. In XXVIII Brazilian Symposium on Programming Languages (Curitiba, Brazil). SBC, Porto Alegre, RS, Brasil, 44–52. DOI: 10.5753/sblp.2024.3470
J. Dongarra. 2004. Trends in High Performance Computing. The Computer Journal 47, 4 (2004), 399–403.
J. Dongarra, S. W. Otto, M. Snir, and D. Walker. 1995. An Introduction to the MPI Standard. Technical Report CS-95-274. University of Tennessee. [link].
J. Dongarra, S. W. Otto, M. Snir, and D. Walker. 1996. A Message Passing Standard for MPP and Workstation. Communications of ACM 39, 7 (1996), 84–90.
P. Grinfeld. 2013. Introduction to Tensor Analysis and the Calculus of Moving Surfaces. Springer. DOI: 10.1007/978-1-4614-7867-6
Torsten Hoefler, Rolf Rabenseifner, Hubert Ritzdorf, Bronis R. de Supinski, Rajeev Thakur, and Jesper Larsson Träff. 2011. The Scalable Process Topology Interface of MPI 2.2. Concurrency and Computation: Practice and Experience 23, 4 (2011), 293–310. DOI: 10.1002/cpe.1643
Torsten Hoefler and Marc Snir. 2011. Generic Topology Mapping Strategies for Large-Scale Parallel Architectures. In Proceedings of the 25th ACM International Conference on Supercomputing (ICS’11). ACM, 75–84. DOI: 10.1145/1995896.1995909
S. Jurić. 2024. Elixir in Action (3 ed.). Manning Publications, Shelter Island, NY. [link]
Nicholas T. Karonis, Bronis R. de Supinski, Ian Foster, William Gropp, Ewing Lusk, and Sébastien Lacour. 2002. A Multilevel Approach to Topology-Aware Collective Operations in Computational Grids. CoRR cs.DC/0206038 (2002). arXiv:cs/0206038 [cs.DC] [link]
Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. 2025. TorchTitan: One-stop Py-Torch Native Solution for Production Ready LLM Pre-training. In The Thirteenth International Conference on Learning Representations (ICLR). [link]
MPI Forum. 2026. MPI Forum. [link]. Official website of the Message Passing Interface (MPI) Forum, accessed 2026-06-02.
Numerical Elixir (Nx). 2026. EXLA. [link]. Hex package for EXLA (Elixir bindings for XLA), accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Nx.BinaryBackend. [link]. HexDocs documentation for Nx.BinaryBackend, accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Nx.Defn. [link]. HexDocs documentation for Nx.Defn, accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Nx.indexed_add/4. [link]. HexDocs documentation for Nx.indexed_add/4 (Nx v0.11.0), accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Torchx. [link]. Hex package for Torchx (LibTorch bindings and backend for Nx), accessed 2026-06-02.
Numerical Elixir (Nx) Contributors. 2026. Nx: Multi-dimensional Arrays (Tensors) and Numerical Definitions for Elixir. [link]. GitHub repository, accessed 2026-06-01.
NVIDIA Corporation. 2024. NCCL: NVIDIA Collective Communications Library. [link].
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. 2018. Mesh-TensorFlow: Deep Learning for Supercomputers. In Advances in Neural Information Processing Systems 31 (NeurIPS). 10435–10444.
The Elixir Team. 2026. Elixir Programming Language. [link]. Official website, accessed 2026-06-01.
TOP500 Project. 2026. TOP500. [link]. Officialwebsite of the TOP500 project for ranking and statistics of the world’s most powerful supercomputers, accessed 2026-06-02.
Jesper Larsson Träff. 2002. Implementing the MPI Process Topology Mechanism. In Proceedings of the 2002 ACM/IEEE Conference on Supercomputing (SC’02). IEEE Computer Society Press, 1–14.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS). 5998–6008.
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, ShiboWang, TaoWang, YonghuiWu, and Zhifeng Chen. 2021. GSPMD: General and Scalable Parallelization for ML Computation Graphs. arXiv preprint arXiv:2105.04663 (2021).
Shiwei Zhang, Lansong Diao, Chuan Wu, Siyu Wang, and Wei Lin. 2022. Accelerating Large-Scale Distributed Neural Network Training with SPMD Parallelism. In Proceedings of the 13th ACM Symposium on Cloud Computing (SoCC ’22). Association for Computing Machinery, New York, NY, USA, 403–418. DOI: 10.1145/3542929.3563487
D. H. Bailey and et al. 1991. The NAS Parallel Benchmarks. International Journal of Supercomputing Applications 5, 3 (1991), 63–73.
F. H. de Carvalho Junior and J. M. de Alencar. 2024. Cloud-based parallel computing across multiple clusters in Julia. In XXVIII Brazilian Symposium on Programming Languages (Curitiba, Brazil). SBC, Porto Alegre, RS, Brasil, 44–52. DOI: 10.5753/sblp.2024.3470
J. Dongarra. 2004. Trends in High Performance Computing. The Computer Journal 47, 4 (2004), 399–403.
J. Dongarra, S. W. Otto, M. Snir, and D. Walker. 1995. An Introduction to the MPI Standard. Technical Report CS-95-274. University of Tennessee. [link].
J. Dongarra, S. W. Otto, M. Snir, and D. Walker. 1996. A Message Passing Standard for MPP and Workstation. Communications of ACM 39, 7 (1996), 84–90.
P. Grinfeld. 2013. Introduction to Tensor Analysis and the Calculus of Moving Surfaces. Springer. DOI: 10.1007/978-1-4614-7867-6
Torsten Hoefler, Rolf Rabenseifner, Hubert Ritzdorf, Bronis R. de Supinski, Rajeev Thakur, and Jesper Larsson Träff. 2011. The Scalable Process Topology Interface of MPI 2.2. Concurrency and Computation: Practice and Experience 23, 4 (2011), 293–310. DOI: 10.1002/cpe.1643
Torsten Hoefler and Marc Snir. 2011. Generic Topology Mapping Strategies for Large-Scale Parallel Architectures. In Proceedings of the 25th ACM International Conference on Supercomputing (ICS’11). ACM, 75–84. DOI: 10.1145/1995896.1995909
S. Jurić. 2024. Elixir in Action (3 ed.). Manning Publications, Shelter Island, NY. [link]
Nicholas T. Karonis, Bronis R. de Supinski, Ian Foster, William Gropp, Ewing Lusk, and Sébastien Lacour. 2002. A Multilevel Approach to Topology-Aware Collective Operations in Computational Grids. CoRR cs.DC/0206038 (2002). arXiv:cs/0206038 [cs.DC] [link]
Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. 2025. TorchTitan: One-stop Py-Torch Native Solution for Production Ready LLM Pre-training. In The Thirteenth International Conference on Learning Representations (ICLR). [link]
MPI Forum. 2026. MPI Forum. [link]. Official website of the Message Passing Interface (MPI) Forum, accessed 2026-06-02.
Numerical Elixir (Nx). 2026. EXLA. [link]. Hex package for EXLA (Elixir bindings for XLA), accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Nx.BinaryBackend. [link]. HexDocs documentation for Nx.BinaryBackend, accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Nx.Defn. [link]. HexDocs documentation for Nx.Defn, accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Nx.indexed_add/4. [link]. HexDocs documentation for Nx.indexed_add/4 (Nx v0.11.0), accessed 2026-06-02.
Numerical Elixir (Nx). 2026. Torchx. [link]. Hex package for Torchx (LibTorch bindings and backend for Nx), accessed 2026-06-02.
Numerical Elixir (Nx) Contributors. 2026. Nx: Multi-dimensional Arrays (Tensors) and Numerical Definitions for Elixir. [link]. GitHub repository, accessed 2026-06-01.
NVIDIA Corporation. 2024. NCCL: NVIDIA Collective Communications Library. [link].
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. 2018. Mesh-TensorFlow: Deep Learning for Supercomputers. In Advances in Neural Information Processing Systems 31 (NeurIPS). 10435–10444.
The Elixir Team. 2026. Elixir Programming Language. [link]. Official website, accessed 2026-06-01.
TOP500 Project. 2026. TOP500. [link]. Officialwebsite of the TOP500 project for ranking and statistics of the world’s most powerful supercomputers, accessed 2026-06-02.
Jesper Larsson Träff. 2002. Implementing the MPI Process Topology Mechanism. In Proceedings of the 2002 ACM/IEEE Conference on Supercomputing (SC’02). IEEE Computer Society Press, 1–14.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS). 5998–6008.
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, ShiboWang, TaoWang, YonghuiWu, and Zhifeng Chen. 2021. GSPMD: General and Scalable Parallelization for ML Computation Graphs. arXiv preprint arXiv:2105.04663 (2021).
Shiwei Zhang, Lansong Diao, Chuan Wu, Siyu Wang, and Wei Lin. 2022. Accelerating Large-Scale Distributed Neural Network Training with SPMD Parallelism. In Proceedings of the 13th ACM Symposium on Cloud Computing (SoCC ’22). Association for Computing Machinery, New York, NY, USA, 403–418. DOI: 10.1145/3542929.3563487
Publicado
08/09/2026
Como Citar
CARVALHO-JUNIOR, Francisco Heron de; MAGALHÃES, Francisco José Lins.
A Collective Communication Library for Distributed Tensors. In: SIMPÓSIO BRASILEIRO DE LINGUAGENS DE PROGRAMAÇÃO (SBLP), 30. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 150-158.
DOI: https://doi.org/10.5753/sblp.2026.29676.
