Scalable GPU Communication with Code Generation on Stencil Applications

• João Victor Tozatti Risso UFPR
• Martin Bauer University of Erlangen-Nürnberg
• Paulo Roberto de Carvalho UFPR
• Ulrich Rüde University of Erlangen-Nürnberg
• Daniel Weingaertner University of Erlangen-Nürnberg

Resumo

Clusters with GPUs are mainstream in HPC as shown by the last edition of the Top500 list, increasing the demand for GPU capable scientific computing software. Programming large scale GPU systems in an efficient and future-proof way present numerous challenges, such as optimizations for a variety of GPUs and interconnect hardware, hiding communication overhead with computation and efficient domain partitioning. We present an improvement to the CUDA-based communication of stencil applications in the WALBERLA framework, achieving scalability while supporting different GPUs and communication infrastructures. We utilize the lattice Boltzmann Method for fluid flows as a representative of stencil-based scientific computing and implement a communication hiding strategy that is capable of adjusting to a system's computing and communication capabilities. We compare the use of CUDAMemCopy with the use of customized pack/unpack kernels and show that packing achieves almost linear weak scaling behavior in the Santos Dumont supercomputer with up to 128 GPUs. We also show that the proposed approach is not sensitive to the direction of the domain partitioning, one of the biggest challenges when communicating 3D domains in GPU-based stencil simulations.

Palavras-chave: Graphics processing units, Kernel, Computational modeling, Lattice Boltzmann methods, Optimization, Hardware, Layout, GPU clusters, Lattice Boltzmann Method, GPU communication, scalability