A Software Defined Network Platform for Converged Parallel Computing Environments

• Alexandre T. Oliveira Universidade Federal de Juiz de Fora
• Alex B. Vieira Universidade Federal de Juiz de Fora
• Antônio Tadeu A. Gomes LNCC
• Artur Ziviani LNCC

DOI: https://doi.org/10.5753/sbrc.2019.7417

Abstract

The growth in data volume has revolutionized business and science while demanding increasing capacity from computing resources. High performance computing platforms (HPC), traditionally employed in massively parallel numerical simulations, offer computational power that can be harnessed in big data analysis. However, the convergence of Big Data and HPC must be examined in several ways; In particular, network infrastructure needs to adjust to very different application demands. The software-defined network (SDN) model can favor this convergence, thanks to its global view of the network and its programmability. In this context, we present an SDN platform capable of convergently meeting Big Data and HPC application requirements. The platform applies routing mechanisms best suited to each traffic profile, thus reducing application execution time. We simulated the viability of our platform by reducing the execution time of real MPI applications in specific scenarios by up to 11% and Hadoop by up to 6%.

Keywords: High Performance Computing, Software Defined Networks, Big Data

References

Alsmadi, I., Khamaiseh, S., and Xu, D. (2016). Network parallelization in HPC clusters. In Proc. of the IEEE CSCI.

Bhatia, S., Sinha, Y., Chalapathi, G., and Kumar, R. (2017). MPI aware routing using SDN. Poster Presented at the 26th HPDC.

Fox, G., Qiu, J., Jha, S., Ekanayake, S., and Kamburugamuve, S. (2015). Big Data, simulations and HPC convergence. In Big Data Benchmarking, pages 3–17. Springer.

Kreutz, D., Ramos, F., Veríssimo, P., Rothenberg, C., Azodolmolky, S., and Uhlig, S. (2015). Software-defined networking: A comprehensive survey. Proc. of the IEEE, 103(1):14–76.

Liang, F. and Lau, F. (2016). BAShuffler: Maximizing network bandwidth utilization in the shuffle of YARN. In Proc. of the ACM HPDC, pages 281–284.

LNCC (2018). Configuração do SDumont. https://sdumont.lncc.br/machine.php?pg=machine#. Accessed: December, 2018.

Mattson, T., Sanders, B., and Massingill, B. (2004). Patterns for Parallel Programming. Pearson Education.

McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., and Turner, J. (2008). OpenFlow: Enabling innovation in campus networks. ACM SIGCOMM Computer Communication Review, 38(2):69–74.

Mellanox (2015). SX6536 Product Brief. http://www.mellanox.com/relateddocs/prod_ib_switch_systems/PB_SX6536.pdf. Accessed: December, 2018.

Narayan, S., Bailey, S., and Daga, A. (2012). Hadoop acceleration in an OpenFlow-based cluster. In Proc. of the IEEE SCC.

Neves, M., De Rose, C., and Katrinis, K. (2015). MRemu: An emulation-based framework for datacenter network experimentation using realistic MapReduce traffic. In IEEE MASCOTS, pages 174–177.

Oliveira, A., Vieira, A., Gomes, A., and Ziviani, A. (2018). Análise de desempenho de rede para aplicações MPI em infraestruturas SDNs convergentes para HPC e Big Data. In Proc. of the WSCAD, pages 397–408. SBC.

ORNL (2018). SUMMIT – Oak Ridge National Laboratory’s next High Performance Supercomputer. https://www.olcf.ornl.gov/olcf-resources/ compute-systems/summit/. Accessed: December, 2018.

Peuster, M., Karl, H., and Van Rossem, S. (2016). MeDICINE: Rapid prototyping of production-ready network services in multi-PoP environments. IEEE NFV-SDN.

Ponce, L., Santos, W., Meira-Jr, W., and Guedes, D. (2018). Extensão de um ambiente de computação de alto desempenho para o processamento de dados massivos. In Proc. of the SBRC.

Porto, F. (2017). Algoritmos e modelos de programação em Big Data XXXVI Jornada de Atualização em Informática (JAI). In Proc. of the SBC Congress.

Qiao, Y., Wang, X., Fang, G., and Lee, B. (2016). Doopnet: An emulator for network performance analysis of Hadoop clusters using Docker and Mininet. In IEEE ISCC, pages 784–790.

Qin, P., Dai, B., Huang, B., and Xu, G. (2015). Bandwidth-aware scheduling with SDN in Hadoop: A new trend for Big Data. IEEE Systems Journal.

Takahashi, K., Date, S., Khureltulga, D., Kido, Y., Yamanaka, H., Kawai, E., and Shimojo, S. (2018). Unisonflow: A software-defined coordination mechanism for message-passing communication and computation. IEEE Access, 6:23372–23382.

Trois, C., Bona, L., Del Fabro, M., Martinello, M., Bidkar, S., Nejabati, R., and Simeonidou, D. (2017). Softening up the network for scientific applications. In 2017 25th Euromicro International Conference on PDP, pages 108–115. IEEE.

Webb, K., Snoeren, A., and Yocum, K. (2011). Topology switching for data center networks. Hot-ICE, 11.