Deep-Learning Inference Performance Across Heterogeneous HPC CPU Systems
Resumo
Choosing a CPU platform for deep-learning inference in heterogeneous HPC systems is not straightforward because performance changes with the workload and execution configuration. We compare five HPC CPU configurations using Detectron2 and MLPerf Inference v5.1 Offline, complemented by controlled BERT and ResNet50 scaling and repeated-run experiments. No platform leads all 11 workloads; median rank similarity is 0.700 within Detectron2, 0.400 within MLPerf, and −0.100 across suites. Scaling is sublinear at every tested multiplier, while cross-platform CV reaches 16.0%. Overall, the results favor reporting performance together with its workload, allocation, placement, and measurement context rather than as a single global platform ranking.Referências
Ahn, H., Chen, T., Alnaasan, N., Shafi, A., Abduljabbar, M., Subramoni, H., and Panda, D. K. (2023). Performance characterization of using quantization for DNN inference on edge devices. In 2023 IEEE 7th International Conference on Fog and Edge Computing (ICFEC), pages 1–6.
Alvarado Rodriguez, O., Dave, D., Liu, W., and Su, B. (2020). A study of machine learning inference benchmarks. In Proceedings of the 4th International Conference on Advances in Image Processing (ICAIP 2020), pages 167–171.
Ben-Nun, T., Besta, M., Huber, S., Ziogas, A. N., Peter, D., and Hoefler, T. (2019). A modular benchmarking infrastructure for high-performance and reproducible deep learning. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 66–77.
Camargo, M. W., Serpa, M. S., Carastan-Santos, D., Carissimi, A., and Navaux, P. O. A. (2021). Accelerating machine learning algorithms with TensorFlow using thread mapping policies. In High Performance Computing: 7th Latin American Conference, CARLA 2020, Revised Selected Papers, volume 1327 of Communications in Computer and Information Science, pages 62–70.
Castelló, A., Barrachina, S., Dolz, M. F., Quintana-Ortí, E. S., San Juan, P., and Tomás, A. E. (2022). High performance and energy efficient inference for deep learning on multicore ARM processors using general optimization techniques and BLIS. Journal of Systems Architecture, 125:102459.
Chunduri, S., Harms, K., Parker, S., Morozov, V., Oshin, S., Cherukuri, N., and Kumaran, K. (2017). Run-to-run variability on Xeon Phi based Cray XC systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2017), pages 1–13.
Cui, M., Papadopoulou, N., and Pericàs, M. (2023). Analysis and characterization of performance variability for OpenMP runtime. In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, pages 1614–1622.
Denoyelle, N., Goglin, B., Jeannot, E., and Ropars, T. (2019). Data and thread placement in NUMA architectures: A statistical learning approach. In Proceedings of the 48th International Conference on Parallel Processing (ICPP 2019), pages 39:1–39:10.
Dokulil, J. and Benkner, S. (2020). NUMA-aware CPU core allocation in cooperating dynamic applications. In 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 950–957.
Elordi, U., Unzueta, L., Goenetxea, J., Sánchez-Carballido, S., Arganda-Carreras, I., and Otaegui, O. (2021). Benchmarking deep neural network inference performance on serverless environments with MLPerf. IEEE Software, 38(1):81–87.
Hodak, M., Ellison, D., and Dholakia, A. (2022). Everyone is a winner: Interpreting MLPerf inference benchmark results. In Performance Evaluation and Benchmarking: 13th TPC Technology Conference (TPCTC 2021), Revised Selected Papers, volume 13169 of Lecture Notes in Computer Science, pages 50–61.
Hoefler, T. and Belli, R. (2015). Scientific benchmarking of parallel computing systems: Twelve ways to tell the masses when reporting performance results. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2015), pages 73:1–73:12.
Jiang, J., Du, J., Huang, D., Chen, Z., Lu, Y., and Liao, X. (2023). Full-stack optimizing transformer inference on ARM many-core CPU. IEEE Transactions on Parallel and Distributed Systems, 34(7):2221–2235.
Kalibera, T. and Jones, R. E. (2013). Rigorous benchmarking in reasonable time. In Proceedings of the 2013 ACM SIGPLAN International Symposium on Memory Management (ISMM 2013), pages 63–74.
Khan, M. A. and Islam, S. (2026). Optimizing deep learning inference on multi-core laptop CPUs with scheduling and thread affinity strategies. In Proceedings of the 2026 ACM Southeast Conference (ACMSE 2026), pages 276–280.
Liu, Y., Wang, Y., Yu, R., Li, M., Sharma, V., and Wang, Y. (2019). Optimizing CNN model inference on CPUs. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 1025–1040. USENIX Association.
Liu, Z., Leng, J., Zhang, Z., Chen, Q., Li, C., and Guo, M. (2022). VELTAIR: Towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2022), pages 388–401.
Mazouz, A., Touati, S.-A.-A., and Barthou, D. (2011). Performance evaluation and analysis of thread pinning strategies on multi-core platforms: Case study of SPEC OMP applications on intel architectures. In 2011 International Conference on High Performance Computing and Simulation (HPCS), pages 273–279.
Mebratu, D., Hasabnis, N., Mercati, P., Sharma, G., and Najnin, S. (2021). Automatic tuning of TensorFlow’s CPU backend using gradient-free optimization algorithms. In High Performance Computing: ISC High Performance 2021 International Workshops, Revised Selected Papers, volume 12761 of Lecture Notes in Computer Science, pages 249–266.
Mittal, S., Rajput, P., and Subramoney, S. (2022). A survey of deep learning on CPUs: Opportunities and co-optimizations. IEEE Transactions on Neural Networks and Learning Systems, 33(10):5095–5115.
MLCommons (2025). MLPerf inference v5.1 reference implementation and configuration files. Official MLCommons inference repository, tag v5.1. commit 75f297d44c5e68d24a2f7643fe1eb11e49313238.
Nori, A. V., Bera, R., Balachandran, S., Rakshit, J., Omer, O. J., Abuhatzera, A., Kuttanna, B., and Subramoney, S. (2021). REDUCT: Keep it close, keep it cool! efficient scaling of DNN inference on multi-core CPUs with near-cache compute. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 167–180.
Pati, S., Aga, S., Jayasena, N., and Sinclair, M. D. (2022). Demystifying BERT: System design implications. In 2022 IEEE International Symposium on Workload Characterization (IISWC), pages 296–309.
Prieto, P., Abad, P., Gregorio, J. A., and Puente, V. (2023). Performance characterization of popular DNN models on out-of-order CPUs. In 2023 32nd International Conference on Parallel Architectures and Compilation Techniques (PACT), pages 199–210.
Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., and Wu, C.-J. (2021). The vision behind MLPerf: Understanding AI inference performance. IEEE Micro, 41(3):10–18.
Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M., et al. (2020). MLPerf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 446–459.
Vellaisamy, P., Labonte, T., Chakraborty, S., Turner, M., Sury, S., and Shen, J. P. (2025). Characterizing and optimizing LLM inference workloads on CPU–GPU coupled architectures. In 2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 49–61.
Alvarado Rodriguez, O., Dave, D., Liu, W., and Su, B. (2020). A study of machine learning inference benchmarks. In Proceedings of the 4th International Conference on Advances in Image Processing (ICAIP 2020), pages 167–171.
Ben-Nun, T., Besta, M., Huber, S., Ziogas, A. N., Peter, D., and Hoefler, T. (2019). A modular benchmarking infrastructure for high-performance and reproducible deep learning. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 66–77.
Camargo, M. W., Serpa, M. S., Carastan-Santos, D., Carissimi, A., and Navaux, P. O. A. (2021). Accelerating machine learning algorithms with TensorFlow using thread mapping policies. In High Performance Computing: 7th Latin American Conference, CARLA 2020, Revised Selected Papers, volume 1327 of Communications in Computer and Information Science, pages 62–70.
Castelló, A., Barrachina, S., Dolz, M. F., Quintana-Ortí, E. S., San Juan, P., and Tomás, A. E. (2022). High performance and energy efficient inference for deep learning on multicore ARM processors using general optimization techniques and BLIS. Journal of Systems Architecture, 125:102459.
Chunduri, S., Harms, K., Parker, S., Morozov, V., Oshin, S., Cherukuri, N., and Kumaran, K. (2017). Run-to-run variability on Xeon Phi based Cray XC systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2017), pages 1–13.
Cui, M., Papadopoulou, N., and Pericàs, M. (2023). Analysis and characterization of performance variability for OpenMP runtime. In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, pages 1614–1622.
Denoyelle, N., Goglin, B., Jeannot, E., and Ropars, T. (2019). Data and thread placement in NUMA architectures: A statistical learning approach. In Proceedings of the 48th International Conference on Parallel Processing (ICPP 2019), pages 39:1–39:10.
Dokulil, J. and Benkner, S. (2020). NUMA-aware CPU core allocation in cooperating dynamic applications. In 2020 IEEE 34th International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 950–957.
Elordi, U., Unzueta, L., Goenetxea, J., Sánchez-Carballido, S., Arganda-Carreras, I., and Otaegui, O. (2021). Benchmarking deep neural network inference performance on serverless environments with MLPerf. IEEE Software, 38(1):81–87.
Hodak, M., Ellison, D., and Dholakia, A. (2022). Everyone is a winner: Interpreting MLPerf inference benchmark results. In Performance Evaluation and Benchmarking: 13th TPC Technology Conference (TPCTC 2021), Revised Selected Papers, volume 13169 of Lecture Notes in Computer Science, pages 50–61.
Hoefler, T. and Belli, R. (2015). Scientific benchmarking of parallel computing systems: Twelve ways to tell the masses when reporting performance results. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2015), pages 73:1–73:12.
Jiang, J., Du, J., Huang, D., Chen, Z., Lu, Y., and Liao, X. (2023). Full-stack optimizing transformer inference on ARM many-core CPU. IEEE Transactions on Parallel and Distributed Systems, 34(7):2221–2235.
Kalibera, T. and Jones, R. E. (2013). Rigorous benchmarking in reasonable time. In Proceedings of the 2013 ACM SIGPLAN International Symposium on Memory Management (ISMM 2013), pages 63–74.
Khan, M. A. and Islam, S. (2026). Optimizing deep learning inference on multi-core laptop CPUs with scheduling and thread affinity strategies. In Proceedings of the 2026 ACM Southeast Conference (ACMSE 2026), pages 276–280.
Liu, Y., Wang, Y., Yu, R., Li, M., Sharma, V., and Wang, Y. (2019). Optimizing CNN model inference on CPUs. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 1025–1040. USENIX Association.
Liu, Z., Leng, J., Zhang, Z., Chen, Q., Li, C., and Guo, M. (2022). VELTAIR: Towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2022), pages 388–401.
Mazouz, A., Touati, S.-A.-A., and Barthou, D. (2011). Performance evaluation and analysis of thread pinning strategies on multi-core platforms: Case study of SPEC OMP applications on intel architectures. In 2011 International Conference on High Performance Computing and Simulation (HPCS), pages 273–279.
Mebratu, D., Hasabnis, N., Mercati, P., Sharma, G., and Najnin, S. (2021). Automatic tuning of TensorFlow’s CPU backend using gradient-free optimization algorithms. In High Performance Computing: ISC High Performance 2021 International Workshops, Revised Selected Papers, volume 12761 of Lecture Notes in Computer Science, pages 249–266.
Mittal, S., Rajput, P., and Subramoney, S. (2022). A survey of deep learning on CPUs: Opportunities and co-optimizations. IEEE Transactions on Neural Networks and Learning Systems, 33(10):5095–5115.
MLCommons (2025). MLPerf inference v5.1 reference implementation and configuration files. Official MLCommons inference repository, tag v5.1. commit 75f297d44c5e68d24a2f7643fe1eb11e49313238.
Nori, A. V., Bera, R., Balachandran, S., Rakshit, J., Omer, O. J., Abuhatzera, A., Kuttanna, B., and Subramoney, S. (2021). REDUCT: Keep it close, keep it cool! efficient scaling of DNN inference on multi-core CPUs with near-cache compute. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 167–180.
Pati, S., Aga, S., Jayasena, N., and Sinclair, M. D. (2022). Demystifying BERT: System design implications. In 2022 IEEE International Symposium on Workload Characterization (IISWC), pages 296–309.
Prieto, P., Abad, P., Gregorio, J. A., and Puente, V. (2023). Performance characterization of popular DNN models on out-of-order CPUs. In 2023 32nd International Conference on Parallel Architectures and Compilation Techniques (PACT), pages 199–210.
Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., and Wu, C.-J. (2021). The vision behind MLPerf: Understanding AI inference performance. IEEE Micro, 41(3):10–18.
Reddi, V. J., Cheng, C., Kanter, D., Mattson, P., Schmuelling, G., Wu, C.-J., Anderson, B., Breughe, M., et al. (2020). MLPerf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 446–459.
Vellaisamy, P., Labonte, T., Chakraborty, S., Turner, M., Sury, S., and Shen, J. P. (2025). Characterizing and optimizing LLM inference workloads on CPU–GPU coupled architectures. In 2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 49–61.
Publicado
03/11/2026
Como Citar
BIANCHINI, Calebe P. et al.
Deep-Learning Inference Performance Across Heterogeneous HPC CPU Systems. In: SIMPÓSIO EM SISTEMAS COMPUTACIONAIS DE ALTO DESEMPENHO (SSCAD), 27. , 2026, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 398-412.
DOI: https://doi.org/10.5753/sscad.2026.34958.
