Can LLMs Diagnose Performance Bottlenecks from Test Metrics? A Grey-Box Evaluation
Resumo
Software performance diagnosis requires engineers to correlate load-test metrics with system telemetry and formulate hypotheses about architectural bottlenecks. This paper investigates whether Large Language Models (LLMs) can support this task without source-code access. We conducted an exploratory Grey-Box evaluation using a Node.js API with two deliberately injected performance problems: simulated N+1 data access and a Memory Leak. Consolidated K6 metrics and server telemetry were analyzed by Gemini 3.5 Flash, Claude Sonnet 4.6, and GPT-5.5 Instant using Zero-Shot Prompting. Three domain experts independently evaluated bottleneck-class diagnostic precision, recommendation actionability, and unsupported-claim severity. Results show scenariodependent behavior. Memory Leak diagnosis received higher scores when monotonic Heap growth provided a highly discriminative signal, whereas aggregate latency evidence in the N+1 scenario produced broader and less precise diagnoses. Some assertive responses also introduced unsupported causal claims. These findings suggest that LLMs can assist post-test performance diagnosis, but their reliability depends on the diagnostic specificity of the available telemetry and requires human validation.
Palavras-chave:
Performance Testing, Root Cause Analysis, Large Language Models, Software Engineering, Node.js
Referências
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. InAdvances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 1877–1901.
Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, Jun Zeng, Supriyo Ghosh, Xuchao Zhang, Chaoyun Zhang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Tianyin Xu. 2024. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, Greece) (EuroSys ’24). Association for Computing Machinery, New York, NY, USA, 674–688. DOI: 10.1145/3627703.3629553
Jeffrey Dean and Luiz André Barroso. 2013. The tail at scale. Commun. ACM 56, 2 (Feb. 2013), 74–80. DOI: 10.1145/2408776.2408794
Grafana Labs. 2024. Grafana K6: Open-Source Load Testing Tool. [link]. Accessed: July 2026.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8, Article 220 (2024). DOI: 10.1145/3695988
ISO/IEC. 2011. ISO/IEC 25010:2011 – Systems and Software Engineering — Systems and Software Quality Requirements and Evaluation (SQuaRE) — System and Software Quality Models. International Standard ISO/IEC 25010:2011. International Organization for Standardization, Geneva, Switzerland.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12, Article 248 (March 2023), 38 pages. DOI: 10.1145/3571730
Bill Karwin. 2013. SQL Antipatterns: Avoiding the Pitfalls of Database Programming. Pragmatic Bookshelf.
Kristian Kolthoff, Felix Kretzer, Lennart Fiebig, Christian Bartelt, Alexander Maedche, and Simone Paolo Ponzetto. 2024. Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generation. arXiv:2412.11328 [cs.SE] [link]
R. Likert. 1987. A Technique for the Measurement of Attitudes. Archives of Psychology, Columbia University. [link]
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. DOI: 10.1162/tacl_a_00638
Rubel Hassan Mollik, Arup Datta, Anamul Haque Mollah, and Wajdi Aljedaani. 2025. RCEGen: A Generative Approach for Automated Root Cause Analysis Using Large LanguageModels (LLMs). Software 4, 4 (2025). DOI: 10.3390/software4040029
Ian Molyneaux. 2014. The Art of Application Performance Testing: From Strategy to Tools (2nd ed.). O’Reilly Media, Inc.
Roger S. Pressman and Bruce R. Maxim. 2021. Software Engineering: A Practitioner’s Approach (9 ed.). McGraw-Hill Education, New York.
Ramu. 2023. PerfDetectiveAI - Performance Gap Analysis and Recommendation in Software Applications. arXiv preprint arXiv:2306.06566 (2023).
Ian Sommerville. 2016. Software Engineering (10 ed.). Pearson, Boston.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS). 5998–6008.
V. Verkholomova and E. Nazaruka. 2025. Overview of the Application of Generative AI in Software Performance Testing. In Proceedings of the 2025 66th International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS 2025) (2025), 100–105. DOI: 10.1109/, ITMS67030.2025.11236704
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). [link]
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] [link]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering (2012 ed.). Springer, Berlin, Germany.
Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, Jun Zeng, Supriyo Ghosh, Xuchao Zhang, Chaoyun Zhang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Tianyin Xu. 2024. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, Greece) (EuroSys ’24). Association for Computing Machinery, New York, NY, USA, 674–688. DOI: 10.1145/3627703.3629553
Jeffrey Dean and Luiz André Barroso. 2013. The tail at scale. Commun. ACM 56, 2 (Feb. 2013), 74–80. DOI: 10.1145/2408776.2408794
Grafana Labs. 2024. Grafana K6: Open-Source Load Testing Tool. [link]. Accessed: July 2026.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8, Article 220 (2024). DOI: 10.1145/3695988
ISO/IEC. 2011. ISO/IEC 25010:2011 – Systems and Software Engineering — Systems and Software Quality Requirements and Evaluation (SQuaRE) — System and Software Quality Models. International Standard ISO/IEC 25010:2011. International Organization for Standardization, Geneva, Switzerland.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12, Article 248 (March 2023), 38 pages. DOI: 10.1145/3571730
Bill Karwin. 2013. SQL Antipatterns: Avoiding the Pitfalls of Database Programming. Pragmatic Bookshelf.
Kristian Kolthoff, Felix Kretzer, Lennart Fiebig, Christian Bartelt, Alexander Maedche, and Simone Paolo Ponzetto. 2024. Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generation. arXiv:2412.11328 [cs.SE] [link]
R. Likert. 1987. A Technique for the Measurement of Attitudes. Archives of Psychology, Columbia University. [link]
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. DOI: 10.1162/tacl_a_00638
Rubel Hassan Mollik, Arup Datta, Anamul Haque Mollah, and Wajdi Aljedaani. 2025. RCEGen: A Generative Approach for Automated Root Cause Analysis Using Large LanguageModels (LLMs). Software 4, 4 (2025). DOI: 10.3390/software4040029
Ian Molyneaux. 2014. The Art of Application Performance Testing: From Strategy to Tools (2nd ed.). O’Reilly Media, Inc.
Roger S. Pressman and Bruce R. Maxim. 2021. Software Engineering: A Practitioner’s Approach (9 ed.). McGraw-Hill Education, New York.
Ramu. 2023. PerfDetectiveAI - Performance Gap Analysis and Recommendation in Software Applications. arXiv preprint arXiv:2306.06566 (2023).
Ian Sommerville. 2016. Software Engineering (10 ed.). Pearson, Boston.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS). 5998–6008.
V. Verkholomova and E. Nazaruka. 2025. Overview of the Application of Generative AI in Software Performance Testing. In Proceedings of the 2025 66th International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS 2025) (2025), 100–105. DOI: 10.1109/, ITMS67030.2025.11236704
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). [link]
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] [link]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering (2012 ed.). Springer, Berlin, Germany.
Publicado
11/09/2026
Como Citar
MUNIZ, Miguel Angelo Bastos; TIADORO, Eduardo Prates; BERNARDINO, Maicon; ANDRADE, Anielle; RODRIGUES, Elder de Macedo.
Can LLMs Diagnose Performance Bottlenecks from Test Metrics? A Grey-Box Evaluation. In: WORKSHOP BRASILEIRO DE ENGENHARIA DE SOFTWARE INTELIGENTE (ISE), 5. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 55-60.
DOI: https://doi.org/10.5753/ise.2026.32908.
