NoseSense: Benchmarking LLMs for Test Smell Detection
Resumo
Unit test quality is a critical concern in software engineering, as anti-patterns in test code (known as test smells) compromise maintainability and reliability of test suites. The rapid evolution of Large Language Models (LLMs) has motivated their application to this domain, yet the pace of model releases renders point-in-time empirical studies quickly outdated. In this paper, we develop NoseSense, a benchmark designed to enable systematic and reproducible evaluation of LLMs for test smell detection.We mined 1,275 open-source Java repositories, applied the JNose static analysis tool to curate a balanced dataset of 1,100 real test smell instances, and evaluated six state-of-the-art models through a dedicated automated tool. The results indicate that model scale and generation significantly impact performance: Gemma-4-31B-it achieved the highest accuracy (67.36%), whereas the smaller-scale Gemma-3n-E4B-it achieved 38.55%. NoseSense provides a robust and reproducible foundation for monitoring LLM capabilities in software quality assurance, with a publicly available dataset, replication package, and benchmark implementation.
Referências
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol. 15, 3, Article 39 (March 2024), 45 pages. DOI: 10.1145/3641289
Magno Macedo, Lucas Lopes, Letícia Ribeiro, Tássio Virgínio, and Ivan Machado. 2026. NoseSense: Benchmarking LLMs for Test Smell Detection. In 11th Brazilian Symposium on Systematic and Automated Software Testing (SAST).
Yuetian Mao, Junjie He, and Chunyang Chen. 2025. From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps. In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE). ACM. Industry Track.
Seyed Mahmoud Sajjadi Mohammadabadi, Burak Cem Kara, Can Eyupoglu, Can Uzay, Mehmet Serkan Tosun, and Oktay Karakuş. 2025. A survey of large language models: evolution, architectures, adaptation, benchmarking, applications, challenges, and societal implications. Electronics 14, 18 (2025).
Luciana Lourdes Silva, Janio Rosa da Silva, Joao Eduardo Montandon, Marcus Andrade, and Marco Tulio Valente. 2024. Detecting Code Smells using ChatGPT: Initial Insights. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Barcelona, Spain) (ESEM ’24). Association for Computing Machinery, New York, NY, USA, 400–406. DOI: 10.1145/3674805.3690742
Alejandro Velasco, Daniel Rodriguez-Cardenas, Luftar Rahman Alif, David N. Palacio, and Denys Poshyvanyk. 2025. How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study. In 2025 IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). 96–100. DOI: 10.1109/ICSE-NIER66352.2025.00025
Tássio Virgínio, Luana Martins, Larissa Rocha, Railana Santana, Adriana Cruz, Heitor Costa, and Ivan Machado. 2020. JNose: Java Test Smell Detector. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering (Natal, Brazil) (SBES ’20). Association for Computing Machinery, New York, NY, USA, 564–569. DOI: 10.1145/3422392.3422499
