Test Smell Detection and Refactoring with Agentic LLMs: An Empirical Study of Small Open Models

• Rian Melo UFCG / Instituto Kunumi
• Rohit Gheyi UFCG

DOI: https://doi.org/10.5753/cbsoft_estendido.2026.30117

Resumo

Test smells are recurring design problems in test code that can hurt readability and complicate maintenance, and although many approaches detect them, few automate their removal. This work investigates small, open language models—Llama-3.2-3B, Gemma-2-9B, DeepSeek-R1-14B, and Phi-4-14B—for the automated detection and refactoring of test smells through agentic workflows with one, two, and four agents. We evaluate 150 instances of five common test smells drawn from 11 real-world Java projects using JUnit 5, with all models executed locally on consumer-grade hardware via Ollama and LangChain. The models detected nearly all instances (up to 100%), and Phi-4-14B reached the highest refactoring accuracy (75.3% pass@5), rivaling proprietary single-agent baselines such as o3 and Gemini-2.5-Pro while running locally. Multi-agent workflows improved results for three of the five smells. A survey with 50 developers preferred the refactored tests in about 66% of the comparisons, and 6 of 10 submitted pull requests were merged into open-source projects. The approach is easily extended to new smells through natural-language definitions and generalizes to other programming languages, such as Python, Go, and JavaScript.

Palavras-chave: Test Smells, Refactoring, Large Language Models, Agentic Workflows, Software Testing

Referências

Wajdi Aljedaani et al. 2021. Test Smell Detection Tools: A Systematic Mapping Study. In International Conference on Evaluation and Assessment in Software Engineering. 170–180. DOI: 10.1145/3463274.3463335

Elvys Soares et al. 2023. Refactoring Test Smells With JUnit 5: Why Should Developers Keep Up-to-Date? IEEE Transactions on Software Engineering 49, 3 (2023), 1152–1170.

Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. [link]

Xinyi Hou et al. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024).

Rian Melo, Pedro Simões, Rohit Gheyi, Marcelo d’Amorim, Márcio Ribeiro, Gustavo Soares, Eduardo Almeida, and Elvys Soares. 2026. Agentic LMs: Hunting Down Test Smells. IEEE Software 43, 1 (2026), 32–40. DOI: 10.1109/MS.2025.3621356

Anthony Peruma et al. 2019. On the Distribution of Test Smells in Open Source Android Applications: An Exploratory Study. In International Conference on Computer Science and Software Engineering. 193–202.

Anthony Peruma et al. 2020. tsDetect: an open source test smells detection tool. In Foundations of Software Engineering. ACM, 1650–1654.

Valeria Pontillo et al. 2024. Machine learning-based test smell detection. Empirical Software Engineering 29, 2 (2024), 1–44.

Railana Santana et al. 2020. RAIDE: A Tool for Assertion Roulette and Duplicate Assert Identification and Refactoring. In 34th Brazilian Symposium on Software Engineering (SBES). 374–379. DOI: 10.1145/3422392.3422510

Arie van Deursen et al. 2001. Refactoring test code. In International conference on extreme programming and flexible processes in software engineering. 92–95.

Junjie Wang et al. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering 50 (2024), 911–936.

Jason Wei et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems.