Avaliando LLMs para Detecção de Refatorações: Um Estudo Preliminar

• Gabriel Henrique Rocha UFOP
• Diego Demétrio UFOP
• Aline Brito UFOP
• Rodrigo Brito UFMG

DOI: https://doi.org/10.5753/ise.2026.32905

Resumo

Ferramentas para detecção de refatorações, como RefactoringMiner e RefDiff, apresentam alta precisão, porém dependem de análises sintáticas e estruturais, o que requer esforço e tempo para extensão para múltiplas linguagens de programação. Visando melhorar este cenário, este trabalho investiga a viabilidade do uso de Large Language Models (LLMs) na detecção automática de refatoração. Para tanto, propõe-se um método no qual LLMs analisam commits e detectam refatorações dos tipos Extract Method, Move Method e Rename Method. A abordagem foi avaliada sobre um conjunto de 150 operações reais extraídas de um oráculo de refatorações amplamente utilizado na literatura. Utilizam-se dois LLMs, GPT e Claude Haiku 4.5. Os resultados revelam perfis complementares: um modelo favorece a precisão (0,73) e o outro, a cobertura (recall de 0,72), enquanto a combinação de suas saídas eleva o recall a 0,77. Esses achados sugerem que os LLMs constituem uma abordagem promissora.

Palavras-chave: Refatoração, LLM, Detecção de refatorações, Prompt Engineering

Referências

Lucas Amaral, EliakimGama,Matheus Paixao, and Lucas Aguiar. 2025. Evaluating Large Language Models on the Classification of Different Technical Debt Types in Stack Overflow Discussions. In IV Workshop Brasileiro de Engenharia de Software Inteligente (ISE).

Aline Brito, Andre Hora, and Marco Tulio Valente. 2021. Characterizing Refactoring Graphs in Java and JavaScript Projects. Empirical Software Engineering 26 (2021).

Rodrigo Brito and Marco Tulio Valente. 2020. RefDiff4Go: Detecting Refactorings in Go. In 14th Brazilian Symposium on Software Components, Architectures, and Reuse (SBCARS). 101–110.

Rodrigo Brito and Marco Tulio Valente. 2021. RAID: Tool Support for Refactoring-Aware Code Reviews. In 29th IEEE/ACM International Conference on Program Comprehension (ICPC). 265–275.

Ivan Moura Campos. 2026. How to Write Well-Structured Prompts for LLMs. [link]. Online; accessed April 2026.

Jonathan Cordeiro, Shayan Noei, and Ying Zou. 2025. An Empirical Study on the Code Refactoring Capability of Large Language Models. ACM Transactions on Software Engineering and Methodology (TOSEM) (2025).

Martin Fowler. 1999. Refactoring: Improving the Design of Existing Code. Addison-Wesley.

Martin Fowler. 2018. Refactoring: improving the design of existing code. Addison-Wesley.

Lujun Li, Lama Sleem, Geoffrey Nichil, Radu State, et al. 2025. Exploring the impact of temperature on large language models: Hot or cold? Procedia Computer Science 264 (2025), 242–251.

Amália Melo, Lucas Almeida, and Lina Garcés. 2025. Investigating the use of Large Language Models in software security requirements: Results of a literature review. In IV Workshop Brasileiro de Engenharia de Software Inteligente (ISE). 37–42.

Danilo Silva, Joao Paulo da Silva, Gustavo Santos, Ricardo Terra, and Marco Tulio Valente. 2021. RefDiff 2.0: A Multi-language Refactoring Detection Tool. IEEE Transactions on Software Engineering 47, 12 (2021), 2786–2802.

Danilo Silva, Nikolaos Tsantalis, and Marco Tulio Valente. 2016. Why We Refactor? Confessions of GitHub Contributors. In 24th International Symposium on the Foundations of Software Engineering (FSE). 858–870.

Danilo Silva and Marco Tulio Valente. 2017. RefDiff: Detecting Refactorings in Version Histories. In 14th International Conference on Mining Software Repositories (MSR). 269–279.

Luciana Lourdes Silva, Janio Rosa da Silva, Joao Eduardo Montandon, Marcus Andrade, and Marco Tulio Valente. 2024. Detecting Code Smells using ChatGPT: Initial Insights. In 18th International Symposium on Empirical Software Engineering and Measurement. 400–406.

Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. 2020. RefactoringMiner 2.0. IEEE Transactions on Software Engineering (TSE) (2020).

N. Tsantalis, M. Mansouri, L. Eshkevari, D. Mazinanian, and A. Ketkar. 2022. Refactoring oracle. [link]. Online; accessed January 2022.

Nikolaos Tsantalis, Matin Mansouri, Laleh M. Eshkevari, Davood Mazinanian, and Danny Dig. 2018. Accurate and Efficient Refactoring Detection in Commit History. In 40th International Conference on Software Engineering (ICSE). 483–494.

Marco Tulio Valente. 2020. Engenharia de Software Moderna: Princípios e Práticas para Desenvolvimento de Software com Produtividade. Independente.

Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023. Large language models meet nl2code: A survey. In 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7443–7464.

Li Zhong and Zilong Wang. 2024. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In AAAI conference on artificial intelligence, Vol. 38. 21841–21849.