An Evaluation of GPT-4.1 in Assisting Java Code Refactoring
Resumo
Code refactoring is an activity widely used in the daily practice of software development with the goal of making a codebase maintainable and enabling the development of new features more efficiently. Since the release of ChatGPT, LLMs have shown great potential for performing diverse software engineering tasks. This study aims to evaluate the performance of GPT-4.1 in carrying out refactorings. To do so, we employ three different metrics: validity, correctness, and normalized edit distance to assess five different types of refactoring, namely: extract method, extract variable, rename attribute, rename method, and rename parameter. Our results suggest that GPT-4.1 produces refactorings with high validity and the generated code is textually close to the reference (low normalized edit distance), but its correctness underperforms. We conclude that, with the prompts used here, GPT-4.1 does not reliably produce correct refactorings, and its outputs should always be reviewed (by running unit tests or checking behavior manually) before being trusted.
Referências
Eman Abdullah AlOmar, Anushkrishna Venkatakrishnan, Mohamed Wiem Mkaouer, Christian D. Newman, and Ali Ouni. 2024. How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations. In Proceedings of the 21st IEEE/ACM International Conference on Mining Software Repositories (MSR 2024). ACM / IEEE, Lisbon, Portugal, 202–206. DOI: 10.1145/3643991.3645081
Dair.AI. 2024. Prompt Engineering Guide. [link]. Accessed: 2025-08-31.
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In Proceedings of the 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, Los Alamitos, USA, 31–53. DOI: 10.1109/ICSE-FoSE59343.2023.00008
Martin Fowler. 2018. Refactoring: Improving the Design of Existing Code (2nd ed.). Addison-Wesley Professional.
Felix Grund, Shaiful Alam Chowdhury, Nick C Bradley, Braxton Hall, and Reid Holmes. 2021. CodeShovel: Constructing Method-Level Source Code Histories. In Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE ’21). IEEE, Madrid, Spain, 1510–1522. DOI: 10.1109/ICSE43902.2021.00135
Bo Liu, Yanjie Jiang, Yuxia Zhang, Nan Niu, Guangjie Li, and Hui Liu. 2025. Exploring the potential of general purpose LLMs in automated software refactoring: an empirical study. Automated Software Engineering 32, 26 (2025). DOI: 10.1007/s10515-025-00500-0
Alessandro Midolo and Massimiliano Di Penta. 2025. Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMS. In 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC). IEEE, Ottawa, Canada, 01–11. DOI: 10.1109/ICPC66645.2025.00020
Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332–2354. DOI: 10.1145/3660810
Sanyogita Piya and Allison Sullivan. 2024. LLM4TDD: Best practices for test driven development using large language models. In Proceedings of the 1st International Workshop on Large Language Models for Code (LLM4Code ’24). ACM, New York, USA, 14–21. DOI: 10.1145/3643795.3648382
Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bogomolov, Timofey Bryksin, and Danny Dig. 2024. Next-Generation Refactoring: Combining LLM Insights and IDE Capabilities for Extract Method. In Proceedings of the 40th IEEE International Conference on Software Maintenance and Evolution (ICSME 2024). IEEE, Flagstaff, USA, 275–287. DOI: 10.1109/ICSME58944.2024.00034
Gustavo Soares. 2010. Making program refactoring safer. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2. ACM, Cape Town, South Africa, 521–522. DOI: 10.1145/1810295.1810461
Yonghao Wu, Zheng Li, Jie M. Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large Language Models in Fault Localisation. (2023). DOI: 10.48550/ARXIV.2308.15276
Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Vienna, Austria, 819–831. DOI: 10.1145/3650212.3680323
