Analysis and Characterization of Automated Refactoring Tools

• Matheus Machado de O. Andrade PUC Minas
• Samara Martins Ferreira PUC Minas
• Cleiton Silva Tavares PUC Minas
• Leonardo Cardoso PUC Minas
• Laerte Xavier PUC Minas
• Lucila Ishitani PUC Minas

DOI: https://doi.org/10.5753/vem.2025.14610

Abstract

Automated refactoring is an increasingly common practice in the software development process. However, the effectiveness of using artificial intelligence for this purpose is still unclear. Therefore, this study evaluates the GPT-4.5 and Claude 3.7 Sonnet for automated refactorings. The research focuses on the application of these tools to the jparse system, aiming to measure their efficiency and applicability. The evaluation involved quantitative and qualitative analyses of 816 refactorings, generating eight criteria for evaluating these tools in real-world scenarios. The results showed that the tools perform better when applied to smaller scopes and when the type of refactoring to be performed is specifically guided.

References

Almogahed, A., Mahdin, H., Zakaria, N. H., Omar, M., Barraood, S. O., and Alawadhi, A. (2023). Empirical investigation of the diverse refactoring effects on software quality: The role of refactoring tools and software size. In International Conference on Emerging Smart Technologies and Applications (eSmartA).

Amazon Web Services (2024). What are embeddings in machine learning? Accessed: 25 Mar. 2025.

Chidamber, S. and Kemerer, C. (1994). A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20(6):476–493.

Eilertsen, A. M. and Murphy, G. C. (2021a). Stepwise refactoring tools. In IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 629–633.

Eilertsen, A. M. and Murphy, G. C. (2021b). The usability (or not) of refactoring tools. In 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 237–248. IEEE.

Fowler, M. (2018). Refactoring: Improving the Design of Existing Code. Addison Wesley.

Ivers, J., Nord, R. L., Ozkaya, I., Seifried, C., Timperley, C. S., and Kessentini, M. (2022). Industry’s cry for tools that support large-scale refactoring. In International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 163–164.

Mealy, E. and Strooper, P. (2006). Evaluating software refactoring tool support. In Australian Software Engineering Conference (ASWEC’06). IEEE.

Tempero, E., Anslow, C., Dietrich, J., Han, T., Li, J., Lumpe, M., Melton, H., and Noble, J. (2010). Qualitas corpus: A curated collection of java code for empirical studies. In 2010 Asia Pacific Software Engineering Conference (APSEC2010), pages 336–345.

Terra, R., Miranda, L. F., Valente, M. T., and Bigonha, R. S. (2013). Qualitas.class corpus: A compiled version of the qualitas corpus. Software Engineering Notes, pages 1–4.

Valente, M. T. (2020). Engenharia de Software Moderna: Princípios e Práticas para Desenvolvimento de Software com Produtividade. Independente.

White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., and Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382.

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., and Wesslén, A. (2012). Experimentation in Software Engineering. Computer Science. Springer Berlin Heidelberg.