Comparação Empírica entre EvoSuite e ChatUniTest na Geração de Testes de Unidade: Um Estudo de Replicação com Extensão
Resumo
Contexto: Testes de unidade são fundamentais para identificar defeitos e apoiar a evolução segura de sistemas de software. Entretanto, sua elaboração demanda tempo e esforço, tornando a geração automática uma alternativa para apoiar sua construção e manutenção. Objetivo: Comparar o ChatUniTest, baseado em grandes modelos de linguagem, ao EvoSuite, baseado em busca evolutiva, estendendo o estudo original de avaliação do ChatUniTest. Método: Avaliamos testes gerados pelas duas ferramentas para 400 métodos de cinco projetos Java, ampliando em 52% a amostra do estudo original. Além da cobertura de linhas, a avaliação contemplou produtividade, robustez, eficácia e qualidade estrutural. Resultados: O EvoSuite superou o ChatUniTest em produtividade, robustez, cobertura de linhas (em contraste ao estudo original) e escore de mutação. O ChatUniTest apresentou maior cobertura de branches e resultados mistos em qualidade estrutural, com menor complexidade ciclomática e menos test smells, porém maior quantidade de code smells. Conclusão: Os resultados indicam que ferramentas baseadas em LLM são promissoras para geração automática de testes, mas ainda apresentam limitações de robustez e eficácia. Ao ampliar a amostra e as dimensões de avaliação, este trabalho fornece evidências mais abrangentes sobre os potenciais e limitações dessas abordagens.
Palavras-chave:
Geração Automática de Testes, Teste de Unidade, Grandes Modelos de Linguagem, EvoSuite, ChatUniTest, Estudo Empírico
Referências
Shaukat Ali, Lionel C Briand, Hadi Hemmati, and Rambod K Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Transactions on Software Engineering 36, 6 (2010), 742–762.
Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In Proceedings of the 39th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 263–272.
Apêndice Online. 2026. [link]
Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE Companion ’24). ACM, 572–576.
Léuson Da Silva, Paulo Borba, Toni Maciel, Wardah Mahmood, Thorsten Berger, João Moisakis, Aldiberg Gomes, and Vinícius Leite. 2024. Detecting semantic conflicts with unit tests. Journal of Systems and Software 214 (2024), 112070. DOI: 10.1016/j.jss.2024.112070
Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11). ACM, 416–419. DOI: 10.1145/2025113.2025179
Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Transactions on Software Engineering and Methodology 24, 2 (2014).
Gordon Fraser, Matt Staats, Phil McMinn, Andrea Arcuri, and Frank Padberg. 2015. Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study. ACM Transactions on Software Engineering and Methodology (TOSEM) 24, 4 (2015), 23.
Giovanni Grano, Fabio Palomba, and Harald C. Gall. 2019. An Empirical Study on Test Smells in Automatically Generated Code. In Proceedings of the 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). 128–138.
Lucas Gren and Vard Antinyan. 2017. On the Relation Between Unit Testing and Code Quality. In 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA). 52–56. DOI: 10.1109/SEAA.2017.36
Mark Harman. 2007. The Current State and Future of Search Based Software Engineering. In Future of Software Engineering (FOSE ’07). IEEE Computer Society, 342–357. DOI: 10.1109/FOSE.2007.29
Mary Jean Harrold. 1999. Testing evolving software. Journal of systems and software 47, 2-3 (1999), 173–181.
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62
René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (San Jose, CA, USA) (ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 437–440. DOI: 10.1145/2610384.2628055
Claus Klammer and Albin Kern. 2015. Writing unit tests: It’s now or never!. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 1–4.
Michael Konstantinou, Renzo Degiovanni, Jie M. Zhang, Mark Harman, and Mike Papadakis. 2025. YATE: The Role of Test Repair in LLM-Based Unit Test Generation. arXiv:2507.18316 [cs.SE] [link]
Anji Li, Mingwei Liu, Zhenxi Chen, Zheng Pei, Zike Li, Dekun Dai, Yanlin Wang, and Zibin Zheng. 2026. Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation. arXiv:2511.14224 [cs.SE] DOI: 10.1145/3744916.3787769
Yun Lin, You Sheng Ong, Jun Sun, Gordon Fraser, and Jin Song Dong. 2021. Graph-Based Seed Object Synthesis for Search-Based Unit Testing. In Proceedings of the 29th ACM Joint Meeting on ESEC/FSE (ESEC/FSE ’21). 1068–1080. DOI: 10.1145/3468264.3468619
Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. 2025. Test Intention Guided LLM-Based Unit Test Generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 1026–1038. DOI: 10.1109/ICSE55347.2025.00243
Rainer Niedermayr, Elmar Juergens, and Stefan Wagner. 2016. Will my tests tell me if I break this code?. In Proceedings of the International Workshop on Continuous Software Evolution and Delivery. ACM, 23–29.
Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. ASTER: Natural and Multi-Language Unit Test Generation with LLMs . In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE Computer Society, Los Alamitos, CA, USA, 413–424. DOI: 10.1109/ICSE-SEIP66354.2025.00042
Anthony Peruma, Khalid Almalki, Christian D. Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2020. tsDetect: an open source test smells detection tool. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1650–1654. DOI: 10.1145/3368089.3417921
Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Kovalenko, and Pouria Derakhshanfar. 2024. TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion. 5 pages. DOI: 10.1145/3639478.3640024
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024), 85–105. DOI: 10.1109/TSE.2023.3334955
Novi Setiani, Ridi Ferdiana, and Rudy Hartanto. 2020. Test Case Understandability Model. IEEE Access 8 (2020), 169036–169046. DOI: 10.1109/ACCESS.2020.3022876
Sina Shamshiri, René Just, José M. Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges. In Proceedings of the International Conference on Automated Software Engineering (ASE). 201–211.
Esdras Silva, Roberta Coelho, and Lyrene Silva. 2025. LLMs as Test Generators: A Comparative Benchmarking Study. In Anais do XXXIX Simpósio Brasileiro de Engenharia de Software (Recife/PE). SBC, Porto Alegre, RS, Brasil, 25–36. DOI: 10.5753/sbes.2025.9618
Philipp Straubinger and Gordon Fraser. 2023. A Survey on What Developers Think About Testing. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 25–36.
Murat Tasarsu. 2026. Test case generation using large language models: a systematic literature review. Cluster Computing 29, 4 (2026). DOI: 10.1007/s10586-026-06021-z
ZejunWang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 1258–1268. DOI: 10.1145/3691620.3695501
Chen Yang, Junjie Chen, Bin Lin, Ziqi Wang, and Jianyi Zhou. 2026. Advancing Code Coverage: Incorporating Program Analysis with Large Language Models. ACM Trans. Softw. Eng. Methodol. 35, 5, Article 118 (April 2026), 31 pages. DOI: 10.1145/3748505
Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proc. ACM Softw. Eng. 1, FSE, Article 76 (July 2024), 24 pages. DOI: 10.1145/3660783
Junwei Zhang, Xing Hu, Cuiyun Gao, Xin Xia, and Shanping Li. 2026. Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. (March 2026). DOI: 10.1145/3802827 Just Accepted.
Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In Proceedings of the 39th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 263–272.
Apêndice Online. 2026. [link]
Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE Companion ’24). ACM, 572–576.
Léuson Da Silva, Paulo Borba, Toni Maciel, Wardah Mahmood, Thorsten Berger, João Moisakis, Aldiberg Gomes, and Vinícius Leite. 2024. Detecting semantic conflicts with unit tests. Journal of Systems and Software 214 (2024), 112070. DOI: 10.1016/j.jss.2024.112070
Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11). ACM, 416–419. DOI: 10.1145/2025113.2025179
Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Transactions on Software Engineering and Methodology 24, 2 (2014).
Gordon Fraser, Matt Staats, Phil McMinn, Andrea Arcuri, and Frank Padberg. 2015. Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study. ACM Transactions on Software Engineering and Methodology (TOSEM) 24, 4 (2015), 23.
Giovanni Grano, Fabio Palomba, and Harald C. Gall. 2019. An Empirical Study on Test Smells in Automatically Generated Code. In Proceedings of the 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). 128–138.
Lucas Gren and Vard Antinyan. 2017. On the Relation Between Unit Testing and Code Quality. In 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA). 52–56. DOI: 10.1109/SEAA.2017.36
Mark Harman. 2007. The Current State and Future of Search Based Software Engineering. In Future of Software Engineering (FOSE ’07). IEEE Computer Society, 342–357. DOI: 10.1109/FOSE.2007.29
Mary Jean Harrold. 1999. Testing evolving software. Journal of systems and software 47, 2-3 (1999), 173–181.
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62
René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (San Jose, CA, USA) (ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 437–440. DOI: 10.1145/2610384.2628055
Claus Klammer and Albin Kern. 2015. Writing unit tests: It’s now or never!. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 1–4.
Michael Konstantinou, Renzo Degiovanni, Jie M. Zhang, Mark Harman, and Mike Papadakis. 2025. YATE: The Role of Test Repair in LLM-Based Unit Test Generation. arXiv:2507.18316 [cs.SE] [link]
Anji Li, Mingwei Liu, Zhenxi Chen, Zheng Pei, Zike Li, Dekun Dai, Yanlin Wang, and Zibin Zheng. 2026. Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation. arXiv:2511.14224 [cs.SE] DOI: 10.1145/3744916.3787769
Yun Lin, You Sheng Ong, Jun Sun, Gordon Fraser, and Jin Song Dong. 2021. Graph-Based Seed Object Synthesis for Search-Based Unit Testing. In Proceedings of the 29th ACM Joint Meeting on ESEC/FSE (ESEC/FSE ’21). 1068–1080. DOI: 10.1145/3468264.3468619
Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. 2025. Test Intention Guided LLM-Based Unit Test Generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 1026–1038. DOI: 10.1109/ICSE55347.2025.00243
Rainer Niedermayr, Elmar Juergens, and Stefan Wagner. 2016. Will my tests tell me if I break this code?. In Proceedings of the International Workshop on Continuous Software Evolution and Delivery. ACM, 23–29.
Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. ASTER: Natural and Multi-Language Unit Test Generation with LLMs . In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE Computer Society, Los Alamitos, CA, USA, 413–424. DOI: 10.1109/ICSE-SEIP66354.2025.00042
Anthony Peruma, Khalid Almalki, Christian D. Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2020. tsDetect: an open source test smells detection tool. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1650–1654. DOI: 10.1145/3368089.3417921
Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Kovalenko, and Pouria Derakhshanfar. 2024. TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion. 5 pages. DOI: 10.1145/3639478.3640024
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024), 85–105. DOI: 10.1109/TSE.2023.3334955
Novi Setiani, Ridi Ferdiana, and Rudy Hartanto. 2020. Test Case Understandability Model. IEEE Access 8 (2020), 169036–169046. DOI: 10.1109/ACCESS.2020.3022876
Sina Shamshiri, René Just, José M. Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges. In Proceedings of the International Conference on Automated Software Engineering (ASE). 201–211.
Esdras Silva, Roberta Coelho, and Lyrene Silva. 2025. LLMs as Test Generators: A Comparative Benchmarking Study. In Anais do XXXIX Simpósio Brasileiro de Engenharia de Software (Recife/PE). SBC, Porto Alegre, RS, Brasil, 25–36. DOI: 10.5753/sbes.2025.9618
Philipp Straubinger and Gordon Fraser. 2023. A Survey on What Developers Think About Testing. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 25–36.
Murat Tasarsu. 2026. Test case generation using large language models: a systematic literature review. Cluster Computing 29, 4 (2026). DOI: 10.1007/s10586-026-06021-z
ZejunWang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 1258–1268. DOI: 10.1145/3691620.3695501
Chen Yang, Junjie Chen, Bin Lin, Ziqi Wang, and Jianyi Zhou. 2026. Advancing Code Coverage: Incorporating Program Analysis with Large Language Models. ACM Trans. Softw. Eng. Methodol. 35, 5, Article 118 (April 2026), 31 pages. DOI: 10.1145/3748505
Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proc. ACM Softw. Eng. 1, FSE, Article 76 (July 2024), 24 pages. DOI: 10.1145/3660783
Junwei Zhang, Xing Hu, Cuiyun Gao, Xin Xia, and Shanping Li. 2026. Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. (March 2026). DOI: 10.1145/3802827 Just Accepted.
Publicado
08/09/2026
Como Citar
MORAES, Djeymisson Rennan De Souza; ROCHA, Thaís Alves Burity.
Comparação Empírica entre EvoSuite e ChatUniTest na Geração de Testes de Unidade: Um Estudo de Replicação com Extensão. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 209-218.
DOI: https://doi.org/10.5753/sast.2026.31499.
