Detecção de Mutantes Equivalentes com MutantBench: Uma Replicação Experimental e Avaliação de Baselines Supervisionados

• Vinicius N. Lopes UNIPAMPA
• Elder Rodrigues UNIPAMPA
• Maicon Bernardino UNIPAMPA

DOI: https://doi.org/10.5753/sast.2026.32319

Resumo

A detecção de mutantes equivalentes permanece como um dos principais desafios do Mutation Testing devido ao elevado custo associado à identificação manual desses casos. Este trabalho apresenta uma replicação parcial baseada no benchmark MutantBench, avaliando classificadores supervisionados leves executados localmente. Foram avaliados Logistic Regression, Linear SVM, Random Forest e KNN com representação TF-IDF baseada em n-gramas de caracteres. O protocolo preserva a divisão oficial entre 1.652 instâncias de treinamento e 1.650 de teste e executa integralmente em CPU. O KNN apresentou o melhor desempenho entre os modelos avaliados, com 92,91% de acurácia, F1-score de 72,08% para a classe equivalente e F1 macro de 84,01%. O Random Forest obteve a maior revocação da classe equivalente, 71,89%, evidenciando perfis de erro distintos. A comparação com resultados da literatura é indireta e não demonstra equivalência entre abordagens. Os resultados estabelecem referências reproduzíveis e de baixo custo para futuras avaliações.

Palavras-chave: Teste de Mutação, Mutantes Equivalentes, Large Language Models, Engenharia de Software, Verificação e Validação

Referências

Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to Represent Programs with Graphs. In International Conference on Learning Representations. OpenReview.net, Vancouver, Canada. [link] //openreview.net/forum?id=BJOFETxR-

Adelaide Danilov, Aria Nourbakhsh, and Christoph Schommer. 2025. Cluster Purge Loss: Structuring Transformer Embeddings for Equivalent Mutants Detection. arXiv:2507.20078 [cs.LG] DOI: 10.48550/arXiv.2507.20078

Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34–41. DOI: 10.1109/C-M.1978.218136

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 1536–1547. DOI: 10.18653/v1/2020.findings-emnlp.139

Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-Training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Dublin, Ireland, 7212–7225. DOI: 10.18653/v1/2022.acl-long.499

Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62

René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, 654–665. DOI: 10.1145/2635868.2635929

Lech Madeyski,Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (2014), 23–42. DOI: 10.1109/TSE.2013.44

A. Jefferson Offutt and Jie Pan. 1997. Automatically Detecting Equivalent Mutants and Infeasible Paths. Software Testing, Verification and Reliability 7, 3 (1997), 165–192. DOI: 10.1002/(SICI)1099-1689(199709)7:3<165::AID-STVR143>3.0.CO;2-U

Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering. IEEE, 936–946. DOI: 10.1109/ICSE.2015.103

Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation Testing Advances: An Analysis and Survey. In Advances in Computers. Vol. 112. Elsevier, 275–378. DOI: 10.1016/bs.adcom.2018.03.015

Zhao Tian, Honglin Shu, Dong Wang, Xuejie Cao, Yasutaka Kamei, and Junjie Chen. 2024. Large Language Models for Equivalent Mutant Detection: How Far Are We? arXiv:2408.01760 [cs.SE] [link]

Lars van Hijfte and Ana Oprescu. 2021. MutantBench: An Equivalent Mutant Problem Comparison Framework. In 2021 IEEE 14th International Conference on Software Testing, Verification and ValidationWorkshops (ICSTW). IEEE, 7–12. DOI: 10.1109/ICSTW52544.2021.00015

BoWang, Mingda Chen, Youfang Lin, Mike Papadakis, and Jie M. Zhang. 2024. An Exploratory Study on Using Large Language Models for Mutation Testing. arXiv:2406.09843 [cs.SE] [link]

Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. arXiv:2305.07922 [cs.CL] [link]

Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-Aware Unified Pre-Trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Punta Cana, Dominican Republic, 8696–8708. DOI: 10.18653/v1/2021.emnlp-main.685

Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A Novel Neural Source Code Representation Based on Abstract Syntax Tree. In Proceedings of the 41st International Conference on Software Engineering (ICSE 2019). IEEE Press, Montreal, QC, Canada, 783–794. DOI: 10.1109/ICSE.2019.00086