LLM-Pyro: LLM-Based Mutation Testing with Equivalence Triage for Python
Resumo
Mutation testing assesses test suite quality by injecting controlled faults into production code. Recent studies show that large language models (LLMs) can generate more diverse mutants, closer to real faults, than traditional syntactic operators, though at the cost of more non-compilable, duplicate, and equivalent mutants. This evidence, however, is concentrated in Java, leaving Python’s dynamic typing largely unexplored, and existing tools either rely on fixed operators or apply LLMs without addressing equivalent mutants. We present LLM-Pyro, a Python adaptation of LLMorpheus that couples mutant generation with an LLM-based oracle classifying survivors as equivalent, non-equivalent, or unknown. We evaluate LLM-Pyro in a pilot study on HumanEval against mutmut and Mutahunter, and on real faults from PyBugHive. On HumanEval, LLM-Pyro produced more syntactically diverse mutants (average edit distance 2.96), with 72.2% compilability, 15.3% duplication, and 10.4% equivalence, generating 1.85× more mutants than mutmut and 4.53× more than Mutahunter (58.4% non-equivalent). On PyBugHive, within the 44 evaluated bugs from four projects, LLM-Pyro reached an 84.9% mutation score (vs. 77.9% for Mutahunter), suggesting its mutants explore code regions relevant to real faults, though only a fraction match the exact fault taxonomy. These results provide initial evidence, within the scope of the subjects evaluated, that LLM-based mutation with equivalence triage is feasible for Python.
Referências
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, PeterWelinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. (2021). arXiv:2107.03374 [cs.LG]
CodeIntegrity AI. 2025. Mutahunter: Open-Source Language Agnostic LLM-based Mutation Testing. [link].
Asma Hamidi, Ahmed Khanfir, and Mike Papadakis. 2025. Intent-Based Mutation Testing: From Naturally Written Programming Intents to Mutants. In 2025 IEEE International Conference on Software Testing, Verification and ValidationWorkshops (ICSTW). 347–357. DOI: 10.1109/ICSTW64639.2025.10962508
Ali Reza Ibrahimzada, Yang Chen, Ryan Rong, and Reyhaneh Jabbarvand. 2025. Challenging Bug Prediction and Repair Models with Synthetic Bugs. In 2025 IEEE International Conference on Source Code Analysis Manipulation (SCAM). 133–144. DOI: 10.1109/SCAM67354.2025.00021
René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (San Jose, CA, USA) (ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 437–440. DOI: 10.1145/2610384.2628055
Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2021. Does Mutation Testing Improve Testing Practices?. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 910–921. DOI: 10.1109/ICSE43902.2021.00087
Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2022. Practical Mutation Testing at Scale: A view from Google. IEEE Transactions on Software Engineering 48, 10 (2022), 3900–3912. DOI: 10.1109/TSE.2021.3107634
Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2023. Please fix this mutant: How do developers resolve mutants surfaced during code review?. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 150–161. DOI: 10.1109/ICSE-SEIP58684.2023.00019
Frank Tip, Jonathan Bell, and Max Schäfer. 2025. LLMorpheus: Mutation Testing Using Large Language Models. IEEE Transactions on Software Engineering 51, 6 (2025), 1645–1665. DOI: 10.1109/TSE.2025.3562025
Marco Tulio Valente. 2020. Engenharia de Software Moderna: Princípios e Práticas para Desenvolvimento de Software com Produtividade. Editora: Independente.
BoWang, Mingda Chen, Ming Deng, Youfang Lin, Mark Harman, Mike Papadakis, and Jie M. Zhang. 2026. A Comprehensive Study on Large Language Models for Mutation Testing. ACM Trans. Softw. Eng. Methodol. (March 2026). DOI: 10.1145/3805038 Just Accepted.
Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. 2020. BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1556–1560. DOI: 10.1145/3368089.3417943
Kaizhong Zhang and D. Shasha. 1989. Simple Fast Algorithms for the Editing Distance Between Trees and Related Problems. SIAM J. Comput. 18 (1989), 1245–1262. DOI: 10.1137/0218082
