Mutation Testing in the Wild: Challenges and Lessons from Bitcoin Core

• Bruno Ely Reis Garcia Vinteum / USP

DOI: https://doi.org/10.5753/sast.2026.30421

Resumo

Mutation testing is a technique for evaluating test suite effectiveness, yet its adoption in industrial settings remains limited due to scalability challenges. In this paper, we report our experience applying mutation testing to Bitcoin Core, the reference implementation of the Bitcoin protocol - a safety-critical, C++ codebase with hundreds of thousands of lines of code and a heterogeneous test suite. We present bcore-mutation, an open-source tool written in Rust and tailored to Bitcoin Core’s specific constraints, and describe the challenges we faced: test selection for incremental analysis, unproductive mutant filtering, and surfacing results to pull request authors without disrupting the review workflow.We further discuss how mutation testing was applied to evaluate the quality of fuzz harnesses, and report an experimental use of Large Language Models to automatically generate tests that kill surviving mutants. Our experience shows that mutation testing is tractable even on large, complex codebases when the tooling is designed around the realities of the project, and offers lessons for practitioners considering similar approaches on safety-critical systems.

Palavras-chave: Mutation Testing, Software Testing, Open Source

Referências

Moritz Beller, Chu-Pan Wong, Johannes Bader, Andrew Scott, Mateusz Machalica, Satish Chandra, and Erik Meijer. 2021. What it would take to use mutation testing in industry—a study at facebook. In 2021 IEEE/ACM43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 268–277.

Bruno Garcia and Simone Souza. 2025. An empirical evaluation of fuzz targets using mutation testing. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 75–83. DOI: 10.5753/sast.2025.14183

K. Jain, G. Kalburgi, C. Le Goues, and A. Groce. 2023. Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE Computer Society, Los Alamitos, CA, USA, 102–113. DOI: 10.1109/ISSRE59848.2023.00036

Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2022. Practical Mutation Testing at Scale: A view from Google. IEEE Transactions on Software Engineering 48, 10 (2022), 3900–3912. DOI: 10.1109/TSE.2021.3107634