Threshold Mutation Operators for Test Data Selection Applied to Machine Learning Testing
Resumo
Machine learning (ML) models are commonly evaluated by sampling-based methods that estimate generalization performance but may fail to uncover fault-exposing inputs. Mutation-based test selection addresses that limitation by introducing controlled perturbations and selecting instances that distinguish the original model from mutated variants. Building on prior decision tree mutation testing (DTMT), this manuscript proposes two threshold-specific mutation operators for decision-tree-based test data selection, namely, local boundary threshold (LBT) and impurity-aware threshold (IAT). Both operators mutate a single internal-node threshold while preserving the original tree topology towards reducing mutant-generation cost and selecting challenging test inputs. Experimental results indicate that LBT and IAT generated fewer mutants than DTMT, selected more challenging test sets than standard sampling references, produced lower equivalent-mutant rates, and recovered around one-third of DTMT-selected instances. Such results suggest that LBT and IAT can serve as complementary, lower-cost test-selection strategies that expand the set of challenging instances identified by mutation-guided selection.
Palavras-chave:
Software Testing, Machine Learning, Decision Tree, Mutation Testing
Referências
Samuel Ackerman, Guy Barash, Eitan Farchi, Orna Raz, and Onn Shehory. 2024. Theory and Practice of Quality Assurance for Machine Learning Systems: An Experiment-Driven Approach. Springer Nature Switzerland, Cham. DOI: 10.1007/978-3-031-70008-8
Gabriel Araujo, Marcos Kalinowski, Markus Endler, and Fabio Calefato. 2024. Professional Insights into Benefits and Limitations of Implementing MLOps Principles. In International Conference on Enterprise Information Systems, ICEIS - Proceedings, Vol. 2. Science and Technology Publications, Lda, 305–312. DOI: 10.5220/0012741100003690
José Augusto Baranauskas and Maria Carolina Monard. 2000. Reviewing Some Machine Learning Concepts and Methods. Technical Report 102. Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos, Brazil. [link] Technical report, ICMC-USP.
Christopher Bishop. 2011. Embracing Uncertainty: Applied Machine Learning Comes of Age. In Machine Learning and Knowledge Discovery in Databases, Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 4–4.
Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Chapman & Hall/CRC, Boca Raton London New York Washington, D.C. DOI: 10.1201/9781315139470
Katia Romero Felizardo et al. 2017. Revisão sistemática da literatura em engenharia de software: teoria e prática. Elsevier.
Katti Faceli, Ana Carolina Lorena, João Gama, Tiago Agostinho de Almeida, and André Carlos Ponce de Leon Ferreira de Carvalho. 2025. Inteligência Artificial: Uma Abordagem de Aprendizado de Máquina (3 ed.). LTC, Rio de Janeiro. 376 pages.
Xiujing Guo, Hiroyuki Okamura, and Tadashi Dohi. 2025. Improving Test Suite Generation Quality Through Machine Learning-Driven Boundary Value Analysis. Array 27 (2025), 100496. DOI: 10.1016/j.array.2025.100496
Khan Mohammad Habibullah, Juan Garcia Diaz, Gregory Gay, and Jennifer Horkoff. 2024. Scoping of Non-Functional Requirements for Machine Learning Systems. In Proc. Int. Conf. Requir. Eng. IEEE Computer Society, 496–497. DOI: 10.1109/RE59067.2024.00061
Kush Jain, Goutamkumar Tulajappa Kalburgi, Claire Le Goues, and Alex Groce. 2023. Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 102–113. DOI: 10.1109/ISSRE59848.2023.00036
Teru Kamogashira. 2022. Machine Learning in Diagnosis Support with Posturography Data. Equilibrium Research 81, 4 (2022), 212–221. DOI: 10.3757/jser.81.212
Markelle Kelly, Rachel Longjohn, and Kolby Notkin. 2025. UCI Machine Learning Repository. DOI: 10.24432/C5K33Z, Available at [link].
Yuta Kobayashi, Yu-Chung Peng, Evan Yu, Brian Bush, Youn-Hoa Jung, Zachary Murphy, Lee Goeddel, Glenn Whitman, Archana Venkataraman, and Charles H. Brown. 2023. Prediction of Lactate Concentrations after Cardiac Surgery Using Machine Learning and Deep Learning Approaches. Frontiers in Medicine 10 (Sept. 2023). DOI: 10.3389/fmed.2023.1165912
José Carlos Maldonado. 2017. Introdução ao teste de software 2a ed. Elsevier.
Beatriz N. C. Silveira, Vinicius H. S. Durelli, Sebastião H. N. Santos, Rafael S. Durelli, Marcio E. Delamaro, and Simone R. S. Souza. 2025. Advancing Test Data Selection by Leveraging Decision Tree Structures: An Investigation into Decision Tree Coverage and Mutation Analysis. Journal of Software Engineering Research and Development 13, 1 (March 2025). DOI: 10.5753/jserd.2025.4084
Peter Norvig. 2022. Inteligência Artificial (4 ed.). Grupo Gen, Rio de Janeiro, RJ.
Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2022. Practical Mutation Testing at Scale: A View from Google. IEEE Transactions on Software Engineering 48, 10 (2022), 3900–3912. DOI: 10.1109/TSE.2021.3107634
Swantje Plambeck and Görschwin Fey. 2023. Data-Driven Test Generation for Black-Box Systems From Learned Decision Tree Models. In 2023 26th International Symposium on Design and Diagnostics of Electronic Circuits and Systems (DDECS). IEEE, 27–32. DOI: 10.1109/DDECS57882.2023.10139633
Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing Machine Learning Based Systems: A Systematic Mapping. Empirical Software Engineering 25, 6 (2020), 5193–5254. DOI: 10.1007/s10664-020-09881-0
Stuart J. Russell and Peter Norvig. 2022. Artificial Intelligence: A Modern Approach (fourth edition, global edition ed.). Pearson, Boston.
Beatriz Silveira, Vinicius Durelli, Sebastião Santos, Rafael Durelli, Marcio Delamaro, and Simone Souza. 2023. Test Data Selection Based on Applying Mutation Testing to Decision Tree Models. In Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing (Campo Grande/MS). SBC, Porto Alegre, RS, Brasil, 38–46. [link]
C C Taylor, etc., D J Spiegelhalter, and D Michie (Eds.). 1994. Machine Learning, Neural and Statistical Classification. Ellis Horwood Ltd, Publisher, Harlow, England.
W. Eric Wong (Ed.). 2001. Mutation Testing for the New Century. Springer US, Boston, MA. DOI: 10.1007/978-1-4757-5939-6
Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Transactions on Software Engineering (2020).
Gabriel Araujo, Marcos Kalinowski, Markus Endler, and Fabio Calefato. 2024. Professional Insights into Benefits and Limitations of Implementing MLOps Principles. In International Conference on Enterprise Information Systems, ICEIS - Proceedings, Vol. 2. Science and Technology Publications, Lda, 305–312. DOI: 10.5220/0012741100003690
José Augusto Baranauskas and Maria Carolina Monard. 2000. Reviewing Some Machine Learning Concepts and Methods. Technical Report 102. Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos, Brazil. [link] Technical report, ICMC-USP.
Christopher Bishop. 2011. Embracing Uncertainty: Applied Machine Learning Comes of Age. In Machine Learning and Knowledge Discovery in Databases, Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 4–4.
Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Chapman & Hall/CRC, Boca Raton London New York Washington, D.C. DOI: 10.1201/9781315139470
Katia Romero Felizardo et al. 2017. Revisão sistemática da literatura em engenharia de software: teoria e prática. Elsevier.
Katti Faceli, Ana Carolina Lorena, João Gama, Tiago Agostinho de Almeida, and André Carlos Ponce de Leon Ferreira de Carvalho. 2025. Inteligência Artificial: Uma Abordagem de Aprendizado de Máquina (3 ed.). LTC, Rio de Janeiro. 376 pages.
Xiujing Guo, Hiroyuki Okamura, and Tadashi Dohi. 2025. Improving Test Suite Generation Quality Through Machine Learning-Driven Boundary Value Analysis. Array 27 (2025), 100496. DOI: 10.1016/j.array.2025.100496
Khan Mohammad Habibullah, Juan Garcia Diaz, Gregory Gay, and Jennifer Horkoff. 2024. Scoping of Non-Functional Requirements for Machine Learning Systems. In Proc. Int. Conf. Requir. Eng. IEEE Computer Society, 496–497. DOI: 10.1109/RE59067.2024.00061
Kush Jain, Goutamkumar Tulajappa Kalburgi, Claire Le Goues, and Alex Groce. 2023. Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 102–113. DOI: 10.1109/ISSRE59848.2023.00036
Teru Kamogashira. 2022. Machine Learning in Diagnosis Support with Posturography Data. Equilibrium Research 81, 4 (2022), 212–221. DOI: 10.3757/jser.81.212
Markelle Kelly, Rachel Longjohn, and Kolby Notkin. 2025. UCI Machine Learning Repository. DOI: 10.24432/C5K33Z, Available at [link].
Yuta Kobayashi, Yu-Chung Peng, Evan Yu, Brian Bush, Youn-Hoa Jung, Zachary Murphy, Lee Goeddel, Glenn Whitman, Archana Venkataraman, and Charles H. Brown. 2023. Prediction of Lactate Concentrations after Cardiac Surgery Using Machine Learning and Deep Learning Approaches. Frontiers in Medicine 10 (Sept. 2023). DOI: 10.3389/fmed.2023.1165912
José Carlos Maldonado. 2017. Introdução ao teste de software 2a ed. Elsevier.
Beatriz N. C. Silveira, Vinicius H. S. Durelli, Sebastião H. N. Santos, Rafael S. Durelli, Marcio E. Delamaro, and Simone R. S. Souza. 2025. Advancing Test Data Selection by Leveraging Decision Tree Structures: An Investigation into Decision Tree Coverage and Mutation Analysis. Journal of Software Engineering Research and Development 13, 1 (March 2025). DOI: 10.5753/jserd.2025.4084
Peter Norvig. 2022. Inteligência Artificial (4 ed.). Grupo Gen, Rio de Janeiro, RJ.
Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2022. Practical Mutation Testing at Scale: A View from Google. IEEE Transactions on Software Engineering 48, 10 (2022), 3900–3912. DOI: 10.1109/TSE.2021.3107634
Swantje Plambeck and Görschwin Fey. 2023. Data-Driven Test Generation for Black-Box Systems From Learned Decision Tree Models. In 2023 26th International Symposium on Design and Diagnostics of Electronic Circuits and Systems (DDECS). IEEE, 27–32. DOI: 10.1109/DDECS57882.2023.10139633
Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing Machine Learning Based Systems: A Systematic Mapping. Empirical Software Engineering 25, 6 (2020), 5193–5254. DOI: 10.1007/s10664-020-09881-0
Stuart J. Russell and Peter Norvig. 2022. Artificial Intelligence: A Modern Approach (fourth edition, global edition ed.). Pearson, Boston.
Beatriz Silveira, Vinicius Durelli, Sebastião Santos, Rafael Durelli, Marcio Delamaro, and Simone Souza. 2023. Test Data Selection Based on Applying Mutation Testing to Decision Tree Models. In Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing (Campo Grande/MS). SBC, Porto Alegre, RS, Brasil, 38–46. [link]
C C Taylor, etc., D J Spiegelhalter, and D Michie (Eds.). 1994. Machine Learning, Neural and Statistical Classification. Ellis Horwood Ltd, Publisher, Harlow, England.
W. Eric Wong (Ed.). 2001. Mutation Testing for the New Century. Springer US, Boston, MA. DOI: 10.1007/978-1-4757-5939-6
Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Transactions on Software Engineering (2020).
Publicado
08/09/2026
Como Citar
BRAGA, Waisman F.; DELAMARO, Marcio E.; SOUZA, Simone R. S..
Threshold Mutation Operators for Test Data Selection Applied to Machine Learning Testing. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 39-48.
DOI: https://doi.org/10.5753/sast.2026.30848.
