Evaluating Different Strategies for Reduction of Mutation Testing Costs

• Jackson A. Prado Lima UFPR
• Giovani Guizzo UFPR
• Silvia R. Vergilio UFPR
• Alan P. C. Silva UFPR
• Helson L. Jakubovski Filho UFPR
• Henrique V. Ehrenfried UFPR

Abstract

Mutation testing presents high efficacy in terms of revealed faults, but with high computational costs, because test cases must be executed against a great number of mutants. To reduce such costs several strategies exist. In general, they select a reduced number of mutants that maintain a great overall mutation score. Recently, Higher-Order Mutation Testing (HOM) was proposed. HOM introduces more than one fault at a time in the mutants such that they are harder to kill. This kind of testing can also be used as a cost reduction strategy. However, few works in literature evaluate the cost reduction potential of HOM or compare it to other existing strategies. In this work, we evaluate HOM as a mutation testing cost reduction strategy. We conducted an experiment to compare four HOM-based strategies (First to Last, Random Mix, Different Operators, and Each-Choice) and three conventional ones (Random Mutant Selection, Selective Mutation, and Search-Based Mutation by using Genetic Algorithm). The analysis considers the number of selected mutants, number of test cases, mutation score and efficiency measures. Selective Mutation presented the best results overall. Among the HOM-based strategies, Each-Choice was the best.

Keywords: Fault-based Testing, Higher-Order Mutation, Mutation Testing Strategies