How far are we from testing a program in a completely automated way, considering the mutation testing criterion at unit level?

• Filipe Santos Araujo UFSCAR https://orcid.org/0000-0001-8969-2717
• Auri Vincenzi UFSCAR https://orcid.org/0000-0001-5902-1672

Resumo

Testing is a mandatory activity to guarantee software quality. Not only knowledge about the software under testing is required to generate high-quality test cases, but also knowledge about the business rules implemented in software product to cover more than 80% of its source code. Therefore, we investigate in this study the adequacy, effectiveness, and cost of smart and random automated generated test sets for Java programs. We observed that the smart generated test sets, in general, are more adequate and less expensive than random generated tests, but regarding effectiveness, random generated test are more efficient. Moreover, we observed that smart automated test sets are complementary between them, and we explored if random generated test sets could be complementary to smart automated test sets as well. When we combined smart generated test sets, we observed an increase of more than 8% in statement coverage and more than 15% in mutation score when compared to random generated test sets. However, when we added random generated test sets to previous combination of smart generated test sets, results show a lower increase of statement coverage and mutation score, while increasing considerably the test set generation cost. Therefore, we advocate that the use of random testing should be integrated with smart generated tests only with a minimization strategy to avoid redundant test sets, keeping the cost reasonable.

Palavras-chave: Software Testing, Automated Test Data Generator, Coverage Testing Mutation Testing, Test Set Combination