A Controlled Experiment for Combinatorial Testing

• Juliana Marino Balera INPE
• Valdivino Alexandre de Santiago Júnior INPE

Resumo

In this paper, we present a controlled experiment for combinatorial designs algorithms aiming at software test case generation. We compare our recently proposed algorithm, TTR, to generate Mixed-Level Covering Array (MCA) with four other well-known combinatorial designs algorithms/tools regarding two aspects: cost in terms of the size of the set of test cases, and cost in terms of the time to generate the test suites. We used a set of 27 instances for this experiment. Results show that our algorithm was the best in terms of the size of the test suite, but was the poorest in terms of the time to generate the test cases. However, the not so good performance of our algorithm regarding the time to generate the test suite can be alleviated by the fact that TTR produces shorter set of test cases to be executed. We also made a comparison about the similarity of the test cases, i.e. to realize how similar are the test suites (test input data) of TTR compared with the other four algorithms/tools. We conclude that the TTR's test suite is not similar to any other test suites meaning that our algorithm has the potential to uncover different software defects by exercising different parts of the Software Under Test (SUT).

Palavras-chave: Combinatorial Designs, Controlled Experiment, Software Testing