Assessing Android Test Data Generation Tools via Mutation Testing

• Henrique Neves da Silva UFPR
• Paulo Roberto Farah UFPR / UDESC
• Willian Douglas Ferrari Mendonça UFPR / Univel
• Silvia Regina Vergilio UFPR

Resumo

A growing number of test data generation techniques and tools for Android applications (apps) has been proposed in the last years. As a consequence, a demand for evaluations comparing such tools has emerged. However, we find few studies only dedicated to this subject and there is a lack of studies considering the mutation score, spite of this is a measure largely used and recognized as effective to assess the quality of the test suites. A possible reason is the fact that Android mutation testing has been recently addressed in the literature, and most tools do not support the analyses of mutants in comparison with the original one, nor provide the score. To fulfill with these gaps, this work presents results from the evaluation of three state-of-the-art tools for Android apps: Monkey, Stoat and APE, regarding the ability of the test data generated by them to reveal faults described by the mutation operators of the tool MDroid+. To this end, we implemented a mechanism to automatically capture screenshots of apps execution and calculate the score, which is also described in the paper. In addition to this, we also evaluate aspects related to code coverage and runtime. Stoat reached the best general mean score, but Monkey takes significantly less time to execute, without great differences in the score and coverage.

Palavras-chave: Mutation score, Mobile test, Android apps