A Road to Find Them All: Towards an Agnostic Strategy for Test Smell Detection

• Gustavo Lopes UFAL
• Davi Romão UFAL
• Elvys Soares UFAL
• Márcio Ribeiro UFAL
• Guilherme Amaral UFAL
• Rohit Gheyi UFCG
• Ivan Machado UFBA

Resumo

Test smells are indications of potential problems in the design and implementation of automated tests. Unit test automation frameworks (xUnit) generally provide similar features for test setup, verification, and teardown steps. Likewise, test smell detection tools perform similar detection steps for smells across programming languages and xUnit frameworks, which might represent a redundant effort. This paper proposes a unified strategy to detect test smells across xUnit frameworks. To do so, we convert the original test code to a standard format among programming languages and xUnit frameworks and search for smells in it, thus promoting the reuse of test smell detection steps. Also, to demonstrate our strategy in practice, we implement a tool named SniffML, which can detect 7 test smells. As the standard format, SniffML considers the srcML library to convert the test code to XML. To evaluate our strategy and tool, we apply SniffML to analyze 300 automated tests from 9 open-source projects written in C++, C#, and Java using GoogleTest, xUnit.NET, and JUnit, respectively. In our study, (i) we evaluate its performance through precision, recall, and f-measure metrics; and (ii) we compare its performance to other popular test smell detection tools (i.e., xNose for C# and JNose and tsDetect for Java). As results, we achieve a precision of (97.99%), a recall of (96.90%), and an f-measure of (97.44%), indicating promising results in detecting test smells across xUnit frameworks. We achieve similar results when comparing SniffML to xNose, JNose and tsDetect, demonstrating the positive potential of our approach. We also contribute to the first test smell detection tool that supports the Google framework, i.e., GoogleTest.

Palavras-chave: verification and validation, automated software testing, test smells