A Multimethod Study of Test Smells: Cataloging Removal and New Types

• Elvys Soares UFPE / IFAL
• Márcio Ribeiro UFAL
• André Santos UFPE

Resumo

Test smells are signs in the test code that can indicate potential design or implementation issues. Despite being the subject of extensive academic literature, several questions about their impact in the industry remain unanswered. Some key areas lacking clarity include the absence of a publicly available catalog compiling their types, the lack of correspondence between new test framework features and their ability to prevent or refactor test smells, and the limited knowledge about test smells in manual test suites. To address these knowledge gaps, a multimethod study was conducted. It involved a Multivocal Literature Review, surveys with software testing professionals, studying new test framework features, contributing to popular open-source software, and analyzing manual and automatic test suites. The study resulted in: (i) a catalog that consolidates 480 distinct test smells, (ii) confirmation that new test framework features can refactor and prevent smells, and (iii) eight new smells for manual tests, (iv) a tool based on Natural Language Processing to identify such smells, and (v) their frequency in the tests of the Brazilian Eletronic Voting Machine, the Ubuntu OS and a large smartphones manufacturer. The findings of this work also provide guidance for further study fronts on test smells.

Palavras-chave: Software Quality, Verification and Validation, Software Testing, Test Smells, Test Automation, Test Refactoring, Natural Language Tests