On the influence of Test Smells on Test Coverage
Resumo
Software testing is a key practice in the software quality assurance process. Usually, the quality of a test is not analyzed before its execution, i.e., there are no tests to check the tests. When the quality of tests is not guaranteed, it may impair the quality of the software. Test Smells are an alternative to indicate problems in the test code that can affect test maintainability, more specifically readability and comprehension. This study investigates correlations between test coverage and test smells types. We also introduce the JNose Test, a tool to automate test smells detection. We analyzed 11 open source projects and detected 21 types of smells and 10 different test coverage metrics to each test class. We identified 63 out of 210 calculated correlations. Our results show that there is a relationship between test smells and test coverage, in which test smells may influence code coverage. Our findings might support software testers and help them understand the behavior and consequences of poorly written and designed tests.
Palavras-chave:
Software engineering
Referências
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? An empirical study. Empirical Software Engineering 20, 4 (2015).
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David W. Binkley. 2012. An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In 28th IEEE International Conference on Software Maintenance (ICSM).
David Bowes, Tracy Hall, Jean Petrić, Thomas Shippey, and Burak Turhan. 2017. How Good Are My Tests?. In 2017 IEEE/ACM 8th Workshop on Emerging Trends in Software Metrics (WETSoM).
Adalberto Nobiato Crespo, Odair J Silva, Carlos Alberto Borges, Clênio Figueiredo Salviano, M Argollo, and Mario Jino. 2004. Uma metodologia para teste de Software no Contexto da Melhoria de Processo. In Simpósio Brasileiro de Qualidade de Software (SBQS).
Arie Deursen, Leon M.F. Moonen, A. Bergh, and Gerard Kok. 2001. Refactoring Test Code. Technical Report. Amsterdam, The Netherlands, The Netherlands.
Gerald D. Everett and Raymond McLeod Jr. 2007. Automated Testing Tools (1 ed.). Wiley-IEEE Computer Society Press.
Vahid Garousi and Banış Küçük. 2018. Smells in software test code: A survey of knowledge in industry and academia. Journal of Systems and Software 138 (2018).
Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code Coverage for Suite Evaluation by Developers. In Proceedings of the 36th International Conference on Software Engineering (ICSE).
Suhaimi Ibrahim and Shahid Muhammad. 2011. An Evaluation of Test Coverage Tools in Software Testing. In International Conference On Telecommunication Technology And Application (ICTTA).
Gerard Meszaros. 2007. xUnit test patterns: refactoring test code (1st ed.). Addison-Wesley.
Fabio Palomba, Dario Di Nucci, Annibale Panichella, Rocco Oliveto, and Andrea De Lucia. 2016. On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study. In IEEE/ACM 9th International Workshop on Search-Based Software Testing (SBST).
Fabio Palomba, Andy Zaidman, and Andrea De Lucia. 2018. Automatic Test Smell Detection Using Information Retrieval Techniques. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME).
Anthony Peruma. 2018. What the Smell? An Empirical Investigation on the Distribution and Severity of Test Smells in Open Source Android Applications.
Roger Pressman. 2016. Software Engineering: A Practitioner's Approach (8th ed.). McGraw-Hill, Inc., New York, NY.
Bart Van Rompaey, Bart Du Bois, Serge Demeyer, and Matthias Rieger. 2007. On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test. IEEE Transactions on Software Engineering 33, 12 (Dec 2007).
Neil J Salkind and Terese Rainwater. 2005. Exploring research (6 ed.). Prentice Hall.
Davide Spadini, Fabio Palomba, Andy Zaidman, Magiel Bruntink, and Alberto Bacchelli. 2018. On the Relation of Test Smells to Software Code Quality. In IEEE International Conference on Software Maintenance and Evolution (ICSME).
Bharti Suri and Shweta Singhal. 2015. Investigating the OO characteristics of software using CKJM metrics. In 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions).
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An Empirical Investigation into the Nature of Test Smells. In 31st IEEE/ACM Intern. Conf. on Automated Software Engineering (ASE). ACM, New York, NY, USA.
Vahid Garousi Yusifoğlu, Yasaman Amannejad, and Aysu Betin Can. 2015. Software test-code engineering: A systematic mapping. Information and Software Technology 58 (2015).
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David W. Binkley. 2012. An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In 28th IEEE International Conference on Software Maintenance (ICSM).
David Bowes, Tracy Hall, Jean Petrić, Thomas Shippey, and Burak Turhan. 2017. How Good Are My Tests?. In 2017 IEEE/ACM 8th Workshop on Emerging Trends in Software Metrics (WETSoM).
Adalberto Nobiato Crespo, Odair J Silva, Carlos Alberto Borges, Clênio Figueiredo Salviano, M Argollo, and Mario Jino. 2004. Uma metodologia para teste de Software no Contexto da Melhoria de Processo. In Simpósio Brasileiro de Qualidade de Software (SBQS).
Arie Deursen, Leon M.F. Moonen, A. Bergh, and Gerard Kok. 2001. Refactoring Test Code. Technical Report. Amsterdam, The Netherlands, The Netherlands.
Gerald D. Everett and Raymond McLeod Jr. 2007. Automated Testing Tools (1 ed.). Wiley-IEEE Computer Society Press.
Vahid Garousi and Banış Küçük. 2018. Smells in software test code: A survey of knowledge in industry and academia. Journal of Systems and Software 138 (2018).
Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code Coverage for Suite Evaluation by Developers. In Proceedings of the 36th International Conference on Software Engineering (ICSE).
Suhaimi Ibrahim and Shahid Muhammad. 2011. An Evaluation of Test Coverage Tools in Software Testing. In International Conference On Telecommunication Technology And Application (ICTTA).
Gerard Meszaros. 2007. xUnit test patterns: refactoring test code (1st ed.). Addison-Wesley.
Fabio Palomba, Dario Di Nucci, Annibale Panichella, Rocco Oliveto, and Andrea De Lucia. 2016. On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study. In IEEE/ACM 9th International Workshop on Search-Based Software Testing (SBST).
Fabio Palomba, Andy Zaidman, and Andrea De Lucia. 2018. Automatic Test Smell Detection Using Information Retrieval Techniques. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME).
Anthony Peruma. 2018. What the Smell? An Empirical Investigation on the Distribution and Severity of Test Smells in Open Source Android Applications.
Roger Pressman. 2016. Software Engineering: A Practitioner's Approach (8th ed.). McGraw-Hill, Inc., New York, NY.
Bart Van Rompaey, Bart Du Bois, Serge Demeyer, and Matthias Rieger. 2007. On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test. IEEE Transactions on Software Engineering 33, 12 (Dec 2007).
Neil J Salkind and Terese Rainwater. 2005. Exploring research (6 ed.). Prentice Hall.
Davide Spadini, Fabio Palomba, Andy Zaidman, Magiel Bruntink, and Alberto Bacchelli. 2018. On the Relation of Test Smells to Software Code Quality. In IEEE International Conference on Software Maintenance and Evolution (ICSME).
Bharti Suri and Shweta Singhal. 2015. Investigating the OO characteristics of software using CKJM metrics. In 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions).
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An Empirical Investigation into the Nature of Test Smells. In 31st IEEE/ACM Intern. Conf. on Automated Software Engineering (ASE). ACM, New York, NY, USA.
Vahid Garousi Yusifoğlu, Yasaman Amannejad, and Aysu Betin Can. 2015. Software test-code engineering: A systematic mapping. Information and Software Technology 58 (2015).
Publicado
23/09/2019
Como Citar
VIRGÍNIO, Tássio; SANTANA, Railana; MARTINS, Luana Almeida; SOARES, Larissa Rocha; COSTA, Heitor; MACHADO, Ivan.
On the influence of Test Smells on Test Coverage. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SOFTWARE (SBES), 33. , 2019, Salvador.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
