Evaluating Test Quality in GitHub Repositories: A Comparative Analysis of CI/CD Practices Using GitHub Actions
Resumo
Tests are present in different stages of software development and perform an important role throughout the lifecycle of an application. Although, test smells are undesirable, as they characterize poorly designed tests that negatively impact their quality. Therefore, the goal of this study is to investigate the quality of tests in GitHub repositories that implement or do not implement Continuous Integration and Continuous Delivery (CI/CD). To achieve this goal, we conduct a comparative analysis between repositories that use GitHub Actions as CI/CD environments and repositories that do not use CI/CD. To evaluate the quality of the test suite, test smells were detected, and bug issues and the time taken to fix these issues were analyzed. In total, 651 repositories that use GitHub Actions and 289 that do not use CI/CD were analyzed. As a result, 1,648,254 test smells were identified in repositories that use GitHub Actions and 709,680 in repositories that do not. It was found that 86.18% of bug-type issues were closed in repositories that use GitHub Actions, compared to 89.20% in those that do not. Furthermore, the median time for resolving bug-type issues was 156 hours in repositories using GitHub Actions, compared to 178 hours in those without CI/CD. Finally, after statistical tests, it was not possible to state that the use of GitHub Actions in the repositories improves the quality of the tests implemented.
Referências
Keheliya Gallaba, Maxime Lamothe, and Shane McIntosh. Lessons from eight years of operational data from a continuous integration service: An exploratory case study of circleci. In 2022 IEEE/ACM ICSE, pages 1330–1342, 2022.
Vahid Garousi, Baris Kucuk, and Michael Felderer. What we know about smells in software test code. IEEE Software, 36(3):61–73, 2019.
Shalini Joshi and Indra Kumari. Analyses of software testing approaches. In 2022 International Interdisciplinary Humanitarian Conference for Sustainability (IIHC), pages 1276–1281, 2022.
Pei Liu, Xiaoyu Sun, Yanjie Zhao, Yonghui Liu, John Grundy, and Li Li. A first look at ci/cd adoptions in open-source android apps. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE ’22, New York, NY, USA, 2023. ACM.
Akshit Raj Patel and Sulabh Tyagi. The state of test automation in devops: A systematic literature review. In Proceedings of the 2022 Fourteenth International Conference on Contemporary Computing, IC3-2022, page 689–695, New York, NY, USA, 2022. ACM.
Sk Golam Saroar and Maleknaz Nayebi. Developers’ perception of github actions: A survey analysis. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, EASE ’23, page 121–130, New York, NY, USA, 2023. ACM.
Davide Spadini, Martin Schvarcbacher, Ana-Maria Oprescu, Magiel Bruntink, and Alberto Bacchelli. Investigating severity thresholds for test smells. In Proceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, page 311–321, New York, NY, USA, 2020. ACM.
Tássio Virgínio, Luana Martins, Larissa Rocha, Railana Santana, Adriana Cruz, Heitor Costa, and Ivan Machado. Jnose: Java test smell detector. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering, pages 564–569, 2020.
C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell, and A. Wesslén. Experimentation in Software Engineering. Computer Science. Springer Berlin Heidelberg, 2012.
Justyna Zander-Nowicka, Pieter J. Mosterman, and Ina Schieferdecker. Quality of test specification by application of patterns. In Proceedings of the 15th Conference on Pattern Languages of Programs, PLoP ’08, New York, NY, USA, 2008. ACM.
Yi Zhao, Yun Hu, and Jiayu Gong. Research on international standardization of software quality and software testing. In 2021 IEEE/ACIS 20th International Fall Conference on Computer and Information Science (ICIS Fall), pages 56–62, 2021.
