Guilherme Freitas
ABSTRACT
Continuous Integration (CI) is a widely adopted practice that automates and enhances the frequency of code integration. Previous research has explored the relationship between CI sub-practices (such as frequent commit activity and high test coverage) and software quality. However, limited knowledge exists regarding the impact of specific CI sub-practices on the Continuous Code Quality (CCQ) inspection outcomes of software projects, such as technical debts, bug density, duplicated lines, and code smells. This paper aims to analyze the extent to which the adoption of CI sub-practices improves CCQ outcome metrics in software projects. First, we investigate the association between Travis CI adoption, Travis CI usage maturity, and improved quality inspection outcomes in a set of 75 popular open-source projects from GitHub. Additionally, we use data from other 97 open-source projects to identify specific CI sub-practices, such as maintaining shorter build durations, that exhibit a strong correlation with enhanced quality inspection outcomes. Our findings reveal that the quality inspection outcomes are improved in projects with higher test coverage and shorter build duration. Merely adopting a CI service is not a guarantee for improved quality outcomes. Therefore, it is essential for developers to consistently adhere to recommended CI sub-practices. By doing so, they can effectively harness the benefits of CI in their projects and achieve improved quality outcomes.
- [n. d.]. SonarQube 7.9 Documentation. https://docs.sonarqube.org/latest/. Accessed: 2019-08-21.Google Scholar
- M. Beller, G. Gousios, and A. Zaidman. 2017. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. In Proceedings of the 14th International Conference on Mining Software Repositories (Buenos Aires, Argentina) (MSR ’17). IEEE Press, Piscataway, NJ, USA, 356–367.Google Scholar
- Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. TravisTorrent: synthesizing Travis CI and GitHub for full-stack research on continuous integration. In Proceedings of the 14th International Conference on Mining Software Repositories, MSR 2017, Buenos Aires, Argentina, May 20-28, 2017. 447–450.Google ScholarDigital Library
- João Helis Bernardo, Daniel Alencar da Costa, and Uirá Kulesza. 2018. Studying the Impact of Adopting Continuous Integration on the Delivery Time of Pull Requests. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). ACM, New York, NY, USA, 131–141.Google ScholarDigital Library
- Graham Brooks. 2008. Team pace keeping build times down. In Agile 2008 Conference. IEEE, 294–297.Google ScholarDigital Library
- Tsun Chow and Dac-Buu Cao. 2008. A survey study of critical success factors in agile software projects. Journal of systems and software 81, 6 (2008), 961–971.Google ScholarDigital Library
- Kevin Crowston, Hala Annabi, and James Howison. 2003. Defining open source software project success. ICIS 2003 Proceedings (2003), 28.Google Scholar
- G. Digkas, M. Lungu, P. Avgeriou, A. Chatzigeorgiou, and A. Ampatzoglou. 2018. How do developers fix issues and pay back technical debt in the Apache ecosystem?. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). 153–163.Google Scholar
- Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.Google Scholar
- Sebastian Elbaum, David Gable, and Gregg Rothermel. 2001. The impact of software evolution on code coverage information. In Proceedings of the IEEE International Conference on Software Maintenance. IEEE Computer Society, 170.Google ScholarDigital Library
- Wagner Felidré, Leonardo Furtado, Daniel A da Costa, Bruno Cartaxo, and Gustavo Pinto. 2019. Continuous integration theater. (2019), 1–10.Google Scholar
- Martin Fowler. 2018. Refactoring: improving the design of existing code. Addison-Wesley Professional.Google ScholarDigital Library
- Martin Fowler and Matthew Foemmel. 2006. Continuous integration. Thought-Works) http://www. thoughtworks. com/Continuous Integration. pdf (2006).Google Scholar
- Keheliya Gallaba, Christian Macho, Martin Pinzger, and Shane McIntosh. 2018. Noise and heterogeneity in historical build data: an empirical study of Travis CI. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 87–97.Google ScholarDigital Library
- T. A. Ghaleb, D. A. da Costa, and Y Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering (2019), 1–38.Google Scholar
- Michael Hilton, Jonathan Bell, and Darko Marinov. 2018. A large-scale study of test coverage evolution. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 53–63.Google ScholarDigital Library
- Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: assurance, security, and flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 197–207.Google ScholarDigital Library
- M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig. 2016. Usage, Costs, and Benefits of Continuous Integration in Open-source Projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE 2016). 426–437.Google ScholarDigital Library
- Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In ASE 2016 - Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. Association for Computing Machinery, Inc, 426–437.Google ScholarDigital Library
- C. Leys, C. Ley, O. Klein, P. Bernard, and L. Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology 49, 4 (2013), 764–766.Google ScholarCross Ref
- Welder Pinheiro Luz, Gustavo Pinto, and Rodrigo Bonifácio. 2018. Building a collaborative culture: a grounded theory of well succeeded devops adoption in practice. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018, Oulu, Finland, October 11-12, 2018. 6:1–6:10.Google ScholarDigital Library
- T. Mårtensson, D. Ståhl, and J Bosch. 2019. Test activities in the continuous integration and delivery pipeline. Journal of Software: Evolution and Process (2019), e2153.Google Scholar
- K. V. R. Paixão, C. Z. Felício, F. M. Delfim, and M. de A. Maia. 2017. On the Interplay Between Non-functional Requirements and Builds on Continuous Integration. In Proceedings of the 14th International Conference on Mining Software Repositories (Buenos Aires, Argentina) (MSR ’17). 479–482.Google ScholarDigital Library
- Gustavo Pinto, Marcel Rebouças, and Fernando Castor. 2017. Inadequate Testing, Time Pressure, and (Over) Confidence: A Tale of Continuous Integration Users. In 10th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE@ICSE 2017. 74–77.Google ScholarDigital Library
- F. Rahman, C. Bird, and P. Devanbu. 2012. Clones: What is that smell?Empirical Software Engineering 17, 4-5 (2012), 503–530.Google Scholar
- John S Reel. 1999. Critical success factors in software projects. IEEE software 16, 3 (1999), 18–23.Google ScholarDigital Library
- Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys. In annual meeting of the Florida Association of Institutional Research. 1–33.Google Scholar
- Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2022. Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects. In Proceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 137–147.Google ScholarDigital Library
- D. Saraiva, D. A. Da Costa, U. Kulesza, G. Sizílio, J. G. Neto, R. Coelho, and M Nagappan. 2023. Unveiling the Relationship Between Continuous Integration and Code Coverage. In IEEE/ACM International Conference on Mining Software Repositories (MSR). 247–259.Google Scholar
- Gustavo Sizílio, Daniel Alencar da Costa, and Uirá Kulesza. 2019. An Empirical Study of the Relationship between Continuous Integration and Test Code Evolution. In Proceedings of the 35th International Conferenceon Software Maintenance and Evolution (Cleveland, United States) (ICSME ’19). IEEE, New York, NY, USA.Google ScholarCross Ref
- Eliezio Soares, Gustavo Sizilio, Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2022. The effects of continuous integration on software development: a systematic literature review. Empirical Software Engineering 27, 3 (2022), 78.Google ScholarDigital Library
- Girish Suryanarayana, Ganesh Samarthyam, and Tushar Sharma. 2014. Refactoring for software design smells: managing technical debt. Morgan Kaufmann.Google Scholar
- Michele Tufano, Fabio Palomba, Gabriele Bavota, Rocco Oliveto, Massimiliano Di Penta, Andrea De Lucia, and Denys Poshyvanyk. 2015. When and why your code starts to smell bad. In Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, 403–414.Google ScholarCross Ref
- B. Vasilescu, S. van Schuylenburg, J. Wulms, A. Serebrenik, and M. G. J. van den Brand. 2014. Continuous Integration in a Social-Coding World: Empirical Evidence from GitHub. In Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution(ICSME ’14). IEEE Computer Society, Washington, DC, USA, 401–405.Google ScholarDigital Library
- B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). 805–816.Google ScholarDigital Library
- B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). ACM, New York, NY, USA, 805–816.Google ScholarDigital Library
- Carmine Vassallo, Fabio Palomba, Alberto Bacchelli, and Harald C. Gall. 2018. Continuous Code Quality: Are We (Really) Doing That?. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France) (ASE 2018). ACM, New York, NY, USA, 790–795.Google ScholarDigital Library
- D. G. Widder, M. Hilton, C. Kästner, and B. Vasilescu. 2019. A conceptual replication of continuous integration pain points in the context of Travis CI. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 647–658.Google Scholar
- Yang Zhang, Bogdan Vasilescu, Huaimin Wang, and Vladimir Filkov. 2018. One size does not fit all: an empirical study of containerized continuous deployment workflows. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, 2018. 295–306.Google ScholarDigital Library
- Y. Zhao, A. Serebrenik, Y. Zhou, V. Filkov, and B. Vasilescu. 2017. The impact of continuous integration on other software development practices: A large-scale empirical study. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 60–71.Google Scholar
Index Terms
- Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical Study
Recommendations
Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects
ESEM '22: Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and MeasurementBackground: Much research has been conducted to investigate the impact of Continuous Integration (CI) on the productivity and quality of open-source projects. Most of studies have analyzed the impact of adopting a CI server service (e.g, Travis-CI) but ...
Usage, costs, and benefits of continuous integration in open-source projects
ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software EngineeringContinuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, ...
Analyzing Continuous Integration Bad Practices in Closed-Source Projects: An Initial Study
SBES '20: Proceedings of the XXXIV Brazilian Symposium on Software EngineeringContinuous integration (CI) is a process widely used in projects of different types and sizes, generally aimed at improving development and also ensuring the quality of the software. However, some bad practices can mitigate or invalidate the beneficial ...
Comments