Holter: Monitoring Continuous Integration Practices
Resumo
Background: Despite widespread adoption and an extensive body of research evaluating Continuous Integration (CI), there is evidence that not all CI practices are fully adopted. Aims: To help the full adoption of CI, we present in this paper Holter - a tool that automatically calculates and monitors a suite of metrics associated with CI practices. The tool enables developers to continuously access these metrics and receive alerts regarding the evolution of CI practices. Method: We illustrate the usage of our tool for monitoring CI practices and compare the Holter monitoring dashboard with those of existing CI services and third-party tools. Results: Leading CI services and complementary tools still offer basic support for monitoring CI practices. Focusing on monitoring fundamental information related to the performance of the CI pipeline, Holter provides a more comprehensive perspective of CI that cannot be found in other tools. Conclusions: Holter can be used to monitor and improve CI practices in software development projects. Additionally, it can provide an overview of a project’s CI maturity. Video link: https://doi.org/10.6084/m9.figshare.25880083
Palavras-chave:
CI Practices, Monitoring, Case Study, CI Maturity
Referências
Thomas Bach, Artur Andrzejak, Ralf Pannemans, and David Lo. 2017. The Impact of Coverage on Bug Density in a Large Industrial Software Project. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 307–313. DOI: 10.1109/ESEM.2017.44
João Helis Bernardo, Daniel Alencar da Costa, and Uirá Kulesza. 2018. Studying the Impact of Adopting Continuous Integration on the Delivery Time of Pull Requests. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New York, NY, USA, 131–141. DOI: 10.1145/3196398.3196421
Nathan Cassee, Bogdan Vasilescu, and Alexander Serebrenik. 2020. The Silent Helper: The Impact of Continuous Integration on Code Reviews. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 423–434. DOI: 10.1109/SANER48275.2020.9054818
Paul Duvall, Stephen M. Matyas, and AndrewGlover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series). Addison-Wesley Professional.
Omar Elazhary, Colin Werner, Ze Shi Li, Derek Lowlind, Neil A Ernst, and Margaret-Anne Storey. 2021. Uncovering the benefits and challenges of continuous integration practices. IEEE Transactions on Software Engineering 48, 7 (2021), 2570–2583.
Wagner Felidré, Leonardo Furtado, Daniel A. da Costa, Bruno Cartaxo, and Gustavo Pinto. 2019. Continuous Integration Theater. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–10. DOI: 10.1109/ESEM.2019.8870152
Martin Fowler. 2006. Continuous Integration. Retrieved november 24, 2021 from [link]
Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology 106 (2019), 101–121.
Vahid Garousi, Michael Felderer, Mika V Mäntylä, and Austen Rainer. 2020. Benefitting from the grey literature in software engineering research. In Contemporary Empirical Methods in Software Engineering. Springer, 385–413.
Taher Ahmed Ghaleb, Daniel Alencar Da Costa, and Ying Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering 24 (2019), 2102–2139.
Mehdi Golzadeh, Alexandre Decan, and Tom Mens. 2022. On the rise and fall of CI services in GitHub. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 662–672. DOI: 10.1109/SANER53432.2022.00084
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-Offs in Continuous Integration: Assurance, Security, and Flexibility. Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/3106237.3106270
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). 426–437.
Eero Laukkanen, Maria Paasivaara, and Teemu Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration – A Case Study. In 2015 Agile Conference. 11–20. DOI: 10.1109/Agile.2015.15
Jadson Santos. 2024. A Deep Dive into Continuous Integration Monitoring Practices. Ph.D. Dissertation. PPGSC/UFRN.
Eliezio Soares, Gustavo Sizílio, Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2021. The Effects of Continuous Integration on Software Development: a Systematic Literature Review. CoRR abs/2103.05451 (2021). arXiv:2103.05451 [link]
Daniel Ståhl and Jan Bosch. 2013. Experienced benefits of continuous integration in industry software product development: A case study. In The 12th iasted international conference on software engineering,(innsbruck, austria, 2013). 736–743.
Daniel Ståhl and Jan Bosch. 2013. Experienced benefits of continuous integration in industry software product development: A case study. In The 12th iasted international conference on software engineering,(innsbruck, austria, 2013). 736–743.
Daniel Ståhl and Jan Bosch. 2014. Modeling continuous integration practice differences in industry software development. Journal of Systems and Software 87 (2014), 48–59. DOI: 10.1016/j.jss.2013.08.032
Christopher Thompson. 2017. Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering (2017). 83–92.
Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 805–816. DOI: 10.1145/2786805.2786850
David Gray Widder, Michael Hilton, Christian Kästner, and Bogdan Vasilescu. 2019. A Conceptual Replication of Continuous Integration Pain Points in the Context of Travis CI. Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/3338906.3338922
João Helis Bernardo, Daniel Alencar da Costa, and Uirá Kulesza. 2018. Studying the Impact of Adopting Continuous Integration on the Delivery Time of Pull Requests. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New York, NY, USA, 131–141. DOI: 10.1145/3196398.3196421
Nathan Cassee, Bogdan Vasilescu, and Alexander Serebrenik. 2020. The Silent Helper: The Impact of Continuous Integration on Code Reviews. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 423–434. DOI: 10.1109/SANER48275.2020.9054818
Paul Duvall, Stephen M. Matyas, and AndrewGlover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series). Addison-Wesley Professional.
Omar Elazhary, Colin Werner, Ze Shi Li, Derek Lowlind, Neil A Ernst, and Margaret-Anne Storey. 2021. Uncovering the benefits and challenges of continuous integration practices. IEEE Transactions on Software Engineering 48, 7 (2021), 2570–2583.
Wagner Felidré, Leonardo Furtado, Daniel A. da Costa, Bruno Cartaxo, and Gustavo Pinto. 2019. Continuous Integration Theater. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–10. DOI: 10.1109/ESEM.2019.8870152
Martin Fowler. 2006. Continuous Integration. Retrieved november 24, 2021 from [link]
Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology 106 (2019), 101–121.
Vahid Garousi, Michael Felderer, Mika V Mäntylä, and Austen Rainer. 2020. Benefitting from the grey literature in software engineering research. In Contemporary Empirical Methods in Software Engineering. Springer, 385–413.
Taher Ahmed Ghaleb, Daniel Alencar Da Costa, and Ying Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering 24 (2019), 2102–2139.
Mehdi Golzadeh, Alexandre Decan, and Tom Mens. 2022. On the rise and fall of CI services in GitHub. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 662–672. DOI: 10.1109/SANER53432.2022.00084
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-Offs in Continuous Integration: Assurance, Security, and Flexibility. Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/3106237.3106270
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). 426–437.
Eero Laukkanen, Maria Paasivaara, and Teemu Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration – A Case Study. In 2015 Agile Conference. 11–20. DOI: 10.1109/Agile.2015.15
Jadson Santos. 2024. A Deep Dive into Continuous Integration Monitoring Practices. Ph.D. Dissertation. PPGSC/UFRN.
Eliezio Soares, Gustavo Sizílio, Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2021. The Effects of Continuous Integration on Software Development: a Systematic Literature Review. CoRR abs/2103.05451 (2021). arXiv:2103.05451 [link]
Daniel Ståhl and Jan Bosch. 2013. Experienced benefits of continuous integration in industry software product development: A case study. In The 12th iasted international conference on software engineering,(innsbruck, austria, 2013). 736–743.
Daniel Ståhl and Jan Bosch. 2013. Experienced benefits of continuous integration in industry software product development: A case study. In The 12th iasted international conference on software engineering,(innsbruck, austria, 2013). 736–743.
Daniel Ståhl and Jan Bosch. 2014. Modeling continuous integration practice differences in industry software development. Journal of Systems and Software 87 (2014), 48–59. DOI: 10.1016/j.jss.2013.08.032
Christopher Thompson. 2017. Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering (2017). 83–92.
Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 805–816. DOI: 10.1145/2786805.2786850
David Gray Widder, Michael Hilton, Christian Kästner, and Bogdan Vasilescu. 2019. A Conceptual Replication of Continuous Integration Pain Points in the Context of Travis CI. Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/3338906.3338922
Publicado
30/09/2024
Como Citar
SANTOS, Jadson; COSTA, Daniel Alencar da; KULESZA, Uirá.
Holter: Monitoring Continuous Integration Practices. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SOFTWARE (SBES), 38. , 2024, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 766-772.
DOI: https://doi.org/10.5753/sbes.2024.3604.
