Unveiling the Relationship Between Continuous Integration and Code Review: A Study with 10 Closed-source Projects
Resumo
Companies have adopted Continuous Integration (CI) and Code Review (CR) as key practices to monitor and improve software quality continuously. These practices enable early detection and correction of issues and have shown promising results in open and closed-source projects. However, there is limited understanding of the relationship between CI and CR in closed-source environments. For instance, it remains unclear how CI practices influence CR (and vice versa) and to what extent bad CI practices hinder the CR process. To address this gap, we conducted a mixed-methods study involving ten closed-source projects developed by industry partners.We collected and analyzed 32 metrics related to CI and CR. In addition to the quantitative analysis, we gathered perceptions from project teams to (i) validate the identified correlations, (ii) explore the effects of bad CI practices on CR, and (iii) understand the perceived benefits and challenges of using CI and CR together. Our results revealed 23 correlations between CI and CR processes, leading to the following key findings: (i) theworkload and execution time of CI can influence code review time; (ii) CI and CR execution times are more correlated when the processes occur sequentially; (iii) bad CI practices (such as long-lived branches, poor testing strategies, or complex build schemes) can negatively impact CR, causing delays, reviewer overload, and undetected issues; (iv) CI adds value to CR by anticipating problems, reducing manual tasks, and supporting the distribution of updates; and (v) the joint use of CI and CR presents challenges, such as ensuring code quality, resolving merge conflicts, and aligning processes. These findings shed light on the interplay between CI and CR and offer insights to improve their combined use in closed-source software development.
Palavras-chave:
continuous integration, code review, closed-source projects
Referências
Cláuvin Almeida, Marcos Kalinowski, Anderson Uchôa, and Bruno Feijó. 2023. Negative effects of gamification in education software. Information and Software Technology 156 (2023), 107142. DOI: 10.1016/j.infsof.2022.107142
A. Bacchelli and C. Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE). 712–721. DOI: 10.1109/ICSE.2013.6606617
João Helis Bernardo, Daniel Alencar da Costa, Uirá Kulesza, and Christoph Treude. 2023. The impact of a continuous integration service on the delivery time of merged pull requests. Empirical Software Engineering 28, 4 (2023), 97.
Ekaba Bisong. 2019. Introduction to Scikit-learn. Apress, Berkeley, CA, 215–229. DOI: 10.1007/978-1-4842-4470-8_18
Nathan Cassee, Bogdan Vasilescu, and Alexander Serebrenik. 2020. The Silent Helper. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 423–434. DOI: 10.1109/SANER48275.2020.9054818
D. Chen, K. T. Stolee, and T. Menzies. 2019. Replication Can Improve Prior Results. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). 179–190. DOI: 10.1109/ICPC.2019.00037
Adam Debbiche, Mikael Dienér, and Richard Berntsson Svensson. 2014. Challenges When Adopting Continuous Integration. In Product-Focused Software Process Improvement, Andreas Jedlitschka, Pasi Kuvaja, Marco Kuhrmann, Tomi Männistö, Jürgen Münch, and Mikko Raatikainen (Eds.). Springer International Publishing, Cham, 17–32.
Wade C Driscoll. 1996. Robustness of the ANOVA and Tukey-Kramer statistical tests. Computers & Industrial Engineering 31, 1-2 (1996), 265–268.
PM Duvall. 2018. Continuous Delivery Patterns and AntiPatterns in the Software LifeCycle. [link]
Paul M Duvall. 2010. Continuous Integration. DZone, Incorporated.
Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.
Omar Elazhary, Colin Werner, Ze Shi Li, Derek Lowlind, Neil A. Ernst, and Margaret-Anne Storey. 2021. Uncovering the Benefits and Challenges of Continuous Integration Practices. IEEE Transactions on Software Engineering (2021). DOI: 10.1109/TSE.2021.3064953
M. E. Fagan. 1976. Design and code inspections to reduce errors in program development. IBM Systems Journal 15, 3 (1976), 182–211. DOI: 10.1147/sj.153.0182
Nargis Fatima, Suriayati Chuprat, and Sumaira Nazir. 2018. Challenges and Benefits of Modern Code Review-Systematic Literature Review Protocol. In 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE). 1–5. DOI: 10.1109/ICSCEE.2018.8538393
Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering. Journal of Systems and Software 123 (2017), 176–189. DOI: 10.1016/j.jss.2015.06.063
Martin Fowler. 2006. Continuous integration. [link]
Taher Ahmed Ghaleb, Daniel Alencar Da Costa, and Ying Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering (2019).
Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An Exploratory Study of the Pull-Based Software Development Model. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 345–355. DOI: 10.1145/2568225.2568260
Georgios Gousios and Andy Zaidman. 2014. A Dataset for Pull-Based Development Research. In Proceedings of the 11th Working Conference on Mining Software Repositories (Hyderabad, India) (MSR 2014). Association for Computing Machinery, New York, NY, USA, 368–371. DOI: 10.1145/2597073.2597122
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-Offs in Continuous Integration. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 197–207. DOI: 10.1145/3106237.3106270
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE 2016). Association for Computing Machinery, New York, NY, USA, 426–437. DOI: 10.1145/2970276.2970358
Ciera Jaspan, Matthew Jorde, Andrea Knight, Caitlin Sadowski, Edward K. Smith, Collin Winter, and Emerson Murphy-Hill. 2018. Advantages and Disadvantages of a Monolithic Repository. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE-SEIP ’18). Association for Computing Machinery, New York, NY, USA, 225–234. DOI: 10.1145/3183519.3183550
Jing Jiang, Jiangfeng Lv, Jiateng Zheng, and Li Zhang. 2021. How Developers Modify Pull Requests in Code Review. IEEE Transactions on Reliability (2021), 1–15. DOI: 10.1109/TR.2021.3093159
Gunnar Kudrjavets and Ayushi Rastogi. 2024. Does code review speed matter for practitioners? Empirical Software Engineering 29, 1 (2024), 7.
Eero Laukkanen, Juha Itkonen, and Casper Lassenius. 2017. Problems, causes and solutions when adopting continuous delivery—A systematic literature review. Information and Software Technology 82 (2017), 55–79. DOI: 10.1016/j.infsof.2016.10.001
E. Laukkanen, M. Paasivaara, and T. Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration – A Case Study. In 2015 Agile Conference. 11–20. DOI: 10.1109/Agile.2015.15
Laura MacLeod, Michaela Greiler, Margaret-Anne Storey, Christian Bird, and Jacek Czerwonka. 2018. Code Reviewing in the Trenches. IEEE Software 35, 4 (2018), 34–42. DOI: 10.1109/MS.2017.265100500
Rungroj Maipradit, Dong Wang, Patanamon Thongtanunam, Raula Gaikovina Kula, Yasutaka Kamei, and Shane McIntosh. 2024. Repeated Builds During Code Review: An Empirical Study of the OpenStack Community. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (Echternach, Luxembourg) (ASE ’23). IEEE Press, 153–165. DOI: 10.1109/ASE56229.2023.00030
Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E Hassan. 2016. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering 21, 5 (2016), 2146–2189.
Mathias Meyer. 2014. Continuous Integration and Its Tools. IEEE Software 31, 3 (2014), 14–16. DOI: 10.1109/MS.2014.58
Ade Miller. 2008. A hundred days of continuous integration. In Agile 2008 conference. IEEE.
R. Morales, S. McIntosh, and F. Khomh. 2015. Do code review practices impact design quality? A case study of the Qt, VTK, and ITK projects. In Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). 171–180. DOI: 10.1109/SANER.2015.7081827
Helena Holmström Olsson, Hiva Alahyari, and Jan Bosch. 2012. Climbing the "Stairway to Heaven" – A Mulitiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software. In Proceedings of the 38th Euromicro Conference on Software Engineering and Advanced Applications. 392–399. DOI: 10.1109/SEAA.2012.54
Matheus Paixao, Jens Krinke, Donggyun Han, and Mark Harman. 2018. CROP. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New York, NY, USA, 46–49. DOI: 10.1145/3196398.3196466
Mohammad Masudur Rahman and Chanchal K. Roy. 2017. Impact of Continuous Integration on Code Reviews. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 499–502. DOI: 10.1109/MSR.2017.39
Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). Association for Computing Machinery, New York, NY, USA, 202–212. DOI: 10.1145/2491411.2491444
Nishrith Saini and Ricardo Britto. 2021. Using Machine Intelligence to Prioritise Code Review Requests. In Proceedings of the 43rd International Conference on Software Engineering. 11–20. DOI: 10.1109/ICSE-SEIP52600.2021.00010
Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2024. Holter: Monitoring Continuous Integration Practices. In Simpósio Brasileiro de Engenharia de Software (SBES). SBC, 775–781.
Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients. Anesthesia & Analgesia 126, 5 (2018), 1763–1768.
Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous integration, delivery and deployment. IEEE Access (2017).
August Shi, Peiyuan Zhao, and Darko Marinov. 2019. Understanding and Improving Regression Test Selection in Continuous Integration. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). 228–238. DOI: 10.1109/ISSRE.2019.00031
Dr. D. I. De Silva, W. A. C Pabasara, S. V Sangkavi, Wijerathne L.G.A.T.D, Wijesundara W.M.K.H, and Reezan S.A. 2023. The Effectiveness of Code Reviews on Improving Software Quality: An Empirical Study. 10 pages. DOI: 10.35940/ijrte.b7666.0712223
Ruben Blenicio Tavares Silva and Carla I. M. Bezerra. 2020. Analyzing Continuous Integration Bad Practices in Closed-Source Projects: An Initial Study. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering (Natal, Brazil) (SBES ’20). Association for Computing Machinery, New York, NY, USA, 642–647. DOI: 10.1145/3422392.3422474
Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 2016. Revisiting Code Ownership and Its Relationship with Software Quality in the Scope of Modern Code Review. In Proceedings of the 38th International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA, 1039–1050. DOI: 10.1145/2884781.2884852
Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Influence of Social and Technical Factors for Evaluating Contribution in GitHub. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 356–366. DOI: 10.1145/2568225.2568315
Alexia Tsotsis. 2011. Meet phabricator, the witty code review tool built inside facebook.
Anderson Uchôa, Caio Barbosa, Willian Oizumi, Publio Blenílio, Rafael Lima, Alessandro Garcia, and Carla Bezerra. 2020. How Does Modern Code Review Impact Software Design Degradation? An In-depth Empirical Study. In 36th ICSME.
M.Wessel, A. Serebrenik, I. Wiese, I. Steinmacher, and M. A. Gerosa. 2020. Effects of Adopting Code Review Bots on Pull Requests to OSS Projects. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 1–11. DOI: 10.1109/ICSME46990.2020.00011
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Yue Yu, Huaimin Wang, Vladimir Filkov, Premkumar Devanbu, and Bogdan Vasilescu. 2015. Wait for It. In 2015 IEEE/ACM 12thWorking Conference on Mining Software Repositories. 367–371. DOI: 10.1109/MSR.2015.42
F. Zampetti, G. Bavota, G. Canfora, and M. D. Penta. 2019. A Study on the Interplay between Pull Request Review and Continuous Integration Builds. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). 38–48. DOI: 10.1109/SANER.2019.8667996
Fiorella Zampetti, Simone Scalabrino, Rocco Oliveto, Gerardo Canfora, and Massimiliano Di Penta. 2017. How Open Source Projects Use Static Code Analysis Tools in Continuous Integration Pipelines. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 334–344. DOI: 10.1109/MSR.2017.2
Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering (2020).
Yang Zhang, Gang Yin, Yue Yu, and Huaimin Wang. 2014. Investigating Social Media in GitHub’s Pull-Requests. In Proceedings of the 1st International Workshop on Crowd-Based Software Development Methods and Technologies (Hong Kong, China) (CrowdSoft 2014). Association for Computing Machinery, New York, NY, USA, 37–41. DOI: 10.1145/2666539.2666572
A. Bacchelli and C. Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE). 712–721. DOI: 10.1109/ICSE.2013.6606617
João Helis Bernardo, Daniel Alencar da Costa, Uirá Kulesza, and Christoph Treude. 2023. The impact of a continuous integration service on the delivery time of merged pull requests. Empirical Software Engineering 28, 4 (2023), 97.
Ekaba Bisong. 2019. Introduction to Scikit-learn. Apress, Berkeley, CA, 215–229. DOI: 10.1007/978-1-4842-4470-8_18
Nathan Cassee, Bogdan Vasilescu, and Alexander Serebrenik. 2020. The Silent Helper. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 423–434. DOI: 10.1109/SANER48275.2020.9054818
D. Chen, K. T. Stolee, and T. Menzies. 2019. Replication Can Improve Prior Results. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). 179–190. DOI: 10.1109/ICPC.2019.00037
Adam Debbiche, Mikael Dienér, and Richard Berntsson Svensson. 2014. Challenges When Adopting Continuous Integration. In Product-Focused Software Process Improvement, Andreas Jedlitschka, Pasi Kuvaja, Marco Kuhrmann, Tomi Männistö, Jürgen Münch, and Mikko Raatikainen (Eds.). Springer International Publishing, Cham, 17–32.
Wade C Driscoll. 1996. Robustness of the ANOVA and Tukey-Kramer statistical tests. Computers & Industrial Engineering 31, 1-2 (1996), 265–268.
PM Duvall. 2018. Continuous Delivery Patterns and AntiPatterns in the Software LifeCycle. [link]
Paul M Duvall. 2010. Continuous Integration. DZone, Incorporated.
Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.
Omar Elazhary, Colin Werner, Ze Shi Li, Derek Lowlind, Neil A. Ernst, and Margaret-Anne Storey. 2021. Uncovering the Benefits and Challenges of Continuous Integration Practices. IEEE Transactions on Software Engineering (2021). DOI: 10.1109/TSE.2021.3064953
M. E. Fagan. 1976. Design and code inspections to reduce errors in program development. IBM Systems Journal 15, 3 (1976), 182–211. DOI: 10.1147/sj.153.0182
Nargis Fatima, Suriayati Chuprat, and Sumaira Nazir. 2018. Challenges and Benefits of Modern Code Review-Systematic Literature Review Protocol. In 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE). 1–5. DOI: 10.1109/ICSCEE.2018.8538393
Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering. Journal of Systems and Software 123 (2017), 176–189. DOI: 10.1016/j.jss.2015.06.063
Martin Fowler. 2006. Continuous integration. [link]
Taher Ahmed Ghaleb, Daniel Alencar Da Costa, and Ying Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering (2019).
Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An Exploratory Study of the Pull-Based Software Development Model. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 345–355. DOI: 10.1145/2568225.2568260
Georgios Gousios and Andy Zaidman. 2014. A Dataset for Pull-Based Development Research. In Proceedings of the 11th Working Conference on Mining Software Repositories (Hyderabad, India) (MSR 2014). Association for Computing Machinery, New York, NY, USA, 368–371. DOI: 10.1145/2597073.2597122
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-Offs in Continuous Integration. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 197–207. DOI: 10.1145/3106237.3106270
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE 2016). Association for Computing Machinery, New York, NY, USA, 426–437. DOI: 10.1145/2970276.2970358
Ciera Jaspan, Matthew Jorde, Andrea Knight, Caitlin Sadowski, Edward K. Smith, Collin Winter, and Emerson Murphy-Hill. 2018. Advantages and Disadvantages of a Monolithic Repository. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE-SEIP ’18). Association for Computing Machinery, New York, NY, USA, 225–234. DOI: 10.1145/3183519.3183550
Jing Jiang, Jiangfeng Lv, Jiateng Zheng, and Li Zhang. 2021. How Developers Modify Pull Requests in Code Review. IEEE Transactions on Reliability (2021), 1–15. DOI: 10.1109/TR.2021.3093159
Gunnar Kudrjavets and Ayushi Rastogi. 2024. Does code review speed matter for practitioners? Empirical Software Engineering 29, 1 (2024), 7.
Eero Laukkanen, Juha Itkonen, and Casper Lassenius. 2017. Problems, causes and solutions when adopting continuous delivery—A systematic literature review. Information and Software Technology 82 (2017), 55–79. DOI: 10.1016/j.infsof.2016.10.001
E. Laukkanen, M. Paasivaara, and T. Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration – A Case Study. In 2015 Agile Conference. 11–20. DOI: 10.1109/Agile.2015.15
Laura MacLeod, Michaela Greiler, Margaret-Anne Storey, Christian Bird, and Jacek Czerwonka. 2018. Code Reviewing in the Trenches. IEEE Software 35, 4 (2018), 34–42. DOI: 10.1109/MS.2017.265100500
Rungroj Maipradit, Dong Wang, Patanamon Thongtanunam, Raula Gaikovina Kula, Yasutaka Kamei, and Shane McIntosh. 2024. Repeated Builds During Code Review: An Empirical Study of the OpenStack Community. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (Echternach, Luxembourg) (ASE ’23). IEEE Press, 153–165. DOI: 10.1109/ASE56229.2023.00030
Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E Hassan. 2016. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering 21, 5 (2016), 2146–2189.
Mathias Meyer. 2014. Continuous Integration and Its Tools. IEEE Software 31, 3 (2014), 14–16. DOI: 10.1109/MS.2014.58
Ade Miller. 2008. A hundred days of continuous integration. In Agile 2008 conference. IEEE.
R. Morales, S. McIntosh, and F. Khomh. 2015. Do code review practices impact design quality? A case study of the Qt, VTK, and ITK projects. In Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). 171–180. DOI: 10.1109/SANER.2015.7081827
Helena Holmström Olsson, Hiva Alahyari, and Jan Bosch. 2012. Climbing the "Stairway to Heaven" – A Mulitiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software. In Proceedings of the 38th Euromicro Conference on Software Engineering and Advanced Applications. 392–399. DOI: 10.1109/SEAA.2012.54
Matheus Paixao, Jens Krinke, Donggyun Han, and Mark Harman. 2018. CROP. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New York, NY, USA, 46–49. DOI: 10.1145/3196398.3196466
Mohammad Masudur Rahman and Chanchal K. Roy. 2017. Impact of Continuous Integration on Code Reviews. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 499–502. DOI: 10.1109/MSR.2017.39
Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). Association for Computing Machinery, New York, NY, USA, 202–212. DOI: 10.1145/2491411.2491444
Nishrith Saini and Ricardo Britto. 2021. Using Machine Intelligence to Prioritise Code Review Requests. In Proceedings of the 43rd International Conference on Software Engineering. 11–20. DOI: 10.1109/ICSE-SEIP52600.2021.00010
Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2024. Holter: Monitoring Continuous Integration Practices. In Simpósio Brasileiro de Engenharia de Software (SBES). SBC, 775–781.
Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients. Anesthesia & Analgesia 126, 5 (2018), 1763–1768.
Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous integration, delivery and deployment. IEEE Access (2017).
August Shi, Peiyuan Zhao, and Darko Marinov. 2019. Understanding and Improving Regression Test Selection in Continuous Integration. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). 228–238. DOI: 10.1109/ISSRE.2019.00031
Dr. D. I. De Silva, W. A. C Pabasara, S. V Sangkavi, Wijerathne L.G.A.T.D, Wijesundara W.M.K.H, and Reezan S.A. 2023. The Effectiveness of Code Reviews on Improving Software Quality: An Empirical Study. 10 pages. DOI: 10.35940/ijrte.b7666.0712223
Ruben Blenicio Tavares Silva and Carla I. M. Bezerra. 2020. Analyzing Continuous Integration Bad Practices in Closed-Source Projects: An Initial Study. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering (Natal, Brazil) (SBES ’20). Association for Computing Machinery, New York, NY, USA, 642–647. DOI: 10.1145/3422392.3422474
Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 2016. Revisiting Code Ownership and Its Relationship with Software Quality in the Scope of Modern Code Review. In Proceedings of the 38th International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA, 1039–1050. DOI: 10.1145/2884781.2884852
Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Influence of Social and Technical Factors for Evaluating Contribution in GitHub. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 356–366. DOI: 10.1145/2568225.2568315
Alexia Tsotsis. 2011. Meet phabricator, the witty code review tool built inside facebook.
Anderson Uchôa, Caio Barbosa, Willian Oizumi, Publio Blenílio, Rafael Lima, Alessandro Garcia, and Carla Bezerra. 2020. How Does Modern Code Review Impact Software Design Degradation? An In-depth Empirical Study. In 36th ICSME.
M.Wessel, A. Serebrenik, I. Wiese, I. Steinmacher, and M. A. Gerosa. 2020. Effects of Adopting Code Review Bots on Pull Requests to OSS Projects. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 1–11. DOI: 10.1109/ICSME46990.2020.00011
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Yue Yu, Huaimin Wang, Vladimir Filkov, Premkumar Devanbu, and Bogdan Vasilescu. 2015. Wait for It. In 2015 IEEE/ACM 12thWorking Conference on Mining Software Repositories. 367–371. DOI: 10.1109/MSR.2015.42
F. Zampetti, G. Bavota, G. Canfora, and M. D. Penta. 2019. A Study on the Interplay between Pull Request Review and Continuous Integration Builds. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). 38–48. DOI: 10.1109/SANER.2019.8667996
Fiorella Zampetti, Simone Scalabrino, Rocco Oliveto, Gerardo Canfora, and Massimiliano Di Penta. 2017. How Open Source Projects Use Static Code Analysis Tools in Continuous Integration Pipelines. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 334–344. DOI: 10.1109/MSR.2017.2
Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering (2020).
Yang Zhang, Gang Yin, Yue Yu, and Huaimin Wang. 2014. Investigating Social Media in GitHub’s Pull-Requests. In Proceedings of the 1st International Workshop on Crowd-Based Software Development Methods and Technologies (Hong Kong, China) (CrowdSoft 2014). Association for Computing Machinery, New York, NY, USA, 37–41. DOI: 10.1145/2666539.2666572
Publicado
22/09/2025
Como Citar
SILVA, Ruben; SILVA, Publio; BEZERRA, Carla; UCHÔA, Anderson; GARCIA, Alessandro.
Unveiling the Relationship Between Continuous Integration and Code Review: A Study with 10 Closed-source Projects. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SOFTWARE (SBES), 39. , 2025, Recife/PE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 393-404.
ISSN 2833-0633.
DOI: https://doi.org/10.5753/sbes.2025.9965.
