On Challenges and Opportunities of Using Continuous Experimentation in the Engineering of Contemporary Software Systems
Resumo
Context: Modern Information Systems require the use of contemporary software systems such as Cyber-Physical Systems, Embedded Systems, and Smart Cities-based Systems, eventually built under the paradigm of the Internet of Things. These Contemporary Software Systems (CSS) add new challenges for their construction, maintainability, and evolution, including the involvement of many actors with the software project and the necessary management of dependencies among hardware/things, software systems, and people. Problem: These technological challenges jeopardize the final quality of modern information systems due to the lack of adequate mechanisms supporting the engineering of CSS. Solution: Continuous Experimentation (CE) deserves some investigation regarding its suitability to mitigate and reduce engineering CSS risks. IS Theory: This research is under the General Systems Theory and is consistent with the Systems Information challenges regarding building smart cities-based systems. Method: To undertake a Structured Literature Review (StLR) supported with snowballing to reveal CE's empirical studies. Results: The StLR identified seven primary studies on CE adoption to support CSS building. Many studies are in the domain of embedded systems and CPS. Besides, the findings allowed us to conjecture a set of challenges and opportunities regarding using CE in CSS engineering. Conclusion: There are emergent technologies to support CE's execution in the context of web-based systems. However, several challenges and gaps surround CE's use for engineering CSS. Furthermore, the lack of software technologies, blueprints, or concrete guidance to promote CE in these software systems can motivate further investigations into its use in engineering the important parts of modern information systems.
Palavras-chave:
Continuous Experimentation, Industry 4.0, Literature Review, Evidence-Based Software Engineering
Referências
Rebeca C. Motta, Káthia M. de Oliveira and Guilherme H. Travassos, 2018. On challenges in engineering IoT software systems. Proceedings of the XXXII Brazilian Symposium on Software Engineering.
Bruno P. de Souza, Rebeca C. Motta and Guilherme H. Travassos, 2019. Towards the description and representation of smartness in IoT scenarios specification. Proceedings of the XXXIII Brazilian Symposium on Software Engineering.
Santiago Matalonga, Domenico Amalfitano, Andrea Doreste, Anna R. Fasolino and Guilherme H. Travassos, 2022. Alternatives for testing context-aware software systems in non-academic settings: results from a Rapid Review. Information and Software Technology.
Bruno P. de Souza, Rebeca C. Motta and Guilherme H. Travassos, 2019. An IoT-based scenario description inspection technique. Proceedings of the XXXIII Brazilian Symposium on Software Quality.
Elisa Y. Nakagawa, Pablo O. Antonino, Frank Schnicke Thomas Kuhn and Peter Liggesmeyer, 2022. Continuous Systems and Software Engineering for Industry 4.0: A disruptive view. Information and Software Technology.
Helena H. Olsson and Jan Bosch, 2019. Data-driven development: Challenges in online, embedded, and on-premise software. International Conference on Product-Focused Software Process Improvement. Springer, Cham, 2019.
Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne, 2009. Controlled experiments on the web: survey and practical guide." Data mining and knowledge discovery, pp. 140-181.
Fabian Fagerholm, Alejandro S. Guinea, Hanna Mäenpää and Jürgen Münch, 2014. Building blocks for continuous experimentation. RCoSE 2014: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering, pp. 26 – 35.
Fabian Fagerholm, Alejandro S. Guinea, Hanna Mäenpääa and Jürgen Münch, 2017. The RIGHT model for Continuous Experimentation. Journal of Systems and Software, pp. 292-305.
Helena H. Olsson, Hiva Alahyari and Jan Bosch, 2012. Climbing the Stairway to Heaven"–A Multiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software. 38th Euromicro conference on software engineering and advanced applications, pp. 392-399. DOI: 10.1109/SEAA.2012.54
Tanja Sauvola, Lucy E. Lwakatare, Teemu Karvonen, Pasi Kuvaja, Helena H. Olsson, Jan Bosch and Markku Oivo, 2015. Towards Customer-Centric Software Development: A Multiple-Case Study. 41st Euromicro Conference on Software Engineering and Advanced Applications, pp. 9-17. DOI: 10.1109/SEAA.2015.63
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell and Anders Wesslén, 2012. Experimentation in Software Engineering. Springer Science & Business Media.
Rebeca C. Motta, Valeria Silva and Guilherme H. Travassos, 2019. Towards a more in-depth understanding of the IoT Paradigm and its challenges." Journal of Software Engineering Research and Development. DOI:10.5753/jserd.2019.14
Jorge Biolchini, Paula G. Mian, Ana C. C. Natali and Guilherme H. Travassos, 2005. Systematic Review in Software Engineering. System Engineering and Computer Science Department COPPE/UFRJ, Technical Report.
Kai Petersen, Robert Feldt, Shahid Mujtaba and Michael Mattsson, 2008. Systematic mapping studies in software engineering. 12th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 1-10.
Florian Auer and Michael Felderer, 2018. Current state of research on continuous experimentation: a systematic mapping study. 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 335-344. IEEE, DOI: 10.1109/SEAA.2018.00062.
Federico Giaimo, Hugo Andrade, and Christian Berger, 2020. Continuous experimentation and the cyber–physical systems challenge An overview of the literature and the industrial perspective. Journal of Systems and Software 170: 110781.
Gerald Schermann, Jürgen Cito and Philipp Leitner, 2018. Continuous experimentation: challenges, implementation techniques, and current research. IEEE Software 35.2: 26-31. DOI: 10.1109/MS.2018.111094748
Kohavi, Ron, Diane Tang, and Ya Xu, 2020. Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.
Chris Parnin, Eric Helms, Chris Atlee, Harley Boughton, Mark Ghattas, Andy Glover, James Holman, John Micco, Brendan Murphy, Tony Savor, Michael Stumm, Shari Whitaker, and Laurie Williams, 2017. The top 10 adages in continuous deployment. IEEE Software, pp. 86-95. DOI: 10.1109/MS.2017.86
Carlos A. Gómez-Uribe and Neil Hunt, 2016. The Netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), pp.1-19.
Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann, 2013. Online controlled experiments at large scale. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1168-1176.
Rodrigo O. Spínola and Guilherme H. Travassos, 2012. Towards a framework to characterize ubiquitous software projects. Information and Software Technology 54.7, pp. 759-785.
Lihui Wang, Martin Törngren, and Mauro Onori, 2015. Current status and advancement of cyber-physical systems in manufacturing. Journal of Manufacturing Systems 37 (2015), pp. 517-527.
Tarana Singh, Arun Solanki, Sanjay K. Sharma, Anand Nayyar, and Anand Paul, 2022. A Decade Review on Smart Cities: Paradigms, Challenges, and Opportunities. IEEE Access, pp. 68319 – 68364. DOI: 10.1109/ACCESS.2022.3184710
Santiago Matalonga, Guilherme H. Travassos, 2017. Testing context-aware software systems: Unchain the context, set it free! Proceedings of the XXXI Brazilian Symposium on Software Engineering, pp. 250–254.
Jan Bosch and Ulrik Eklund, 2012. Eternal embedded software: Towards innovation experiment systems. International Symposium On Leveraging Applications of Formal Methods, Verification, and Validation. Springer, Berlin, Heidelberg, pp. 19-31.
Federico Giaimo, Hang Yi, Christian Berger and Ivica Crnkovic, 2016. Continuous Experimentation on Cyber-Physical Systems: Challenges and Opportunities. Proceedings of the Scientific Workshop Proceedings of XP2016.
Sezin G.Yamana, Myriam Munezero, Jürgen Müncha, Fabian Fagerholm, Ossi Syd, Mika Aaltola, Christina Palmu and Tomi Männistö, 2017. Introducing continuous experimentation in large software-intensive product and service organizations. Journal of Systems and Software 133, pp. 195-211.
Rasmus Ros and Per Runeson, 2018. Continuous experimentation and A/B testing: a mapping study. IEEE/ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE). IEEE, 2018.
Claes Wohlin, 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: 18th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–10.
Victor R. Basili and Dieter H. Rombach, 1994. The goal question metric approach. Encyclopedia of software engineering, pp. 528-532.
Colin Werner, Ze S. Li, Derek Lowlind, Omar Elazhary, Neil Ernst, and Daniela Damian, 2021. Continuously Managing NFRs: Opportunities and Challenges in Practice, pp. 2629 – 2642. DOI: 10.1109/TSE.2021.3066330
Xiong, Z., Sheng, H., Rong, W., & Cooper, D. E. (2012). Intelligent transportation systems for smart cities: a progress review. Science China Information Sciences, 55(12), 2908-2914.
Vladimir Erthal, Bruno P. de Souza, Paulo S. M. Santos and Guilherme Horta Travassos, 2022. A Literature Study to Characterize Continuous Experimentation in Software Engineering. Proceedings of XXV Congresso Ibero-Americano em Engenharia de Software (CIbSE), pp. 1-15.
Kuhrmann, M., Fernández, D.M. & Daneva, M. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir Software Eng 22, 2852–2891 (2017).
S1 - Philip Masek, Magnus Thulin, Hugo Andrade, Christian Berger and Ola Benderius, 2016. Systematic evaluation of sandboxed software deployment for real-time software on the example of a self-driving heavy vehicle, In IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp. 2398-2403. DOI: 10.1109/ITSC.2016.7795942
S2 - Federico Giaimo and Christian Berger, 2017. Design criteria to architect continuous experimentation for self-driving vehicles. In: IEEE International Conference on Software Architecture (ICSA). IEEE, pp. 203-210. DOI: 10.1109/ICSA.2017.36
S3 - David I. Mattos, Jan Bosch and Helena H. Olsson 2017. Your system gets better every day you use it: towards automated continuous experimentation. 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 256 – 265. IEEE. DOI: 10.1109/SEAA.2017.15
S4 - Federico Giaimo, Christian Berger and Crispin Kirchner, 2017. Considerations about continuous experimentation for resource-constrained platforms in self-driving vehicles. In European Conference on Software Architecture, pp. 84-91. DOI: 10.1007/978-3-319-65831-5_6
S5 - Helena H. Olsson, 2018. Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives. Agile Processes in Software Engineering and Extreme Programming, pp. 277–292.
S6 – Federico Giaimo, Hugo Andrade and Christian Berger, 2019. The automotive take on continuous experimentation: a multiple case study. 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 123 - 130. IEEE. DOI: 10.1109/SEAA.2019.00028
S7 - David I. Mattos, Jan Bosch, Helena H. Olsson, Aita M. Korshani and Jonn Lantz. Automotive A/B testing: Challenges and lessons learned from practice. 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 101-109. IEEE. DOI: 10.1109/SEAA51224.2020.00026
Bruno P. de Souza, Rebeca C. Motta and Guilherme H. Travassos, 2019. Towards the description and representation of smartness in IoT scenarios specification. Proceedings of the XXXIII Brazilian Symposium on Software Engineering.
Santiago Matalonga, Domenico Amalfitano, Andrea Doreste, Anna R. Fasolino and Guilherme H. Travassos, 2022. Alternatives for testing context-aware software systems in non-academic settings: results from a Rapid Review. Information and Software Technology.
Bruno P. de Souza, Rebeca C. Motta and Guilherme H. Travassos, 2019. An IoT-based scenario description inspection technique. Proceedings of the XXXIII Brazilian Symposium on Software Quality.
Elisa Y. Nakagawa, Pablo O. Antonino, Frank Schnicke Thomas Kuhn and Peter Liggesmeyer, 2022. Continuous Systems and Software Engineering for Industry 4.0: A disruptive view. Information and Software Technology.
Helena H. Olsson and Jan Bosch, 2019. Data-driven development: Challenges in online, embedded, and on-premise software. International Conference on Product-Focused Software Process Improvement. Springer, Cham, 2019.
Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne, 2009. Controlled experiments on the web: survey and practical guide." Data mining and knowledge discovery, pp. 140-181.
Fabian Fagerholm, Alejandro S. Guinea, Hanna Mäenpää and Jürgen Münch, 2014. Building blocks for continuous experimentation. RCoSE 2014: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering, pp. 26 – 35.
Fabian Fagerholm, Alejandro S. Guinea, Hanna Mäenpääa and Jürgen Münch, 2017. The RIGHT model for Continuous Experimentation. Journal of Systems and Software, pp. 292-305.
Helena H. Olsson, Hiva Alahyari and Jan Bosch, 2012. Climbing the Stairway to Heaven"–A Multiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software. 38th Euromicro conference on software engineering and advanced applications, pp. 392-399. DOI: 10.1109/SEAA.2012.54
Tanja Sauvola, Lucy E. Lwakatare, Teemu Karvonen, Pasi Kuvaja, Helena H. Olsson, Jan Bosch and Markku Oivo, 2015. Towards Customer-Centric Software Development: A Multiple-Case Study. 41st Euromicro Conference on Software Engineering and Advanced Applications, pp. 9-17. DOI: 10.1109/SEAA.2015.63
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell and Anders Wesslén, 2012. Experimentation in Software Engineering. Springer Science & Business Media.
Rebeca C. Motta, Valeria Silva and Guilherme H. Travassos, 2019. Towards a more in-depth understanding of the IoT Paradigm and its challenges." Journal of Software Engineering Research and Development. DOI:10.5753/jserd.2019.14
Jorge Biolchini, Paula G. Mian, Ana C. C. Natali and Guilherme H. Travassos, 2005. Systematic Review in Software Engineering. System Engineering and Computer Science Department COPPE/UFRJ, Technical Report.
Kai Petersen, Robert Feldt, Shahid Mujtaba and Michael Mattsson, 2008. Systematic mapping studies in software engineering. 12th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 1-10.
Florian Auer and Michael Felderer, 2018. Current state of research on continuous experimentation: a systematic mapping study. 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 335-344. IEEE, DOI: 10.1109/SEAA.2018.00062.
Federico Giaimo, Hugo Andrade, and Christian Berger, 2020. Continuous experimentation and the cyber–physical systems challenge An overview of the literature and the industrial perspective. Journal of Systems and Software 170: 110781.
Gerald Schermann, Jürgen Cito and Philipp Leitner, 2018. Continuous experimentation: challenges, implementation techniques, and current research. IEEE Software 35.2: 26-31. DOI: 10.1109/MS.2018.111094748
Kohavi, Ron, Diane Tang, and Ya Xu, 2020. Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.
Chris Parnin, Eric Helms, Chris Atlee, Harley Boughton, Mark Ghattas, Andy Glover, James Holman, John Micco, Brendan Murphy, Tony Savor, Michael Stumm, Shari Whitaker, and Laurie Williams, 2017. The top 10 adages in continuous deployment. IEEE Software, pp. 86-95. DOI: 10.1109/MS.2017.86
Carlos A. Gómez-Uribe and Neil Hunt, 2016. The Netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), pp.1-19.
Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann, 2013. Online controlled experiments at large scale. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1168-1176.
Rodrigo O. Spínola and Guilherme H. Travassos, 2012. Towards a framework to characterize ubiquitous software projects. Information and Software Technology 54.7, pp. 759-785.
Lihui Wang, Martin Törngren, and Mauro Onori, 2015. Current status and advancement of cyber-physical systems in manufacturing. Journal of Manufacturing Systems 37 (2015), pp. 517-527.
Tarana Singh, Arun Solanki, Sanjay K. Sharma, Anand Nayyar, and Anand Paul, 2022. A Decade Review on Smart Cities: Paradigms, Challenges, and Opportunities. IEEE Access, pp. 68319 – 68364. DOI: 10.1109/ACCESS.2022.3184710
Santiago Matalonga, Guilherme H. Travassos, 2017. Testing context-aware software systems: Unchain the context, set it free! Proceedings of the XXXI Brazilian Symposium on Software Engineering, pp. 250–254.
Jan Bosch and Ulrik Eklund, 2012. Eternal embedded software: Towards innovation experiment systems. International Symposium On Leveraging Applications of Formal Methods, Verification, and Validation. Springer, Berlin, Heidelberg, pp. 19-31.
Federico Giaimo, Hang Yi, Christian Berger and Ivica Crnkovic, 2016. Continuous Experimentation on Cyber-Physical Systems: Challenges and Opportunities. Proceedings of the Scientific Workshop Proceedings of XP2016.
Sezin G.Yamana, Myriam Munezero, Jürgen Müncha, Fabian Fagerholm, Ossi Syd, Mika Aaltola, Christina Palmu and Tomi Männistö, 2017. Introducing continuous experimentation in large software-intensive product and service organizations. Journal of Systems and Software 133, pp. 195-211.
Rasmus Ros and Per Runeson, 2018. Continuous experimentation and A/B testing: a mapping study. IEEE/ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE). IEEE, 2018.
Claes Wohlin, 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: 18th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–10.
Victor R. Basili and Dieter H. Rombach, 1994. The goal question metric approach. Encyclopedia of software engineering, pp. 528-532.
Colin Werner, Ze S. Li, Derek Lowlind, Omar Elazhary, Neil Ernst, and Daniela Damian, 2021. Continuously Managing NFRs: Opportunities and Challenges in Practice, pp. 2629 – 2642. DOI: 10.1109/TSE.2021.3066330
Xiong, Z., Sheng, H., Rong, W., & Cooper, D. E. (2012). Intelligent transportation systems for smart cities: a progress review. Science China Information Sciences, 55(12), 2908-2914.
Vladimir Erthal, Bruno P. de Souza, Paulo S. M. Santos and Guilherme Horta Travassos, 2022. A Literature Study to Characterize Continuous Experimentation in Software Engineering. Proceedings of XXV Congresso Ibero-Americano em Engenharia de Software (CIbSE), pp. 1-15.
Kuhrmann, M., Fernández, D.M. & Daneva, M. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir Software Eng 22, 2852–2891 (2017).
S1 - Philip Masek, Magnus Thulin, Hugo Andrade, Christian Berger and Ola Benderius, 2016. Systematic evaluation of sandboxed software deployment for real-time software on the example of a self-driving heavy vehicle, In IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp. 2398-2403. DOI: 10.1109/ITSC.2016.7795942
S2 - Federico Giaimo and Christian Berger, 2017. Design criteria to architect continuous experimentation for self-driving vehicles. In: IEEE International Conference on Software Architecture (ICSA). IEEE, pp. 203-210. DOI: 10.1109/ICSA.2017.36
S3 - David I. Mattos, Jan Bosch and Helena H. Olsson 2017. Your system gets better every day you use it: towards automated continuous experimentation. 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 256 – 265. IEEE. DOI: 10.1109/SEAA.2017.15
S4 - Federico Giaimo, Christian Berger and Crispin Kirchner, 2017. Considerations about continuous experimentation for resource-constrained platforms in self-driving vehicles. In European Conference on Software Architecture, pp. 84-91. DOI: 10.1007/978-3-319-65831-5_6
S5 - Helena H. Olsson, 2018. Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives. Agile Processes in Software Engineering and Extreme Programming, pp. 277–292.
S6 – Federico Giaimo, Hugo Andrade and Christian Berger, 2019. The automotive take on continuous experimentation: a multiple case study. 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 123 - 130. IEEE. DOI: 10.1109/SEAA.2019.00028
S7 - David I. Mattos, Jan Bosch, Helena H. Olsson, Aita M. Korshani and Jonn Lantz. Automotive A/B testing: Challenges and lessons learned from practice. 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 101-109. IEEE. DOI: 10.1109/SEAA51224.2020.00026
Publicado
29/05/2023
Como Citar
SOUZA, Bruno Pedraça; SANTOS, Paulo Sérgio Medeiros; TRAVASSOS, Guilherme Horta.
On Challenges and Opportunities of Using Continuous Experimentation in the Engineering of Contemporary Software Systems. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 19. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
