Bruno Pedraça Souza
ABSTRACT
Context: Modern Information Systems require the use of contemporary software systems such as Cyber-Physical Systems, Embedded Systems, and Smart Cities-based Systems, eventually built under the paradigm of the Internet of Things. These Contemporary Software Systems (CSS) add new challenges for their construction, maintainability, and evolution, including the involvement of many actors with the software project and the necessary management of dependencies among hardware/things, software systems, and people. Problem: These technological challenges jeopardize the final quality of modern information systems due to the lack of adequate mechanisms supporting the engineering of CSS. Solution: Continuous Experimentation (CE) deserves some investigation regarding its suitability to mitigate and reduce engineering CSS risks. IS Theory: This research is under the General Systems Theory and is consistent with the Systems Information challenges regarding building smart cities-based systems. Method: To undertake a Structured Literature Review (StLR) supported with snowballing to reveal CE's empirical studies. Results: The StLR identified seven primary studies on CE adoption to support CSS building. Many studies are in the domain of embedded systems and CPS. Besides, the findings allowed us to conjecture a set of challenges and opportunities regarding using CE in CSS engineering. Conclusion: There are emergent technologies to support CE's execution in the context of web-based systems. However, several challenges and gaps surround CE's use for engineering CSS. Furthermore, the lack of software technologies, blueprints, or concrete guidance to promote CE in these software systems can motivate further investigations into its use in engineering the important parts of modern information systems.
- Rebeca C. Motta, Káthia M. de Oliveira and Guilherme H. Travassos, 2018. On challenges in engineering IoT software systems. Proceedings of the XXXII Brazilian Symposium on Software Engineering.Google Scholar
- Bruno P. de Souza, Rebeca C. Motta and Guilherme H. Travassos, 2019. Towards the description and representation of smartness in IoT scenarios specification. Proceedings of the XXXIII Brazilian Symposium on Software Engineering.Google Scholar
- Santiago Matalonga, Domenico Amalfitano, Andrea Doreste, Anna R. Fasolino and Guilherme H. Travassos, 2022. Alternatives for testing context-aware software systems in non-academic settings: results from a Rapid Review. Information and Software Technology.Google Scholar
- Bruno P. de Souza, Rebeca C. Motta and Guilherme H. Travassos, 2019. An IoT-based scenario description inspection technique. Proceedings of the XXXIII Brazilian Symposium on Software Quality.Google Scholar
- Elisa Y. Nakagawa, Pablo O. Antonino, Frank Schnicke Thomas Kuhn and Peter Liggesmeyer, 2022. Continuous Systems and Software Engineering for Industry 4.0: A disruptive view. Information and Software Technology.Google Scholar
- Helena H. Olsson and Jan Bosch, 2019. Data-driven development: Challenges in online, embedded, and on-premise software. International Conference on Product-Focused Software Process Improvement. Springer, Cham, 2019.Google Scholar
- Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne, 2009. Controlled experiments on the web: survey and practical guide." Data mining and knowledge discovery, pp. 140-181.Google Scholar
- Fabian Fagerholm, Alejandro S. Guinea, Hanna Mäenpää and Jürgen Münch, 2014. Building blocks for continuous experimentation. RCoSE 2014: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering, pp. 26 – 35.Google Scholar
- Fabian Fagerholm, Alejandro S. Guinea, Hanna Mäenpääa and Jürgen Münch, 2017. The RIGHT model for Continuous Experimentation. Journal of Systems and Software, pp. 292-305.Google ScholarCross Ref
- Helena H. Olsson, Hiva Alahyari and Jan Bosch, 2012. Climbing the Stairway to Heaven"–A Multiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software. 38th Euromicro conference on software engineering and advanced applications, pp. 392-399. DOI: 10.1109/SEAA.2012.54Google ScholarDigital Library
- Tanja Sauvola, Lucy E. Lwakatare, Teemu Karvonen, Pasi Kuvaja, Helena H. Olsson, Jan Bosch and Markku Oivo, 2015. Towards Customer-Centric Software Development: A Multiple-Case Study. 41st Euromicro Conference on Software Engineering and Advanced Applications, pp. 9-17. DOI: 10.1109/SEAA.2015.63Google ScholarDigital Library
- Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell and Anders Wesslén, 2012. Experimentation in Software Engineering. Springer Science & Business Media.Google Scholar
- Rebeca C. Motta, Valeria Silva and Guilherme H. Travassos, 2019. Towards a more in-depth understanding of the IoT Paradigm and its challenges." Journal of Software Engineering Research and Development. DOI:10.5753/jserd.2019.14Google ScholarCross Ref
- Jorge Biolchini, Paula G. Mian, Ana C. C. Natali and Guilherme H. Travassos, 2005. Systematic Review in Software Engineering. System Engineering and Computer Science Department COPPE/UFRJ, Technical Report.Google Scholar
- Kai Petersen, Robert Feldt, Shahid Mujtaba and Michael Mattsson, 2008. Systematic mapping studies in software engineering. 12th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 1-10.Google ScholarCross Ref
- Florian Auer and Michael Felderer, 2018. Current state of research on continuous experimentation: a systematic mapping study. 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 335-344. IEEE, DOI: 10.1109/SEAA.2018.00062.Google ScholarCross Ref
- Federico Giaimo, Hugo Andrade, and Christian Berger, 2020. Continuous experimentation and the cyber–physical systems challenge An overview of the literature and the industrial perspective. Journal of Systems and Software 170: 110781.Google ScholarCross Ref
- Gerald Schermann, Jürgen Cito and Philipp Leitner, 2018. Continuous experimentation: challenges, implementation techniques, and current research. IEEE Software 35.2: 26-31. DOI: 10.1109/MS.2018.111094748Google ScholarCross Ref
- Kohavi, Ron, Diane Tang, and Ya Xu, 2020. Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.Google Scholar
- Chris Parnin, Eric Helms, Chris Atlee, Harley Boughton, Mark Ghattas, Andy Glover, James Holman, John Micco, Brendan Murphy, Tony Savor, Michael Stumm, Shari Whitaker, and Laurie Williams, 2017. The top 10 adages in continuous deployment. IEEE Software, pp. 86-95. DOI: 10.1109/MS.2017.86Google ScholarDigital Library
- Carlos A. Gómez-Uribe and Neil Hunt, 2016. The Netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), pp.1-19.Google Scholar
- Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann, 2013. Online controlled experiments at large scale. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1168-1176.Google ScholarDigital Library
- Rodrigo O. Spínola and Guilherme H. Travassos, 2012. Towards a framework to characterize ubiquitous software projects. Information and Software Technology 54.7, pp. 759-785.Google ScholarDigital Library
- Lihui Wang, Martin Törngren, and Mauro Onori, 2015. Current status and advancement of cyber-physical systems in manufacturing. Journal of Manufacturing Systems 37 (2015), pp. 517-527.Google ScholarCross Ref
- Tarana Singh, Arun Solanki, Sanjay K. Sharma, Anand Nayyar, and Anand Paul, 2022. A Decade Review on Smart Cities: Paradigms, Challenges, and Opportunities. IEEE Access, pp. 68319 – 68364. DOI: 10.1109/ACCESS.2022.3184710Google ScholarCross Ref
- Santiago Matalonga, Guilherme H. Travassos, 2017. Testing context-aware software systems: Unchain the context, set it free! Proceedings of the XXXI Brazilian Symposium on Software Engineering, pp. 250–254.Google Scholar
- Jan Bosch and Ulrik Eklund, 2012. Eternal embedded software: Towards innovation experiment systems. International Symposium On Leveraging Applications of Formal Methods, Verification, and Validation. Springer, Berlin, Heidelberg, pp. 19-31.Google ScholarDigital Library
- Federico Giaimo, Hang Yi, Christian Berger and Ivica Crnkovic, 2016. Continuous Experimentation on Cyber-Physical Systems: Challenges and Opportunities. Proceedings of the Scientific Workshop Proceedings of XP2016.Google ScholarDigital Library
- Sezin G.Yamana, Myriam Munezero, Jürgen Müncha, Fabian Fagerholm, Ossi Syd, Mika Aaltola, Christina Palmu and Tomi Männistö, 2017. Introducing continuous experimentation in large software-intensive product and service organizations. Journal of Systems and Software 133, pp. 195-211.Google ScholarCross Ref
- Rasmus Ros and Per Runeson, 2018. Continuous experimentation and A/B testing: a mapping study. IEEE/ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE). IEEE, 2018.Google ScholarDigital Library
- Claes Wohlin, 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: 18th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–10.Google ScholarDigital Library
- Victor R. Basili and Dieter H. Rombach, 1994. The goal question metric approach. Encyclopedia of software engineering, pp. 528-532.Google Scholar
- Colin Werner, Ze S. Li, Derek Lowlind, Omar Elazhary, Neil Ernst, and Daniela Damian, 2021. Continuously Managing NFRs: Opportunities and Challenges in Practice, pp. 2629 – 2642. DOI: 10.1109/TSE.2021.3066330Google ScholarCross Ref
- Xiong, Z., Sheng, H., Rong, W., & Cooper, D. E. (2012). Intelligent transportation systems for smart cities: a progress review. Science China Information Sciences, 55(12), 2908-2914.Google Scholar
- Vladimir Erthal, Bruno P. de Souza, Paulo S. M. Santos and Guilherme Horta Travassos, 2022. A Literature Study to Characterize Continuous Experimentation in Software Engineering. Proceedings of XXV Congresso Ibero-Americano em Engenharia de Software (CIbSE), pp. 1-15.Google Scholar
- Kuhrmann, M., Fernández, D.M. & Daneva, M. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir Software Eng 22, 2852–2891 (2017).Google Scholar
- S1 - Philip Masek, Magnus Thulin, Hugo Andrade, Christian Berger and Ola Benderius, 2016. Systematic evaluation of sandboxed software deployment for real-time software on the example of a self-driving heavy vehicle, In IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp. 2398-2403. DOI: 10.1109/ITSC.2016.7795942Google ScholarDigital Library
- S2 - Federico Giaimo and Christian Berger, 2017. Design criteria to architect continuous experimentation for self-driving vehicles. In: IEEE International Conference on Software Architecture (ICSA). IEEE, pp. 203-210. DOI: 10.1109/ICSA.2017.36Google ScholarCross Ref
- S3 - David I. Mattos, Jan Bosch and Helena H. Olsson 2017. Your system gets better every day you use it: towards automated continuous experimentation. 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 256 – 265. IEEE. DOI: 10.1109/SEAA.2017.15Google ScholarCross Ref
- S4 - Federico Giaimo, Christian Berger and Crispin Kirchner, 2017. Considerations about continuous experimentation for resource-constrained platforms in self-driving vehicles. In European Conference on Software Architecture, pp. 84-91. DOI: 10.1007/978-3-319-65831-5_6Google ScholarCross Ref
- S5 - Helena H. Olsson, 2018. Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives. Agile Processes in Software Engineering and Extreme Programming, pp. 277–292.Google Scholar
- S6 – Federico Giaimo, Hugo Andrade and Christian Berger, 2019. The automotive take on continuous experimentation: a multiple case study. 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 123 - 130. IEEE. DOI: 10.1109/SEAA.2019.00028Google ScholarCross Ref
- S7 - David I. Mattos, Jan Bosch, Helena H. Olsson, Aita M. Korshani and Jonn Lantz. Automotive A/B testing: Challenges and lessons learned from practice. 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 101-109. IEEE. DOI: 10.1109/SEAA51224.2020.00026Google ScholarCross Ref
Index Terms
- On Challenges and Opportunities of Using Continuous Experimentation in the Engineering of Contemporary Software Systems
Recommendations
Experimental Software Engineering: A New Conference
An interesting issue facing software engineering relates to the evidence for adopting new techniques, tools, languages methodologies, and so on. We shouldn't always reject new models based on pure argument and logic, but ideally, we should subject such ...
Ten years with evidence-based software engineering. What is it? Has it had any impact? What's next?
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software EngineeringAn evidence-based software engineer is one who is able to: 1) Formulate a question, related to a decision or judgment, so that it can be answered by the use of evidence, 2) Collect, critically evaluate and summarise relevant evidence from research, ...
Engaging the net generation with evidence-based software engineering through a community-driven web database
Software engineering faculty face the challenge of educating future researchers and industry practitioners regarding the generation of empirical software engineering studies and their use in evidence-based software engineering. In order to engage the ...
Comments