FOSTER Taxonomy-based Open Science-Related Practices in Software Engineering: Review and Observations
Abstract
The Open Science (OS) movement, primarily led by UNESCO, advocates for making research and its outcomes accessible to everyone. This movement has explored various issues across different fields, including reproducibility, experimentation, design science methodologies, and education, to provide access to research resources and discoveries. While the adoption of OS principles and practices in Software Engineering (SE) is increasing, it is still in its early stages, prompting further investigation into the topic. This paper aims to identify OS concepts, practices, and challenges related to their implementation in SE. To achieve this, we performed a Scoping Review Study of the literature. The main findings highlighted several OS-related practices in SE, categorized into Open Data, Open Platforms, Open Review, Open Access, Artifact Sharing, and Open Source. Additionally, challenges were noted in applying these practices. These findings culminated in a series of observations intended to support the adoption of OS in SE.
Keywords:
Open Science, Software Engineering, Openness, Practices, Open Research, Artifacts, Observations, Scoping Review
References
Timo Aaltonen, Tommi Mikkonen, Heikki Peltola, and Arto Salminen. 2014. From mashup applications to open data ecosystems. In Proceedings of The International Symposium on Open Collaboration. ACM, [link], 1–8.
Apostolos Ampatzoglou, Stamatia Bibi, Paris Avgeriou, Marijn Verbeek, and Alexander Chatzigeorgiou. 2019. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology 106 (2019), 201–230.
Carlos E Anchundia et al. 2020. Resources for reproducibility of experiments in empirical software engineering: Topics derived from a secondary study. IEEE Access 8 (2020), 8992–9004.
Iker Azpeitia, Jon Iturrioz, and Oscar Díaz. 2020. Volunteering for Linked Data Wrapper maintenance: A platform perspective. Information Systems 89 (2020), 101468.
Deepika Badampudi, Claes Wohlin, and Kai Petersen. 2015. Experiences from using snowballing and database searches in systematic literature studies. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering. ACM, [link], 1–10.
Sebastian Baltes and Paul Ralph. 2022. Sampling in software engineering research: A critical review and guidelines. Empirical Software Engineering 27, 4 (2022), 94.
Victor R Basili, Marvin V Zelkowitz, Dag IK Sjøberg, Philip Johnson, and Anthony J Cowling. 2007. Protocols in the use of empirical software engineering artifacts. Empirical Software Engineering 12, 1 (2007), 107–119.
Kellyton S. Brito, Marcos Antônio da Silva Costa, Vinicius Cardoso Garcia, and Silvio Romero de Lemos Meira. 2014. Brazilian government open data: implementation, challenges, and potential opportunities. In Proceedings of the 15th annual international conference on digital government research. ACM, [link], 11–16.
Digital Curation Centre. 2023. Digital Curation Centre Data Lifecycle Model. [link].
Jürgen Cito and Harald C Gall. 2016. Using docker containers to improve reproducibility in software engineering research. In Proceedings of the 38th international conference on software engineering companion. ACM, [link], 906–907.
Gennaro Cordasco, Delfina Malandrino, Donato Pirozzi, Vittorio Scarano, and Carmine Spagnuolo. 2018. A layered architecture for open data: Design, implementation and experiences. In Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance. ACM, [link], 371–381.
Andreiwid Sheffer Correa, Pär-Ola Zander, and Flavio Soares Correa Da Silva. 2018. Investigating open data portals automatically: a methodology and some illustrations. In Proceedings of the 19th Annual International Conference on Digital Government Research: Governance in the Data Age. ACM, [link], 1–10.
Carlos Diego Nascimento Damasceno, Isotilia Costa Melo, and Daniel Struber. 2021. Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing. arXiv preprint arXiv:2109.02304 1 (2021), 1–5.
Jack B Dennis. 2017. Principles to Support Modular Software Construction. Journal of Computer Science and Technology 32 (2017), 3–10.
Oussama El Bacha, Othmane Jmad, Younes El Bouzekri El Idrissi, and Nabil Hmina. 2017. Exploiting Open Data to Improve the Business Intelligence & Business Discovery Experience. In Proceedings of the 2nd international Conference on Big Data, Cloud and Applications. ACM, [link], 1–6.
Emelie Engström, Margaret-Anne Storey, Per Runeson, Martin Höst, and Maria Teresa Baldassarre. 2020. How software engineering research aligns with design science: a review. Empirical Software Engineering 25 (2020), 2630–2660.
N. A. Ernst and M. T. Baldassarre. 2023. Registered reports in software engineering. Empirical Software Engineering 28 (2023), 1–11. DOI: 10.1007/s10664-022-10277-5
Neil A Ernst, Jeffrey C Carver, Daniel Mendez, and Marco Torchiano. 2021. Understanding peer review of software engineering papers. Empirical Software Engineering 26 (2021), 1–29.
European Organization For Nuclear Research and OpenAIRE. 2013. Zenodo. DOI: 10.25495/7GXK-RD71
Andrea Farnham, Christoph Kurz, Mehmet Ali Öztürk, Monica Solbiati, Oona Myllyntaus, Jordy Meekes, Tra My Pham, Clara Paz, Magda Langiewicz, Sophie Andrews, et al. 2017. Early career researchers want Open Science. Genome biology 18 (2017), 1–4.
Figshare. 2024. Figshare. [link]
Shayne Flint. 2009. A Conceptual Model of Software Engineering Research Approaches. In 2009 Australian Software Engineering Conference. IEEE, IEEE, [link], 229–236.
Foster. 2022. Foster Open Science Taxonomy. [link].
E. Frachtenberg. 2022. Research artifacts and citations in computer systems papers. PeerJ Comput Sci 7 (2022), e887. DOI: 10.7717/peerj-cs.887
Parisa Ghazi and Martin Glinz. 2017. Challenges of working with artifacts in requirements engineering and software engineering. Requirements Engineering 22, 3 (2017), 359–385.
Jesús M González-Barahona and Gregorio Robles. 2012. On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Software Engineering 17, 1 (2012), 75–89.
Daniel Graziotin, Xiaofeng Wang, and Pekka Abrahamsson. 2014. A framework for systematic analysis of open access journals and its application in software engineering and information systems. Scientometrics 101 (2014), 1627–1656.
Mark Grechanik, Collin McMillan, Tathagata Dasgupta, Denys Poshyvanyk, and Malcom Gethers. 2014. Redacting sensitive information in software artifacts. In Proceedings of the 22Nd International Conference on Program Comprehension. ACM, [link], 314–325.
Sarah Higgins. 2008. The DCC curation lifecycle model. International Journal of Digital Curation 1, 1 (2008), 1–7.
John Lalit Jagtiani and Neal Lewis. 2016. Enhancing Software Engineering Curricula By Incorporating Open, Data-Driven Planning Methods. In 2016 ASEE Annual Conference & Exposition. ASEE PEER, [link], 1–11.
Hanan Khalil and Andrea C. Tricco. 2022. Differentiating between mapping reviews and scoping reviews in the evidence synthesis ecosystem. Journal of Clinical Epidemiology 149 (2022), 175–182. DOI: 10.1016/j.jclinepi.2022.05.012
Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering.
Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2015. Evidencebased software engineering and systematic reviews. Vol. 4. CRC Press, CRC Press.
Shriram Krishnamurthi. 2013. Artifact evaluation for software conferences. ACM SIGSOFT Software Engineering Notes 38, 3 (2013), 7–10.
Adrian Kuhn. 2012. Ides need become open data platforms (as need languages and vms). In 2012 Second International Workshop on Developing Tools as PlugIns (TOPI). IEEE, IEEE, [link], 31–36.
Adriana Lopes, Tayana Conte, and Clarisse Sieckenius de Souza. 2020. Exploring the Directives of Communicability for Improving the Quality of Software Artifacts. In 19th Brazilian Symposium on Software Quality. ACM, [link], 1–10.
Adriana Lopes Damian, Clarisse Sieckenius de Souza, and Tayana Conte. 2021. Directives of Communicability for Software Artifacts. In XX Brazilian Symposium on Software Quality. ACM, [link], 1–9.
Paolo Manghi, Alessia Bardi, Claudio Atzori, Miriam Baglioni, Natalia Manola, Jochen Schirrwagen, Pedro Principe, Michele Artini, Amelie Becker, Michele De Bonis, et al. 2019. The OpenAIRE research graph data model. Zenodo 1, 1 (2019), 1–23.
Wolfgang Mauerer and Stefanie Scherzinger. 2022. 1-2-3 reproducibility for quantum software experiments. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, IEEE, [link], 1247–1248.
Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold. 2020. Open science in software engineering. In Contemporary empirical methods in software engineering. Springer, New York, 477–501.
Daniel Méndez Fernández, Wolfgang Böhm, Andreas Vogelsang, Jakob Mund, Manfred Broy, Marco Kuhrmann, and ThorstenWeyer. 2019. Artefacts in software engineering: a fundamental positioning. Software & Systems Modeling 18, 5 (2019), 2777–2786.
Daniel Méndez Fernández,Martin Monperrus, Robert Feldt, and Thomas Zimmermann. 2019. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24, 3 (2019), 1057–1060.
Gaia Mosconi, Qinyu Li, Dave Randall, Helena Karasti, Peter Tolmie, Jana Barutzky, Matthias Korn, and Volkmar Pipek. 2019. Three gaps in opening science. Computer Supported Cooperative Work (CSCW) 28, 3 (2019), 749–789.
Zachary Munn, Micah DJ Peters, Cindy Stern, Catalin Tufanaru, Alexa McArthur, and Edoardo Aromataris. 2018. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology 18, 18 (2018), 1–7. DOI: 10.1186/s12874-018-0611-x
Elisa Yumi Nakagawa, Kátia Romero Felizardo Scannavino, Sandra Camargo Pinto Ferraz Fabbri, and Fabiano Cutigi Ferrari. 2017. Revisão sistemática da literatura em engenharia de software: teoria e prática. Elsevier Brasil, [link].
Joaquín Nicolás and Ambrosio Toval. 2009. On the generation of requirements specifications from software engineering models: A systematic literature review. Information and Software Technology 51, 9 (2009), 1291–1307.
Apostolos Ampatzoglou, Stamatia Bibi, Paris Avgeriou, Marijn Verbeek, and Alexander Chatzigeorgiou. 2019. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology 106 (2019), 201–230.
Carlos E Anchundia et al. 2020. Resources for reproducibility of experiments in empirical software engineering: Topics derived from a secondary study. IEEE Access 8 (2020), 8992–9004.
Iker Azpeitia, Jon Iturrioz, and Oscar Díaz. 2020. Volunteering for Linked Data Wrapper maintenance: A platform perspective. Information Systems 89 (2020), 101468.
Deepika Badampudi, Claes Wohlin, and Kai Petersen. 2015. Experiences from using snowballing and database searches in systematic literature studies. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering. ACM, [link], 1–10.
Sebastian Baltes and Paul Ralph. 2022. Sampling in software engineering research: A critical review and guidelines. Empirical Software Engineering 27, 4 (2022), 94.
Victor R Basili, Marvin V Zelkowitz, Dag IK Sjøberg, Philip Johnson, and Anthony J Cowling. 2007. Protocols in the use of empirical software engineering artifacts. Empirical Software Engineering 12, 1 (2007), 107–119.
Kellyton S. Brito, Marcos Antônio da Silva Costa, Vinicius Cardoso Garcia, and Silvio Romero de Lemos Meira. 2014. Brazilian government open data: implementation, challenges, and potential opportunities. In Proceedings of the 15th annual international conference on digital government research. ACM, [link], 11–16.
Digital Curation Centre. 2023. Digital Curation Centre Data Lifecycle Model. [link].
Jürgen Cito and Harald C Gall. 2016. Using docker containers to improve reproducibility in software engineering research. In Proceedings of the 38th international conference on software engineering companion. ACM, [link], 906–907.
Gennaro Cordasco, Delfina Malandrino, Donato Pirozzi, Vittorio Scarano, and Carmine Spagnuolo. 2018. A layered architecture for open data: Design, implementation and experiences. In Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance. ACM, [link], 371–381.
Andreiwid Sheffer Correa, Pär-Ola Zander, and Flavio Soares Correa Da Silva. 2018. Investigating open data portals automatically: a methodology and some illustrations. In Proceedings of the 19th Annual International Conference on Digital Government Research: Governance in the Data Age. ACM, [link], 1–10.
Carlos Diego Nascimento Damasceno, Isotilia Costa Melo, and Daniel Struber. 2021. Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing. arXiv preprint arXiv:2109.02304 1 (2021), 1–5.
Jack B Dennis. 2017. Principles to Support Modular Software Construction. Journal of Computer Science and Technology 32 (2017), 3–10.
Oussama El Bacha, Othmane Jmad, Younes El Bouzekri El Idrissi, and Nabil Hmina. 2017. Exploiting Open Data to Improve the Business Intelligence & Business Discovery Experience. In Proceedings of the 2nd international Conference on Big Data, Cloud and Applications. ACM, [link], 1–6.
Emelie Engström, Margaret-Anne Storey, Per Runeson, Martin Höst, and Maria Teresa Baldassarre. 2020. How software engineering research aligns with design science: a review. Empirical Software Engineering 25 (2020), 2630–2660.
N. A. Ernst and M. T. Baldassarre. 2023. Registered reports in software engineering. Empirical Software Engineering 28 (2023), 1–11. DOI: 10.1007/s10664-022-10277-5
Neil A Ernst, Jeffrey C Carver, Daniel Mendez, and Marco Torchiano. 2021. Understanding peer review of software engineering papers. Empirical Software Engineering 26 (2021), 1–29.
European Organization For Nuclear Research and OpenAIRE. 2013. Zenodo. DOI: 10.25495/7GXK-RD71
Andrea Farnham, Christoph Kurz, Mehmet Ali Öztürk, Monica Solbiati, Oona Myllyntaus, Jordy Meekes, Tra My Pham, Clara Paz, Magda Langiewicz, Sophie Andrews, et al. 2017. Early career researchers want Open Science. Genome biology 18 (2017), 1–4.
Figshare. 2024. Figshare. [link]
Shayne Flint. 2009. A Conceptual Model of Software Engineering Research Approaches. In 2009 Australian Software Engineering Conference. IEEE, IEEE, [link], 229–236.
Foster. 2022. Foster Open Science Taxonomy. [link].
E. Frachtenberg. 2022. Research artifacts and citations in computer systems papers. PeerJ Comput Sci 7 (2022), e887. DOI: 10.7717/peerj-cs.887
Parisa Ghazi and Martin Glinz. 2017. Challenges of working with artifacts in requirements engineering and software engineering. Requirements Engineering 22, 3 (2017), 359–385.
Jesús M González-Barahona and Gregorio Robles. 2012. On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Software Engineering 17, 1 (2012), 75–89.
Daniel Graziotin, Xiaofeng Wang, and Pekka Abrahamsson. 2014. A framework for systematic analysis of open access journals and its application in software engineering and information systems. Scientometrics 101 (2014), 1627–1656.
Mark Grechanik, Collin McMillan, Tathagata Dasgupta, Denys Poshyvanyk, and Malcom Gethers. 2014. Redacting sensitive information in software artifacts. In Proceedings of the 22Nd International Conference on Program Comprehension. ACM, [link], 314–325.
Sarah Higgins. 2008. The DCC curation lifecycle model. International Journal of Digital Curation 1, 1 (2008), 1–7.
John Lalit Jagtiani and Neal Lewis. 2016. Enhancing Software Engineering Curricula By Incorporating Open, Data-Driven Planning Methods. In 2016 ASEE Annual Conference & Exposition. ASEE PEER, [link], 1–11.
Hanan Khalil and Andrea C. Tricco. 2022. Differentiating between mapping reviews and scoping reviews in the evidence synthesis ecosystem. Journal of Clinical Epidemiology 149 (2022), 175–182. DOI: 10.1016/j.jclinepi.2022.05.012
Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering.
Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2015. Evidencebased software engineering and systematic reviews. Vol. 4. CRC Press, CRC Press.
Shriram Krishnamurthi. 2013. Artifact evaluation for software conferences. ACM SIGSOFT Software Engineering Notes 38, 3 (2013), 7–10.
Adrian Kuhn. 2012. Ides need become open data platforms (as need languages and vms). In 2012 Second International Workshop on Developing Tools as PlugIns (TOPI). IEEE, IEEE, [link], 31–36.
Adriana Lopes, Tayana Conte, and Clarisse Sieckenius de Souza. 2020. Exploring the Directives of Communicability for Improving the Quality of Software Artifacts. In 19th Brazilian Symposium on Software Quality. ACM, [link], 1–10.
Adriana Lopes Damian, Clarisse Sieckenius de Souza, and Tayana Conte. 2021. Directives of Communicability for Software Artifacts. In XX Brazilian Symposium on Software Quality. ACM, [link], 1–9.
Paolo Manghi, Alessia Bardi, Claudio Atzori, Miriam Baglioni, Natalia Manola, Jochen Schirrwagen, Pedro Principe, Michele Artini, Amelie Becker, Michele De Bonis, et al. 2019. The OpenAIRE research graph data model. Zenodo 1, 1 (2019), 1–23.
Wolfgang Mauerer and Stefanie Scherzinger. 2022. 1-2-3 reproducibility for quantum software experiments. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, IEEE, [link], 1247–1248.
Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold. 2020. Open science in software engineering. In Contemporary empirical methods in software engineering. Springer, New York, 477–501.
Daniel Méndez Fernández, Wolfgang Böhm, Andreas Vogelsang, Jakob Mund, Manfred Broy, Marco Kuhrmann, and ThorstenWeyer. 2019. Artefacts in software engineering: a fundamental positioning. Software & Systems Modeling 18, 5 (2019), 2777–2786.
Daniel Méndez Fernández,Martin Monperrus, Robert Feldt, and Thomas Zimmermann. 2019. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24, 3 (2019), 1057–1060.
Gaia Mosconi, Qinyu Li, Dave Randall, Helena Karasti, Peter Tolmie, Jana Barutzky, Matthias Korn, and Volkmar Pipek. 2019. Three gaps in opening science. Computer Supported Cooperative Work (CSCW) 28, 3 (2019), 749–789.
Zachary Munn, Micah DJ Peters, Cindy Stern, Catalin Tufanaru, Alexa McArthur, and Edoardo Aromataris. 2018. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology 18, 18 (2018), 1–7. DOI: 10.1186/s12874-018-0611-x
Elisa Yumi Nakagawa, Kátia Romero Felizardo Scannavino, Sandra Camargo Pinto Ferraz Fabbri, and Fabiano Cutigi Ferrari. 2017. Revisão sistemática da literatura em engenharia de software: teoria e prática. Elsevier Brasil, [link].
Joaquín Nicolás and Ambrosio Toval. 2009. On the generation of requirements specifications from software engineering models: A systematic literature review. Information and Software Technology 51, 9 (2009), 1291–1307.
Published
2025-09-22
How to Cite
CORDEIRO, André F. R.; OLIVEIRAJR, Edson.
FOSTER Taxonomy-based Open Science-Related Practices in Software Engineering: Review and Observations. In: BRAZILIAN SYMPOSIUM ON SOFTWARE ENGINEERING (SBES), 39. , 2025, Recife/PE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 1-12.
ISSN 2833-0633.
DOI: https://doi.org/10.5753/sbes.2025.9563.
