Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing
Resumo
Research artifact sharing is known to strengthen the transparency of scientific studies. However, in the lack of common discipline-specific guidelines for artifacts evaluation, subjective and conflicting expectations may happen and threaten artifact quality. In this paper, we discuss our preliminary ideas for a framework based on quality management principles (5W2H) that can aid in the establishment of common guidelines for artifact evaluation and sharing. Also, using the Analytic Hierarchy Process, we discuss how research communities could join efforts to aid the guidelines’ adequacy to research priorities. These combined methodologies constitute a novelty for software engineering research which can foster research software sustainability.Referências
ACM (2020). Artifact Review and Badging - Current.
Basciani, F., Di Rocco, J., Di Ruscio, D., Iovino, L., and Pierantonio, A. (2015). Model repositories: Will they become reality? In CloudMDE@ MoDELS, pages 37–42.
Bose, A. (2020). Using genetic algorithm to improve consistency and retain authenticity in the analytic hierarchy process. OPSEARCH, 57(4):1070–1092.
Colin, E. C. (2000). Pesquisa Operacional. 170 Aplicações em Estratégia, Finanças, Logística, Produção, Marketing e Vendas. Atlas.
Damasceno, C. D. N. and Strüber, D. (2021). Quality guidelines for research artifacts in model-driven engineering. In MoDELS’21: ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems,Virtual Event, Japan, 10-15 October, 2021. ACM. http://arxiv.org/abs/2108.04652.
Hermann, B., Winter, S., and Siegmund, J. (2020). Community expectations for research artifacts and evaluation processes. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM.
Hui, W., Lui, S. M., and Lau, W. K. (2019). A reporting guideline for IS survey research. 126:113136.
LCRDM (2021). Research software sustainability in the Netherlands: Current practices and recommendations. Technical report, [Zenodo] DOI:10.5281/zenodo.4543569.
Leal, J. E. (2020). AHP-express: A simplified version of the analytical hierarchy process method. MethodsX, 7:100748.
Lindauer, M. and Hutter, F. (2020). Best practices for scientific research on neural architecture search. Journal of Machine Learning Research, 21(243):1–18.
Marjan Grootveld, Leenarts, E., Jones, S., Hermans, E., and Fankhauser, E. (2018). OpenAIRE and FAIR Data Expert Group survey about Horizon 2020 template for Data Management Plans.
Pitangueira, A. M., Maciel, R. S. P., de Oliveira Barros, M., and Andrade, A. S. (2013). A systematic review of software requirements selection and prioritization using sbse approaches. In Ruhe, G. and Zhang, Y., editors, Search Based Software Engineering, pages 188–208, Berlin, Heidelberg. Springer Berlin Heidelberg.
PMI (2017). A guide to the project management body of knowledge. PMBOK guide. Project Management Institute (PMI), Newtown Square, PA, 6th edition.
Ralph, P., Baltes, S., Bianculli, D., Dittrich, Y., Felderer, M., Feldt, R., Filieri, A., Furia, C. A., Graziotin, D., He, P., Hoda, R., Juristo, N., Kitchenham, B., Robbes, R., Mendez, D., Molleri, J., Spinellis, D., Staron, M., Stol, K., Tamburri, D., Torchiano, M., Treude, C., Turhan, B., and Vegas, S. (2020). ACM SIGSOFT Empirical Standards. arXiv:2010.03525 [cs]. arXiv: 2010.03525.
Saaty, T. L. and Vargas, L. G. (2012). Models, Methods, Concepts & Applications of the Analytic Hierarchy Process, volume 175 of International Series in Operations Research & Management Science. Springer US.
Tague, N. R. (2005). The quality toolbox. ASQ Quality Press, 2nd edition.
Timperley, C. S., Herckis, L., Le Goues, C., and Hilton, M. (2021). Understanding and improving artifact sharing in software engineering research. Empirical Software Engineering, 26(4):67.
Yoo, S., Harman, M., Tonella, P., and Susi, A. (2009). Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In Proceedings of the 18th International Symposium on Software Testing and Analysis, ISSTA ’09. ACM.
Basciani, F., Di Rocco, J., Di Ruscio, D., Iovino, L., and Pierantonio, A. (2015). Model repositories: Will they become reality? In CloudMDE@ MoDELS, pages 37–42.
Bose, A. (2020). Using genetic algorithm to improve consistency and retain authenticity in the analytic hierarchy process. OPSEARCH, 57(4):1070–1092.
Colin, E. C. (2000). Pesquisa Operacional. 170 Aplicações em Estratégia, Finanças, Logística, Produção, Marketing e Vendas. Atlas.
Damasceno, C. D. N. and Strüber, D. (2021). Quality guidelines for research artifacts in model-driven engineering. In MoDELS’21: ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems,Virtual Event, Japan, 10-15 October, 2021. ACM. http://arxiv.org/abs/2108.04652.
Hermann, B., Winter, S., and Siegmund, J. (2020). Community expectations for research artifacts and evaluation processes. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM.
Hui, W., Lui, S. M., and Lau, W. K. (2019). A reporting guideline for IS survey research. 126:113136.
LCRDM (2021). Research software sustainability in the Netherlands: Current practices and recommendations. Technical report, [Zenodo] DOI:10.5281/zenodo.4543569.
Leal, J. E. (2020). AHP-express: A simplified version of the analytical hierarchy process method. MethodsX, 7:100748.
Lindauer, M. and Hutter, F. (2020). Best practices for scientific research on neural architecture search. Journal of Machine Learning Research, 21(243):1–18.
Marjan Grootveld, Leenarts, E., Jones, S., Hermans, E., and Fankhauser, E. (2018). OpenAIRE and FAIR Data Expert Group survey about Horizon 2020 template for Data Management Plans.
Pitangueira, A. M., Maciel, R. S. P., de Oliveira Barros, M., and Andrade, A. S. (2013). A systematic review of software requirements selection and prioritization using sbse approaches. In Ruhe, G. and Zhang, Y., editors, Search Based Software Engineering, pages 188–208, Berlin, Heidelberg. Springer Berlin Heidelberg.
PMI (2017). A guide to the project management body of knowledge. PMBOK guide. Project Management Institute (PMI), Newtown Square, PA, 6th edition.
Ralph, P., Baltes, S., Bianculli, D., Dittrich, Y., Felderer, M., Feldt, R., Filieri, A., Furia, C. A., Graziotin, D., He, P., Hoda, R., Juristo, N., Kitchenham, B., Robbes, R., Mendez, D., Molleri, J., Spinellis, D., Staron, M., Stol, K., Tamburri, D., Torchiano, M., Treude, C., Turhan, B., and Vegas, S. (2020). ACM SIGSOFT Empirical Standards. arXiv:2010.03525 [cs]. arXiv: 2010.03525.
Saaty, T. L. and Vargas, L. G. (2012). Models, Methods, Concepts & Applications of the Analytic Hierarchy Process, volume 175 of International Series in Operations Research & Management Science. Springer US.
Tague, N. R. (2005). The quality toolbox. ASQ Quality Press, 2nd edition.
Timperley, C. S., Herckis, L., Le Goues, C., and Hilton, M. (2021). Understanding and improving artifact sharing in software engineering research. Empirical Software Engineering, 26(4):67.
Yoo, S., Harman, M., Tonella, P., and Susi, A. (2009). Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In Proceedings of the 18th International Symposium on Software Testing and Analysis, ISSTA ’09. ACM.
Publicado
28/09/2021
Como Citar
DAMASCENO, Carlos Diego Nascimento; MELO, Isotilia Costa; STRÜBER, Daniel.
Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing. In: WORKSHOP DE PRÁTICAS DE CIÊNCIA ABERTA PARA ENGENHARIA DE SOFTWARE (OPENSCIENSE), 1. , 2021, Joinville.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 1-6.
DOI: https://doi.org/10.5753/opensciense.2021.17137.
