Rethinking the Evaluation of Data Science and Data Management Research

• Dimas Cassimiro Nascimento UFAPE
• Daliton da Silva UFAPE

DOI: https://doi.org/10.5753/eniac.2026.30080

Resumo

The evaluation paradigm in Data Science and Data Management has become predominantly focused on methodological novelty and quantitative metrics, constraining the diversity of valuable contributions. This article presents a framework for recognizing and evaluating broader research outputs, including theoretical foundations, replication studies, reflective analyses, and visionary perspectives. We contribute a critical analysis of evaluation biases, a classification of underrecognized contribution types, a visual mapping of contribution relationships, a formal multi-criteria evaluation model, and practical implementation guidelines. This framework addresses critical gaps in current assessment practices and provides guidance for fostering more inclusive research ecosystems.

Referências

Abadi, D., Ailamaki, A., Andersen, D., Bailis, P., Balazinska, M., Bernstein, P., Boncz, P., Chaudhuri, S., Cheung, A., Doan, A., Dong, L., Franklin, M. J., Freire, J., Halevy, A., Hellerstein, J. M., Idreos, S., Kossmann, D., Kraska, T., Krishnamurthy, S., Markl, V., Melnik, S., Milo, T., Mohan, C., Neumann, T., Ooi, B. C., Ozcan, F., Patel, J., Pavlo, A., Popa, R., Ramakrishnan, R., Ré, C., Stonebraker, M., and Suciu, D. (2020). The seattle report on database research. ACM SIGMOD Record, 48(4):44–53.

Abadi, D. J., Agrawal, R., Ailamaki, A., Balazinska, M., Bernstein, P. A., Carey, M. J., Chaudhuri, S., Dean, J., Doan, A., Franklin, M. J., Gehrke, J., Haas, L. M., Halevy, A. Y., Hellerstein, J. M., Ioannidis, Y. E., Jagadish, H. V., Kossmann, D., Madden, S., Mehrotra, S., Milo, T., Naughton, J. F., Ramakrishnan, R., Markl, V., Olston, C., Ooi, B. C., Ré, C., Suciu, D., Stonebraker, M., Walter, T., and Widom, J. (2014). The beckman report on database research. ACM SIGMOD Record, 43(3):61–70.

Bausell, R. B. (2021). The Problem with Science: The Reproducibility Crisis and What to Do About It. Oxford University Press.

Castro, J. P. C. and Aguiar, C. D. (2023). Big data architectures for fair-compliant repositories: A systematic review. In Proceedings of the Brazilian Symposium on Databases (SBBD), pages 76–88. SBC.

Dhar, V. (2013). Data science and prediction. Communications of the ACM, 56(12):64–73.

Dwan, K., Gamble, C., Williamson, P. R., and Kirkham, J. J. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLOS ONE, 8(7):e66844.

Hernandez, J. A. and Colom, M. (2025). Reproducible research policies and software/data management in scientific computing journals: A survey, discussion, and perspectives. Frontiers in Computer Science, 6:1491823.

Ioannidis, J. P. A. (2014). How to make more published research true. PLOS Medicine, 11(10):e1001747.

Kapoor, S. and Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9):100804.

Larivière, V., Haustein, S., and Mongeon, P. (2015). The oligopoly of academic publishers in the digital era. PLOS ONE, 10(6):e0127502.

Lee, C. J., Sugimoto, C. R., Zhang, G., and Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2–17.

Lipton, Z. C. and Steinhardt, J. (2019). Troubling trends in machine learning scholarship. Queue, 17(1):45–77.

McLeish, T. and Strang, V. (2016). Evaluating interdisciplinary research: The elephant in the peer-reviewers’ room. Palgrave Communications, 2:16055.

Olszewski, D., Lu, A., Stillman, C., Warren, K., Kitroser, C., Pascual, A., Ukirde, D., Butler, K. R. B., and Traynor, P. (2023). “Get in Researchers; We’re Measuring Reproducibility”: A reproducibility study of machine learning papers in tier 1 security conferences. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 3433–3459.

Prechelt, L. (2021). On implicit assumptions underlying software engineering research. In Evaluation and Assessment in Software Engineering (EASE 2021), pages 336–339. ACM.

Provost, F. and Fawcett, T. (2013). Data science and its relationship to big data and data-driven decision making. Big Data, 1(1):51–59.

Wagstaff, K. L. (2012). Machine learning that matters. In Proceedings of the 29th International Conference on Machine Learning, pages 529–536.