Frameworks and Methods for Evaluating Open Data Platforms Under the Lens of Human-Data Interaction: A Systematic Review
Resumo
Introduction: This systematic review analyzes the literature on Human-Data Interaction (HDI) within the context of open data platforms. Objective: To investigate HDI definitions and challenges, evaluate approaches for assessing platform quality and complex data visualizations, and understand user needs, including those of non-experts. Methodology: A predefined protocol guided a search in digital databases, followed by a rigorous analysis of 25 selected studies. Results: The review reveals a lack of frameworks for evaluating open data platforms. It also highlights significant interaction challenges, such as difficulties in data interpretation and the need for better support for non-specialized users.
Palavras-chave:
Human-Data Interaction, Open Data Platforms, Evaluation
Referências
Barcellos, R., Bernardini, F. and Viterbo, J. (2022). “Towards defining data interpretability in open data portals: Challenges and research opportunities.” In Information Systems, v. 106, p. 101961.
Barcellos, R., Bernardini, F., Viterbo, J. and Zuiderwijk, A. (2023). “Hippolyta: a framework to enhance open data interpretability and empower citizens.” In Proceedings of the 24th Annual International Conference on Digital Government Research, pp.191–198, ACM.
Barcellos, R., Bernardini, F., Zuiderwijk, A. and Viterbo, J. (2024). “Exploring Interpretability in Open Government Data with ChatGPT.” In Proceedings of the 25th Annual International Conference on Digital Government Research, (pp. 186-195).
Barcellos, R., Viterbo, J., Bernardini, F. and Trevisan, D. (2018). “An Instrument for Evaluating the Quality of Data Visualizations”. In 22nd International Conference Information Visualisation. (IV) (pp. 169-174). IEEE.
Cantador, I., Viejo-Tardío, J., Cortés-Cediel, M. and Pedro, M. (2021). “A Chatbot for Searching and Exploring Open Data: Implementation and Evaluation in E-Government.” In Proceedings of the 22nd Annual International Conference on Digital Government Research. (pp. 168-179).
Coleti, T. A., Zacarias, R., Divino, S., Saraiva, J., Salgado, A., Gonçalves, D., Morandini, M., Santos, R. (2024) “GranDIHC-BR 2025-2035-GC5-Human-Data Interaction, Data Literacy and Usable Privacy” In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems. (pp. 1-24).
Degbelo, A. (2020). “Open Data User Needs: A Preliminary Synthesis.” In Proceedings of the Web Conference 2020, pp.834–839.
Elmqvist, N. (2011). “Embodied Human-Data Interaction” In ACM Workshop “Embodied Interaction: Theory and Practice in HCI , (Vol. 1, pp. 104-107).
Ferreira, B., Rodrigues, A., Barbosa, F., Barbosa, S. (2024). “Investigating Difficulties and Decisions in Information Visualization Design” In IHC '24: Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems (pp. 1-11).
Flausino, M., Kozievitch, N., Fonseca, K. and Liu, E. (2024). “Open Data Portals -A case study of Challenges and Opportunities.” In Brazilian e-Science Workshop (BreSci) (pp. 104-111). SBC.
Garcia, I., Andrade, E., Bernardini, F. and Viterbo, J. (2022). “Assessing the Quality of Covid-19 Open Data Portals.” In International Conference on Electronic Government, (pp. 212-227). Cham: Springer International Publishing.
Gil, A. C. (2008). Métodos e técnicas de pesquisa social, 6a edição. Editora Atlas.
Grant, M. J., & Booth, A. (2009). “A Typology of reviews: An Analysis of 14 Review Types and Associated Methodologies” In Health Information & Libraries Journal, 26(2), 91–108.
Hornung, H., Pereira, R., Baranauskas, M.C.C., Liu, K., 2015. “Challenges for Human-Data Interaction – A Semiotic Perspective” In 17th International Conference on Human-Computer Interaction. (pp. 37-48). Cham: Springer International Publishing.
Hsu, J., Ravichandran, R., Zhang, E. and Keung, C. (2021). “Open Data Standard and Analysis Framework: Towards Response Equity in Local Governments.” In 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pp.1–8.
Isotani, S., and Bittencourt, I. I. (2015). “Dados abertos conectados: em busca da web do conhecimento” Novatec Editora.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. 2.
Kitchin, R and McArdle, G. (2016). “What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets” In Big Data and Society. v. 3, n. 1, p. 2053951716631130.
Koesten, L., Simperl, E., Blount, T., Kacprzak, E. and Tennison, J. (2020). “Everything you always wanted to know about a dataset: Studies in data summarisation.” In
International Journal of Human-Computer Studies, v. 135, p. 102367.
Kotis, K., Vouros, G.A. (2006). “Human-centered ontology engineering: The HCOME methodology.” In Knowl Inf Syst 10, 109–131 .
Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012). “Benefits, Adoption Barriers and Myths of Open Data and Open Government.” In Information Systems Management, 29(4), 258–268.
Lee, B., Choe, E., Isenberg, P., Marriott, K., Stasko, J. and Rhyne, T.-M. (2020). “Reaching Broader Audiences With Data Visualization.” In IEEE Computer Graphics and Applications, 40(2), pp.82–90.
Liang, Y., Cao, Y., Chen, M., Dong, H. and Wang, H. (2025). “Determinants of open government data continuance usage and value creation: A self-regulation framework analysis.” In Government Information Quarterly, v. 42, n. 2, p. 102022.
Máchová, R. and Lnenicka, M. (2017). “Evaluating the Quality of Open Data Portals on the National Level.” In Journal of theoretical and applied electronic commerce research, v. 12, n. 1, p. 21-41.
Mortier, R., Haddadi, H., Henderson, T., McAuley, D., Crowcroft, J. (2014). “Human-Data Interaction: The Human Face of the Data-Driven Society” in SSRN Electronic Journal.
Natvig, M., Jiang, S., Stav E. (2021)”Using open data for digital innovation: Barriers for use and recommendations for publishers” In JeDEM-eJournal of eDemocracy and Open Government, 13(2), 28-57.
Oliveira, R., Cappelli, C. and Oliveira, J. (2023). “Guia de Linguagem Simples para Visualização de Dados: Construção, Design, Avaliação e Contribuições”. In Anais do II Workshop Investigações em Interação Humano-Dados, (pp. 76-82). SBC.
Pereira, R., Darin, T. and Silveira, M. (2024). “GranDIHC-BR: Grand Research Challenges in Human-Computer Interaction in Brazil for 2025-2035”. In IHC '24: Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems. (pp. 1-24).
Pinto, H.S., Tempich, C., Staab, S. (2009). “Ontology Engineering and Evolution in a Distributed World Using DILIGENT”. In: Staab, S., Studer, R. (eds) Handbook on
Ontologies. International Handbooks on Information Systems. (pp. 153-176). Berlin, Heidelberg: Springer Berlin Heidelberg.
Puussaar, A., Montague, K., Peacock, S., Nappey, T., Anderson, R., Jonczyk, J., Wright, P. and James, P. (2022). “SenseMyStreet: Sensor Commissioning Toolkit for Communities” In Proceedings of the ACM on Human-Computer Interaction, v. 6, n. CSCW2, p. 1-26.
Runeson, P., Skoglund, M. (2009). “Reference-based search strategies in systematic reviews” In Proceedings of the 13th International Conference on Empirical Assessment and Evaluation in Software Engineering. Electronic Workshops in Computing (eWIC). BCS, Durham University, UK
Sommerville, Ian (2011). Engenharia de Software. 9. ed. — São Paulo: Pearson Prentice Hall
Tylosky, N., Knutas, A., Qureshi, M. and Wolff, A. (2023). “User Experience for Non-Expert Audiences in Data Exploration.” In Proceedings of the 11th International Conference on Communities and Technologies. p. 12-16.
Tylosky, N., Knutas, A. and Wolff, A. (2023). “Principles for Designing for Data Exploration for a Non-Expert Audience.” In 46th ICT and Electronics Convention -Proceedings, pp.88–91.
Tylosky, N., Knutas, A. and Wolff, A. (2025). “Design practices in visualization driven data exploration for non-expert audiences”. In Computer Science Review. v. 56, p. 100731.
Vasconcelos, L., Barcellos, R., Viterbo, J., Bernardini, F. and Salgado, L. (2020). “Investigating Communicability Issues in the Open Data Manipulation Flow.” In AMCIS 2020 Proceedings.
Victorelli, E. and Reis, J.C. (2020). “Human-data interaction design guidelines for visualization systems.” In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems,pp.1–10.
Victorelli, E., dos Reis, J., Hornung, H., Prado, A. (2019). “Understanding Human-Data Interaction: Literature Review and Recommendations for Design” In International Journal of Human-Computer Studies. v. 134, p. 13-32.
Xiao, F., Chi, Y. and He, D. (2023). “Promoting data use through understanding user behaviors: A model for human open government data interaction.” In Journal of the Association for Information Science and Technology, v. 74, n. 13, p. 1498-1514
Xiao, F., Thaker, K. and He, D. (2022). “Categorizing Open Government Data Users by Exploring their Challenges and Proficiency.” In CHI Conference on Human Factors in Computing Systems, pp.1–7.
Xiao, F., Wang, Z. and He, D. (2020). “Understanding users’ accessing behaviors to local Open Government Data via transaction log analysis.” In Proceedings of the 83rd Annual Association for Information Science and Technology. v. 57, n. 1, p. e27
Barcellos, R., Bernardini, F., Viterbo, J. and Zuiderwijk, A. (2023). “Hippolyta: a framework to enhance open data interpretability and empower citizens.” In Proceedings of the 24th Annual International Conference on Digital Government Research, pp.191–198, ACM.
Barcellos, R., Bernardini, F., Zuiderwijk, A. and Viterbo, J. (2024). “Exploring Interpretability in Open Government Data with ChatGPT.” In Proceedings of the 25th Annual International Conference on Digital Government Research, (pp. 186-195).
Barcellos, R., Viterbo, J., Bernardini, F. and Trevisan, D. (2018). “An Instrument for Evaluating the Quality of Data Visualizations”. In 22nd International Conference Information Visualisation. (IV) (pp. 169-174). IEEE.
Cantador, I., Viejo-Tardío, J., Cortés-Cediel, M. and Pedro, M. (2021). “A Chatbot for Searching and Exploring Open Data: Implementation and Evaluation in E-Government.” In Proceedings of the 22nd Annual International Conference on Digital Government Research. (pp. 168-179).
Coleti, T. A., Zacarias, R., Divino, S., Saraiva, J., Salgado, A., Gonçalves, D., Morandini, M., Santos, R. (2024) “GranDIHC-BR 2025-2035-GC5-Human-Data Interaction, Data Literacy and Usable Privacy” In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems. (pp. 1-24).
Degbelo, A. (2020). “Open Data User Needs: A Preliminary Synthesis.” In Proceedings of the Web Conference 2020, pp.834–839.
Elmqvist, N. (2011). “Embodied Human-Data Interaction” In ACM Workshop “Embodied Interaction: Theory and Practice in HCI , (Vol. 1, pp. 104-107).
Ferreira, B., Rodrigues, A., Barbosa, F., Barbosa, S. (2024). “Investigating Difficulties and Decisions in Information Visualization Design” In IHC '24: Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems (pp. 1-11).
Flausino, M., Kozievitch, N., Fonseca, K. and Liu, E. (2024). “Open Data Portals -A case study of Challenges and Opportunities.” In Brazilian e-Science Workshop (BreSci) (pp. 104-111). SBC.
Garcia, I., Andrade, E., Bernardini, F. and Viterbo, J. (2022). “Assessing the Quality of Covid-19 Open Data Portals.” In International Conference on Electronic Government, (pp. 212-227). Cham: Springer International Publishing.
Gil, A. C. (2008). Métodos e técnicas de pesquisa social, 6a edição. Editora Atlas.
Grant, M. J., & Booth, A. (2009). “A Typology of reviews: An Analysis of 14 Review Types and Associated Methodologies” In Health Information & Libraries Journal, 26(2), 91–108.
Hornung, H., Pereira, R., Baranauskas, M.C.C., Liu, K., 2015. “Challenges for Human-Data Interaction – A Semiotic Perspective” In 17th International Conference on Human-Computer Interaction. (pp. 37-48). Cham: Springer International Publishing.
Hsu, J., Ravichandran, R., Zhang, E. and Keung, C. (2021). “Open Data Standard and Analysis Framework: Towards Response Equity in Local Governments.” In 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pp.1–8.
Isotani, S., and Bittencourt, I. I. (2015). “Dados abertos conectados: em busca da web do conhecimento” Novatec Editora.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. 2.
Kitchin, R and McArdle, G. (2016). “What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets” In Big Data and Society. v. 3, n. 1, p. 2053951716631130.
Koesten, L., Simperl, E., Blount, T., Kacprzak, E. and Tennison, J. (2020). “Everything you always wanted to know about a dataset: Studies in data summarisation.” In
International Journal of Human-Computer Studies, v. 135, p. 102367.
Kotis, K., Vouros, G.A. (2006). “Human-centered ontology engineering: The HCOME methodology.” In Knowl Inf Syst 10, 109–131 .
Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012). “Benefits, Adoption Barriers and Myths of Open Data and Open Government.” In Information Systems Management, 29(4), 258–268.
Lee, B., Choe, E., Isenberg, P., Marriott, K., Stasko, J. and Rhyne, T.-M. (2020). “Reaching Broader Audiences With Data Visualization.” In IEEE Computer Graphics and Applications, 40(2), pp.82–90.
Liang, Y., Cao, Y., Chen, M., Dong, H. and Wang, H. (2025). “Determinants of open government data continuance usage and value creation: A self-regulation framework analysis.” In Government Information Quarterly, v. 42, n. 2, p. 102022.
Máchová, R. and Lnenicka, M. (2017). “Evaluating the Quality of Open Data Portals on the National Level.” In Journal of theoretical and applied electronic commerce research, v. 12, n. 1, p. 21-41.
Mortier, R., Haddadi, H., Henderson, T., McAuley, D., Crowcroft, J. (2014). “Human-Data Interaction: The Human Face of the Data-Driven Society” in SSRN Electronic Journal.
Natvig, M., Jiang, S., Stav E. (2021)”Using open data for digital innovation: Barriers for use and recommendations for publishers” In JeDEM-eJournal of eDemocracy and Open Government, 13(2), 28-57.
Oliveira, R., Cappelli, C. and Oliveira, J. (2023). “Guia de Linguagem Simples para Visualização de Dados: Construção, Design, Avaliação e Contribuições”. In Anais do II Workshop Investigações em Interação Humano-Dados, (pp. 76-82). SBC.
Pereira, R., Darin, T. and Silveira, M. (2024). “GranDIHC-BR: Grand Research Challenges in Human-Computer Interaction in Brazil for 2025-2035”. In IHC '24: Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems. (pp. 1-24).
Pinto, H.S., Tempich, C., Staab, S. (2009). “Ontology Engineering and Evolution in a Distributed World Using DILIGENT”. In: Staab, S., Studer, R. (eds) Handbook on
Ontologies. International Handbooks on Information Systems. (pp. 153-176). Berlin, Heidelberg: Springer Berlin Heidelberg.
Puussaar, A., Montague, K., Peacock, S., Nappey, T., Anderson, R., Jonczyk, J., Wright, P. and James, P. (2022). “SenseMyStreet: Sensor Commissioning Toolkit for Communities” In Proceedings of the ACM on Human-Computer Interaction, v. 6, n. CSCW2, p. 1-26.
Runeson, P., Skoglund, M. (2009). “Reference-based search strategies in systematic reviews” In Proceedings of the 13th International Conference on Empirical Assessment and Evaluation in Software Engineering. Electronic Workshops in Computing (eWIC). BCS, Durham University, UK
Sommerville, Ian (2011). Engenharia de Software. 9. ed. — São Paulo: Pearson Prentice Hall
Tylosky, N., Knutas, A., Qureshi, M. and Wolff, A. (2023). “User Experience for Non-Expert Audiences in Data Exploration.” In Proceedings of the 11th International Conference on Communities and Technologies. p. 12-16.
Tylosky, N., Knutas, A. and Wolff, A. (2023). “Principles for Designing for Data Exploration for a Non-Expert Audience.” In 46th ICT and Electronics Convention -Proceedings, pp.88–91.
Tylosky, N., Knutas, A. and Wolff, A. (2025). “Design practices in visualization driven data exploration for non-expert audiences”. In Computer Science Review. v. 56, p. 100731.
Vasconcelos, L., Barcellos, R., Viterbo, J., Bernardini, F. and Salgado, L. (2020). “Investigating Communicability Issues in the Open Data Manipulation Flow.” In AMCIS 2020 Proceedings.
Victorelli, E. and Reis, J.C. (2020). “Human-data interaction design guidelines for visualization systems.” In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems,pp.1–10.
Victorelli, E., dos Reis, J., Hornung, H., Prado, A. (2019). “Understanding Human-Data Interaction: Literature Review and Recommendations for Design” In International Journal of Human-Computer Studies. v. 134, p. 13-32.
Xiao, F., Chi, Y. and He, D. (2023). “Promoting data use through understanding user behaviors: A model for human open government data interaction.” In Journal of the Association for Information Science and Technology, v. 74, n. 13, p. 1498-1514
Xiao, F., Thaker, K. and He, D. (2022). “Categorizing Open Government Data Users by Exploring their Challenges and Proficiency.” In CHI Conference on Human Factors in Computing Systems, pp.1–7.
Xiao, F., Wang, Z. and He, D. (2020). “Understanding users’ accessing behaviors to local Open Government Data via transaction log analysis.” In Proceedings of the 83rd Annual Association for Information Science and Technology. v. 57, n. 1, p. e27
Publicado
08/09/2025
Como Citar
MENDES, Luana B.; MACIEL, Cristiano; BERARDI, Rita; NUNES, Eunice P. dos Santos.
Frameworks and Methods for Evaluating Open Data Platforms Under the Lens of Human-Data Interaction: A Systematic Review. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 24. , 2025, Belo Horizonte/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 1651-1674.
DOI: https://doi.org/10.5753/ihc.2025.10973.
