Survey and classification of recommendations for evaluating visualization and usability quality criteria in Business Intelligence systems
Resumo
The growing reliance on data for strategic decision-making intensifies the relevance of Business Intelligence (BI) tools. In this context, it is crucial to ensure the quality of user interactions and visualizations provided by these platforms to the end-user. However, there are gaps in the evaluation of BI tools quality in terms of usability and visualization quality criteria, especially with respect to the heuristic evaluation method. This paper presents partial results of an ongoing doctoral research project aimed at filling this gap. Through a literature review, we surveyed 22 recommendations for BI tools regarding visualization and usability aspects. We also mapped these recommendations to the visualization heuristics of Morroni et al. to identify similarities between groups of recommendations. We expect to use this classification to generate a new set of heuristics for evaluating BI systems, with the goal of enabling the improvement of these systems’ quality.
Palavras-chave:
Business Intelligence, Heuristic Evaluation, Data Visualization, Usability
Referências
Ö. Işik, M. C. Jones, and A. Sidorova, “Business Intelligence Success: The Roles of BI Capabilities and Decision Environments,” Information & Management, vol. 50, no. 1, pp. 13–23, December 2013.
J. Ekanem, “Evaluating the usability of data preparation tools for self-service business intelligence users,” Utrecht University Student Theses Repository, 2022.
M. Smuts, B. Scholtz, and A. Calitz, “Design Guidelines for Business Intelligence Tools for Novice Users,” in Proceedings of the Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, ser. SAICSIT ’15, 2015, pp. 1–10.
Salesforce, Inc. (2023) O que é Business Intelligence (BI)? Online. [Accessed: 22-jul-2025]. [Online]. Available: [link]
D. Bassic and R. Henry, “The Role of Business Information Visualization in Knowledge Creation,” in AMCIS 2012 Proceedings, ser. AMCIS 2012, 2012.
J. Nielsen, “Enhancing the Explanatory Power of Usability Heuristics,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’94. New York, NY, USA: ACM, 1994, p. 152–158. [Online]. DOI: 10.1145/191666.191729
A. Muppidi, A. S. Hashim, M. H. Hasan, and A. A. Muazu, “A conceptual ux model for designing and developing the business intelligence dashboards,” Journal of Computer Science, vol. 19, no. 12, pp. 1505–1519, 2023.
J. M. Villamarín and B. Diaz Pinzon, “Key Success Factors to Business Intelligence Solution Implementation,” Journal of Intelligence Studies in Business, vol. 7, no. 1, pp. 48–69, 2017.
S. Few, Show Me the Numbers: Designing Tables and Graphs to Enlighten. Analytics Press, 2004.
——, Information Dashboard Design: The Effective Visual Communication of Data. Sebastopol, CA: O’Reilly Media, 2006, vol. 2.
E. R. Tufte, The Visual Display of Quantitative Information, 2nd ed. Graphics Press, 2001.
S. K. Card, J. D. Mackinlay, and B. Shneiderman, Eds., Readings in Information Visualization: Using Vision to Think. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1999.
S. Liu, H. Zhang, Z. Yang, J. Kong, L. Zhang, and C. Gao, “UXBIV: An Evaluation Framework for Business Intelligence Visualization,” IEEE Access, vol. 11, pp. 92 391–92 415, 2023.
R. Sharda, D. Delen, and E. Turban, Business Intelligence, Analytics, and Data Science: A Managerial Perspective, 4th ed. Pearson, 2017.
L. Daly and M. Reynolds, Dashboards for Excel: The Business Intelligence Workbench, 2nd ed. Wiley, 2016.
B. Bach, E. Freeman, A. Abdul-Rahman, C. Turkay, S. Khan, Y. Fan, and M. Chen, “Dashboard Design Patterns,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 1, pp. 342–352, 2023.
R. Kitchin, T. P. Lauriault, and G. McArdle, “Knowing and Governing Cities through Urban Indicators, City Benchmarking and Real-Time Dashboards,” Regional Studies, Regional Science, vol. 2, no. 1, pp. 6–28, 2015.
J. Nielsen and R. L. Mack, Usability Inspection Methods. John Wiley & Sons, 1994.
G. Beelders and P. Kotzé, “Augmenting the Business Intelligence Life-cycle Model with Usability: Using Eye-Tracking to Discover the Why of Usability Problems,” Journal of Business Intelligence, vol. 2, no. 1, 2020.
C. M. D. S. Freitas, P. R. G. Luzzardi, R. A. Cava, M. Winckler, M. S. Pimenta, and L. P. Nedel, “Evaluating Usability of Information Visualization Techniques,” in Proceedings of 5th Symposium on Human Factors in Computer Systems, 2002, pp. 40–51.
R. Amar and J. Stasko, “A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations,” in IEEE Symposium on Information Visualization, 2004, pp. 143–150.
T. Zuk and S. Carpendale, “Theoretical analysis of uncertainty visualizations,” in Visualization and Data Analysis 2006, vol. 6060, International Society for Optics and Photonics. SPIE, 2006, pp. 66 – 79. [Online]. DOI: 10.1117/12.643631
D. L. Scapin and J. M. C. Bastien, “Ergonomic criteria for evaluating the ergonomic quality of interactive systems,” Behaviour & Information Technology, vol. 16, no. 4-5, pp. 220–231, 1997. [Online]. DOI: 10.1080/014492997119806
C. Forsell and J. Johansson, “An Heuristic Set for Evaluation in Information Visualization,” in Proceedings of the International Conference on Advanced Visual Interfaces, ser. AVI ’10. New York, NY, USA: ACM, 2010, p. 199–206. [Online]. DOI: 10.1145/1842993.1843029
M. R. Oliveira and C. G. Silva, “Adapting Heuristic Evaluation to Information Visualization - A Method for Defining a Heuristic Set by Heuristic Grouping,” in Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - IVAPP, (VISIGRAPP 2017), INSTICC. SciTePress, 2017, pp. 225–232.
B. S. Santos, S. Silva, and P. Dias, “Heuristic Evaluation in Visualization: An Empirical Study,” in 2018 IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV), 2018, pp. 78–85.
H. Väätäjä, J. Varsaluoma, T. Heimonen, K. Tiitinen, J. Hakulinen, M. Turunen, H. Nieminen, and P. Ihantola, “Information Visualization Heuristics in Practical Expert Evaluation,” in Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, ser. BELIV ’16. New York, NY, USA: ACM, 2016, p. 36–43. [Online]. DOI: 10.1145/2993901.2993918
A. Tarrell, A. Fruhling, R. Borgo, C. Forsell, G. Grinstein, and J. Scholtz, “Toward Visualization-Specific Heuristic Evaluation,” in Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, ser. BELIV ’14. New York, NY, USA: ACM, 2014, p. 110–117. [Online]. DOI: 10.1145/2669557.2669580
Y. Zhu and J. A. Gumieniak, “Computer-Assisted Heuristic Evaluation of Data Visualization,” in Advances in Visual Computing. Cham: Springer International Publishing, 2021, pp. 408–420.
V. Setlur, M. Correll, A. Satyanarayan, and M. Tory, “Heuristics for Supporting Cooperative Dashboard Design,” IEEE Transactions on Visualization and Computer Graphics, vol. 30, no. 1, pp. 370–380, 2024.
J. A. Morroni, O. P. Praça, V. A. S. de Freitas, G. L. Girardi, and C. G. da Silva, “Defining Information Visualization Heuristics Based on Heuristic Grouping by Experts,” in Proceedings of the International Conference on Interfaces and Human Computer Interaction; Computer Graphics, Visualization, Computer Vision and Image Processing; and Game and Entertainment Technologies, 2023, pp. 133–140.
B. Shneiderman, “The eyes have it: a task by data type taxonomy for information visualizations,” in Proceedings 1996 IEEE Symposium on Visual Languages, 1996, pp. 336–343.
R. Magdalena, Y. Ruldeviyani, D. I. Sensuse, and C. Bernando, “Methods to Enhance the Utilization of Business Intelligence Dashboard by Integration of Evaluation and User Testing,” in 2019 3rd International Conference on Informatics and Computational Sciences (ICICoS), 2019.
C. Jooste, J. Van Biljon, and J. Mentz, “Usability Evaluation for Business Intelligence Applications: A User Support Perspective,” South African Computer Journal, vol. 53, pp. 1–13, Aug. 2014.
I. Khanal, “A Study on Usages of Visualization Tools to Remove Biases in Decision Making,” Master’s thesis, Tribhuvan University, Institute of Engineering, 2023.
A. Even, Y. Kolodner, and R. Varshavsky, “Designing Business-Intelligence Tools with Value-Driven Recommendations,” in Lecture Notes in Computer Science. Springer, 2010, pp. 286–301.
T. Gunadham, “A Study of Business Intelligence Tools from Users’ Perspectives,” Interdisciplinary Research Review, vol. 18, no. 3, pp. 8–14, Jun. 2023.
D. Bačić and A. Fadlalla, “Business Information Visualization Intellectual Contributions: An Integrative Framework of Visualization Capabilities and Dimensions of Visual Intelligence,” Decision Support Systems, vol. 89, pp. 77–86, 2016.
M. Durra and G. Al-Naymat, “Self-Service Business Intelligence: Meeting User Needs in a Data-Rich World,” in 2023 24th International Arab Conference on Information Technology (ACIT), 2023.
A. Hussain, S. Diamond, S. Szigeti, M. A. Gordon, F. Yuan, M. Diep, and L.-X. Dong, “HCI Design Principles and Visual Analytics for Media Analytics Platform,” in HCI International 2019 - Posters, ser. Communications in Computer and Information Science, C. Stephanidis, Ed. Springer, 2019, vol. 1034, pp. 28–35.
J. Ekanem, “Evaluating the usability of data preparation tools for self-service business intelligence users,” Utrecht University Student Theses Repository, 2022.
M. Smuts, B. Scholtz, and A. Calitz, “Design Guidelines for Business Intelligence Tools for Novice Users,” in Proceedings of the Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, ser. SAICSIT ’15, 2015, pp. 1–10.
Salesforce, Inc. (2023) O que é Business Intelligence (BI)? Online. [Accessed: 22-jul-2025]. [Online]. Available: [link]
D. Bassic and R. Henry, “The Role of Business Information Visualization in Knowledge Creation,” in AMCIS 2012 Proceedings, ser. AMCIS 2012, 2012.
J. Nielsen, “Enhancing the Explanatory Power of Usability Heuristics,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’94. New York, NY, USA: ACM, 1994, p. 152–158. [Online]. DOI: 10.1145/191666.191729
A. Muppidi, A. S. Hashim, M. H. Hasan, and A. A. Muazu, “A conceptual ux model for designing and developing the business intelligence dashboards,” Journal of Computer Science, vol. 19, no. 12, pp. 1505–1519, 2023.
J. M. Villamarín and B. Diaz Pinzon, “Key Success Factors to Business Intelligence Solution Implementation,” Journal of Intelligence Studies in Business, vol. 7, no. 1, pp. 48–69, 2017.
S. Few, Show Me the Numbers: Designing Tables and Graphs to Enlighten. Analytics Press, 2004.
——, Information Dashboard Design: The Effective Visual Communication of Data. Sebastopol, CA: O’Reilly Media, 2006, vol. 2.
E. R. Tufte, The Visual Display of Quantitative Information, 2nd ed. Graphics Press, 2001.
S. K. Card, J. D. Mackinlay, and B. Shneiderman, Eds., Readings in Information Visualization: Using Vision to Think. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1999.
S. Liu, H. Zhang, Z. Yang, J. Kong, L. Zhang, and C. Gao, “UXBIV: An Evaluation Framework for Business Intelligence Visualization,” IEEE Access, vol. 11, pp. 92 391–92 415, 2023.
R. Sharda, D. Delen, and E. Turban, Business Intelligence, Analytics, and Data Science: A Managerial Perspective, 4th ed. Pearson, 2017.
L. Daly and M. Reynolds, Dashboards for Excel: The Business Intelligence Workbench, 2nd ed. Wiley, 2016.
B. Bach, E. Freeman, A. Abdul-Rahman, C. Turkay, S. Khan, Y. Fan, and M. Chen, “Dashboard Design Patterns,” IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 1, pp. 342–352, 2023.
R. Kitchin, T. P. Lauriault, and G. McArdle, “Knowing and Governing Cities through Urban Indicators, City Benchmarking and Real-Time Dashboards,” Regional Studies, Regional Science, vol. 2, no. 1, pp. 6–28, 2015.
J. Nielsen and R. L. Mack, Usability Inspection Methods. John Wiley & Sons, 1994.
G. Beelders and P. Kotzé, “Augmenting the Business Intelligence Life-cycle Model with Usability: Using Eye-Tracking to Discover the Why of Usability Problems,” Journal of Business Intelligence, vol. 2, no. 1, 2020.
C. M. D. S. Freitas, P. R. G. Luzzardi, R. A. Cava, M. Winckler, M. S. Pimenta, and L. P. Nedel, “Evaluating Usability of Information Visualization Techniques,” in Proceedings of 5th Symposium on Human Factors in Computer Systems, 2002, pp. 40–51.
R. Amar and J. Stasko, “A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations,” in IEEE Symposium on Information Visualization, 2004, pp. 143–150.
T. Zuk and S. Carpendale, “Theoretical analysis of uncertainty visualizations,” in Visualization and Data Analysis 2006, vol. 6060, International Society for Optics and Photonics. SPIE, 2006, pp. 66 – 79. [Online]. DOI: 10.1117/12.643631
D. L. Scapin and J. M. C. Bastien, “Ergonomic criteria for evaluating the ergonomic quality of interactive systems,” Behaviour & Information Technology, vol. 16, no. 4-5, pp. 220–231, 1997. [Online]. DOI: 10.1080/014492997119806
C. Forsell and J. Johansson, “An Heuristic Set for Evaluation in Information Visualization,” in Proceedings of the International Conference on Advanced Visual Interfaces, ser. AVI ’10. New York, NY, USA: ACM, 2010, p. 199–206. [Online]. DOI: 10.1145/1842993.1843029
M. R. Oliveira and C. G. Silva, “Adapting Heuristic Evaluation to Information Visualization - A Method for Defining a Heuristic Set by Heuristic Grouping,” in Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - IVAPP, (VISIGRAPP 2017), INSTICC. SciTePress, 2017, pp. 225–232.
B. S. Santos, S. Silva, and P. Dias, “Heuristic Evaluation in Visualization: An Empirical Study,” in 2018 IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV), 2018, pp. 78–85.
H. Väätäjä, J. Varsaluoma, T. Heimonen, K. Tiitinen, J. Hakulinen, M. Turunen, H. Nieminen, and P. Ihantola, “Information Visualization Heuristics in Practical Expert Evaluation,” in Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, ser. BELIV ’16. New York, NY, USA: ACM, 2016, p. 36–43. [Online]. DOI: 10.1145/2993901.2993918
A. Tarrell, A. Fruhling, R. Borgo, C. Forsell, G. Grinstein, and J. Scholtz, “Toward Visualization-Specific Heuristic Evaluation,” in Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, ser. BELIV ’14. New York, NY, USA: ACM, 2014, p. 110–117. [Online]. DOI: 10.1145/2669557.2669580
Y. Zhu and J. A. Gumieniak, “Computer-Assisted Heuristic Evaluation of Data Visualization,” in Advances in Visual Computing. Cham: Springer International Publishing, 2021, pp. 408–420.
V. Setlur, M. Correll, A. Satyanarayan, and M. Tory, “Heuristics for Supporting Cooperative Dashboard Design,” IEEE Transactions on Visualization and Computer Graphics, vol. 30, no. 1, pp. 370–380, 2024.
J. A. Morroni, O. P. Praça, V. A. S. de Freitas, G. L. Girardi, and C. G. da Silva, “Defining Information Visualization Heuristics Based on Heuristic Grouping by Experts,” in Proceedings of the International Conference on Interfaces and Human Computer Interaction; Computer Graphics, Visualization, Computer Vision and Image Processing; and Game and Entertainment Technologies, 2023, pp. 133–140.
B. Shneiderman, “The eyes have it: a task by data type taxonomy for information visualizations,” in Proceedings 1996 IEEE Symposium on Visual Languages, 1996, pp. 336–343.
R. Magdalena, Y. Ruldeviyani, D. I. Sensuse, and C. Bernando, “Methods to Enhance the Utilization of Business Intelligence Dashboard by Integration of Evaluation and User Testing,” in 2019 3rd International Conference on Informatics and Computational Sciences (ICICoS), 2019.
C. Jooste, J. Van Biljon, and J. Mentz, “Usability Evaluation for Business Intelligence Applications: A User Support Perspective,” South African Computer Journal, vol. 53, pp. 1–13, Aug. 2014.
I. Khanal, “A Study on Usages of Visualization Tools to Remove Biases in Decision Making,” Master’s thesis, Tribhuvan University, Institute of Engineering, 2023.
A. Even, Y. Kolodner, and R. Varshavsky, “Designing Business-Intelligence Tools with Value-Driven Recommendations,” in Lecture Notes in Computer Science. Springer, 2010, pp. 286–301.
T. Gunadham, “A Study of Business Intelligence Tools from Users’ Perspectives,” Interdisciplinary Research Review, vol. 18, no. 3, pp. 8–14, Jun. 2023.
D. Bačić and A. Fadlalla, “Business Information Visualization Intellectual Contributions: An Integrative Framework of Visualization Capabilities and Dimensions of Visual Intelligence,” Decision Support Systems, vol. 89, pp. 77–86, 2016.
M. Durra and G. Al-Naymat, “Self-Service Business Intelligence: Meeting User Needs in a Data-Rich World,” in 2023 24th International Arab Conference on Information Technology (ACIT), 2023.
A. Hussain, S. Diamond, S. Szigeti, M. A. Gordon, F. Yuan, M. Diep, and L.-X. Dong, “HCI Design Principles and Visual Analytics for Media Analytics Platform,” in HCI International 2019 - Posters, ser. Communications in Computer and Information Science, C. Stephanidis, Ed. Springer, 2019, vol. 1034, pp. 28–35.
Publicado
30/09/2025
Como Citar
LOMONACO JÚNIOR, Fabiano Gonçalves; SILVA, Celmar Guimarães da.
Survey and classification of recommendations for evaluating visualization and usability quality criteria in Business Intelligence systems. In: WORKSHOP DE TRABALHOS EM ANDAMENTO - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 38. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 184-189.
