Assessing Data Visualization Literacy: Design, Implementation, and Analysis of a Comprehensive Test
Resumo
Data Visualization Literacy involves recognizing a given chart, reading it correctly, and extracting information from it. Familiarity with a specific type of visualization does not imply that the person can read or interpret it correctly. Data visualization researchers have attempted to explore and promote solutions to support data visualization comprehension activities. Studies address developing and applying tests to assess literacy, understanding how analysts interpret visualizations and how unfamiliar visualizations are taught, and even identifying the cognitive activities involved in creating visualizations. However, we have not found comprehensive studies relating to these different aspects, which we believe are essential for teaching and learning about data visualizations and data analysis tasks. This paper presents our procedure for designing a new visualization literacy assessment. We devised a test that covered 15 different visualizations and applied it with 68 participants. We analyzed each item, assessing their difficulty level and discrimination. After removing eight items that did not discriminate well, our final test ended with 37 items, all with fair or good discrimination. It means that the test should genuinely represent the test takers’ learning ability, i.e., the items can discriminate well between the high and low-performing groups.
Palavras-chave:
Visualization Literacy, Assessment Test
Referências
Basak Alper, Nathalie Henry Riche, Fanny Chevalier, Jeremy Boy, and Metin Sezgin. 2017. Visualization Literacy at Elementary School. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, Denver, Colorado, USA, 5485–5497. DOI: 10.1145/3025453.3025877
Fearn Bishop, Johannes Zagermann, Ulrike Pfeil, Gemma Sanderson, Harald Reiterer, and Uta Hinrichs. 2020. Construct-A-Vis: exploring the free-form visualization processes of children. IEEE Transactions on Visualization and Computer Graphics 26, 1 (Jan 2020), 451–460. DOI: 10.1109/TVCG.2019.2934804
Michelle A. Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S. Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. 2015. Beyond memorability: Visualization recognition and recall. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2015), 519–528.
Jeremy Boy, Ronald A. Rensink, Enrico Bertini, and Jean-Daniel Fekete. 2014. A Principled Way of Assessing Visualization Literacy. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec 2014), 1963–1972. DOI: 10.1109/TVCG.2014.2346984
Sabrina Bresciani. 2009. The risks of visualization: A classification of disadvantages associated with graphic representations of information. In Identität und Vielfalt der Kommunikationswissenschaft (2009). UVK Verlagsgesellschaft GmbH, 165–178.
Katy Börner, Andreas Bueckle, and Michael Ginda. 2019. Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences 116, 6 (2019), 1857–1864. DOI: 10.1073/pnas.1807180116
Katy Börner, Adam Maltese, Russell Nelson Balliet, and Joe Heimlich. 2016. Investigating aspects of data visualization literacy using 20 information visualizations and 273 science museum visitors. Information Visualization 15, 3 (2016), 198–213. DOI: 10.1177/1473871615594652
Fanny Chevalier, Nathalie Henry Riche, Basak Alper, Catherine Plaisant, Jeremy Boy, and Niklas Elmqvist. 2018. Observations and Reflections on Visualization Literacy in Elementary School. IEEE Computer Graphics and Applications 38, 3 (May 2018), 21–29. DOI: 10.1109/MCG.2018.032421650
Lee J. Cronbach. 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16, 3 (Sept. 1951), 297–334. DOI: 10.1007/BF02310555
Cynthya Letícia Teles de Oliveira, Alan Trindade de Almeida Silva, Jefferson Magalhães de Morais, and Marcelle Pereira Mota. 2020. ChartVision: accessible vertical bar charts. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 9, 10 pages. DOI: 10.1145/3424953.3426644
E. Firat, Alena Denisova, and R. Laramee. 2020. Treemap literacy: A classroom-based investigation. In Eurographics Proceedings. The Eurographics Association, 29–38. DOI: 10.2312/eged.20201032
Elif E. Firat, Alena Denisova, Max L. Wilson, and Robert S. Laramee. 2022. P-lite: A study of parallel coordinate plot literacy. Visual Informatics 6, 3 (2022), 81–99.
Gustavo Romão Gonzales and Flávio Horita. 2020. Supporting visual analytics in decision support system: a systematic mapping study. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 34, 10 pages. DOI: 10.1145/3424953.3426483
S. Huron, Y. Jansen, and S. Carpendale. 2014. Constructing visual representations: investigating the use of tangible tokens. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec. 2014), 2102–2111. DOI: 10.1109/TVCG.2014.2346292.
Daniel A. Keim, Florian Mansmann, Daniela Oelke, and Hartmut Ziegler. 2008. Visual analytics: combining automated discovery with interactive visualizations. In Proceedings of the 11th International Conference on Discovery Science (Budapest, Hungary). Springer-Verlag, 2–14. DOI: 10.1007/978-3-540-88411-8_2
Alicia Key, Bill Howe, Daniel Perry, and Cecilia Aragon. 2012. VizDeck: self-organizing dashboards for visual analytics. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (Scottsdale, Arizona, USA). Association for Computing Machinery, 681–684. DOI: 10.1145/2213836.2213931.
Andy Kirk. 2016. Data Visualisation: A Handbook for Data Driven Design. Sage Publications Ltd.
Neesha Kodagoda, B. L. William Wong, Chris Rooney, and Nawaz Khan. 2012. Interactive Visualization for Low Literacy Users: From Lessons Learnt to Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1159–1168. DOI: 10.1145/2207676.2208565
Kenneth R. Koedinger, R. S. Baker, and A. T. Corbett. 2001. Toward a model of learning data representations. In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society. Scholarship University of California, 45–50.
Andrey Krekhov, Michael Michalski, and Jens Krüger. 2019. Integrating Visualization Literacy into Computer Graphics Education Using the Example of Dear Data. In Eurographics 2019 Education Papers. The Eurographics Association, 1–8. DOI: 10.2312/EGED.20191022
S. Lee, S. Kim, and B. C. Kwon. 2017. VLAT: Development of a visualization literacy assessment test. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 551–560. DOI: 10.1109/TVCG.2016.2598920
Sukwon Lee, Bum Chul Kwon, Jiming Yang, Byung Cheol Lee, and Sung-Hee Kim. 2019. The correlation between users’ cognitive characteristics and visualization literacy. Applied Sciences 9, 3 (2019), 488.
Franklin M. da C. Lima, Leonardo Cunha de Miranda, and M. Cecília C. Baranauskas. 2022. Visualizing and analyzing the evolution of authorship and co-authorship networks of articles from the Brazilian symposium on human factors in computing systems. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’22). Association for Computing Machinery, New York, NY, USA, Article 31, 11 pages. DOI: 10.1145/3554364.3559132
Maylon Pires Macedo, Ranilson Oscar Araújo Paiva, Isabela Gasparini, and Luciana Aparecida Martinez Zaina. 2020. Vis2Learning: a scenario-based guide of recommendations for building educational data visualizations. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 36, 10 pages. DOI: 10.1145/3424953.3426627
Adam V. Maltese, Joseph A. Harsh, and Dubravka Svetina. 2015. Data Visualization Literacy: Investigating Data Interpretation Along the Novice—Expert Continuum. Journal of College Science Teaching 45, 1 (2015), 84–90. [link]
Tamara Munzner. 2014. Visualization analysis and design. CRC press.
Saugat Pandey and Alvitta Ottley. 2023. Mini-VLAT: A Short and Effective Measure of Visualization Literacy. Computer Graphics Forum 42, 3 (2023), 1–11. DOI: 10.1111/cgf.14809 arXiv: [link]
Patrícia Martinková and Adéla Drabinová. 2018. ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal 10, 2 (2018), 503–515. DOI:
Ariane Moraes Bueno Rodrigues, Gabriel Diniz Junqueira Barbosa, Hélio Côrtes Vieira Lopes, and Simone Diniz Junqueira Barbosa. 2021. What questions reveal about novices’ attempts to make sense of data visualizations: Patterns and misconceptions. Computers & Graphics 94 (2021), 32–42. DOI: //doi.org/10.1016/j.cag.2020.09.015
Ariane Moraes Bueno Rodrigues, Gabriel Diniz Junqueira Barbosa, Hélio Côrtes Vieira Lopes, and Simone Diniz Junqueira Barbosa. 2022. Investigating whether people identify how suitable a data visualization is for answering specific analysis questions. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’22). Association for Computing Machinery, New York, NY, USA, Article 22, 11 pages. DOI: 10.1145/3554364.3560904
Puripant Ruchikachorn and Klaus Mueller. 2015. Learning visualizations by analogy: Promoting visual literacy through visualization morphing. IEEE Transactions on Visualization and Computer Graphics 21, 9 (2015), 1028–1044.
Bahador Saket, Arjun Srinivasan, Eric D. Ragan, and Alex Endert. 2018. Evaluating Interactive Graphical Encodings for Data Visualization. IEEE Transactions on Visualization and Computer Graphics 24, 3 (March 2018), 1316–1330. DOI: 10.1109/TVCG.2017.2680452. Conference Name: IEEE Transactions on Visualization and Computer Graphics.
John R. Thompson, Jesse J. Martinez, Alper Sarikaya, Edward Cutrell, and Bongshin Lee. 2023. Chart Reader: Accessible Visualization Experiences Designed with Screen Reader Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 802, 18 pages.
Robert M Thorndike, George K Cunningham, Robert Ladd Thorndike, and Elizabeth P Hagen. 1991. Measurement and evaluation in psychology and education. Macmillan Publishing Co, Inc, 175 5th Avenue New York, NY 10010 United States.
Edward R Tufte and Peter R Graves-Morris. 1986. The visual display of quantitative information. Information Design Journal 4, 3 (Jan. 1986), 235–236. DOI: 10.1075/idj.4.3.12cos
José M. F. Vieira and Luciana A. M. Zaina. 2020. Representation, navigation and exploration: a three layered approach on learning trajectories. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 31, 10 pages. DOI: 10.1145/3424953.3426629
Zezhong Wang, Lovisa Sundin, Dave Murray-Rust, and Benjamin Bach. 2020. Cheat Sheets for Data Visualization Techniques. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1–13. DOI: 10.1145/3313831.3376271
Colin Ware. 2019. Information visualization: perception for design. Morgan Kaufmann.
Fearn Bishop, Johannes Zagermann, Ulrike Pfeil, Gemma Sanderson, Harald Reiterer, and Uta Hinrichs. 2020. Construct-A-Vis: exploring the free-form visualization processes of children. IEEE Transactions on Visualization and Computer Graphics 26, 1 (Jan 2020), 451–460. DOI: 10.1109/TVCG.2019.2934804
Michelle A. Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S. Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. 2015. Beyond memorability: Visualization recognition and recall. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2015), 519–528.
Jeremy Boy, Ronald A. Rensink, Enrico Bertini, and Jean-Daniel Fekete. 2014. A Principled Way of Assessing Visualization Literacy. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec 2014), 1963–1972. DOI: 10.1109/TVCG.2014.2346984
Sabrina Bresciani. 2009. The risks of visualization: A classification of disadvantages associated with graphic representations of information. In Identität und Vielfalt der Kommunikationswissenschaft (2009). UVK Verlagsgesellschaft GmbH, 165–178.
Katy Börner, Andreas Bueckle, and Michael Ginda. 2019. Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences 116, 6 (2019), 1857–1864. DOI: 10.1073/pnas.1807180116
Katy Börner, Adam Maltese, Russell Nelson Balliet, and Joe Heimlich. 2016. Investigating aspects of data visualization literacy using 20 information visualizations and 273 science museum visitors. Information Visualization 15, 3 (2016), 198–213. DOI: 10.1177/1473871615594652
Fanny Chevalier, Nathalie Henry Riche, Basak Alper, Catherine Plaisant, Jeremy Boy, and Niklas Elmqvist. 2018. Observations and Reflections on Visualization Literacy in Elementary School. IEEE Computer Graphics and Applications 38, 3 (May 2018), 21–29. DOI: 10.1109/MCG.2018.032421650
Lee J. Cronbach. 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16, 3 (Sept. 1951), 297–334. DOI: 10.1007/BF02310555
Cynthya Letícia Teles de Oliveira, Alan Trindade de Almeida Silva, Jefferson Magalhães de Morais, and Marcelle Pereira Mota. 2020. ChartVision: accessible vertical bar charts. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 9, 10 pages. DOI: 10.1145/3424953.3426644
E. Firat, Alena Denisova, and R. Laramee. 2020. Treemap literacy: A classroom-based investigation. In Eurographics Proceedings. The Eurographics Association, 29–38. DOI: 10.2312/eged.20201032
Elif E. Firat, Alena Denisova, Max L. Wilson, and Robert S. Laramee. 2022. P-lite: A study of parallel coordinate plot literacy. Visual Informatics 6, 3 (2022), 81–99.
Gustavo Romão Gonzales and Flávio Horita. 2020. Supporting visual analytics in decision support system: a systematic mapping study. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 34, 10 pages. DOI: 10.1145/3424953.3426483
S. Huron, Y. Jansen, and S. Carpendale. 2014. Constructing visual representations: investigating the use of tangible tokens. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec. 2014), 2102–2111. DOI: 10.1109/TVCG.2014.2346292.
Daniel A. Keim, Florian Mansmann, Daniela Oelke, and Hartmut Ziegler. 2008. Visual analytics: combining automated discovery with interactive visualizations. In Proceedings of the 11th International Conference on Discovery Science (Budapest, Hungary). Springer-Verlag, 2–14. DOI: 10.1007/978-3-540-88411-8_2
Alicia Key, Bill Howe, Daniel Perry, and Cecilia Aragon. 2012. VizDeck: self-organizing dashboards for visual analytics. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (Scottsdale, Arizona, USA). Association for Computing Machinery, 681–684. DOI: 10.1145/2213836.2213931.
Andy Kirk. 2016. Data Visualisation: A Handbook for Data Driven Design. Sage Publications Ltd.
Neesha Kodagoda, B. L. William Wong, Chris Rooney, and Nawaz Khan. 2012. Interactive Visualization for Low Literacy Users: From Lessons Learnt to Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1159–1168. DOI: 10.1145/2207676.2208565
Kenneth R. Koedinger, R. S. Baker, and A. T. Corbett. 2001. Toward a model of learning data representations. In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society. Scholarship University of California, 45–50.
Andrey Krekhov, Michael Michalski, and Jens Krüger. 2019. Integrating Visualization Literacy into Computer Graphics Education Using the Example of Dear Data. In Eurographics 2019 Education Papers. The Eurographics Association, 1–8. DOI: 10.2312/EGED.20191022
S. Lee, S. Kim, and B. C. Kwon. 2017. VLAT: Development of a visualization literacy assessment test. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 551–560. DOI: 10.1109/TVCG.2016.2598920
Sukwon Lee, Bum Chul Kwon, Jiming Yang, Byung Cheol Lee, and Sung-Hee Kim. 2019. The correlation between users’ cognitive characteristics and visualization literacy. Applied Sciences 9, 3 (2019), 488.
Franklin M. da C. Lima, Leonardo Cunha de Miranda, and M. Cecília C. Baranauskas. 2022. Visualizing and analyzing the evolution of authorship and co-authorship networks of articles from the Brazilian symposium on human factors in computing systems. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’22). Association for Computing Machinery, New York, NY, USA, Article 31, 11 pages. DOI: 10.1145/3554364.3559132
Maylon Pires Macedo, Ranilson Oscar Araújo Paiva, Isabela Gasparini, and Luciana Aparecida Martinez Zaina. 2020. Vis2Learning: a scenario-based guide of recommendations for building educational data visualizations. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 36, 10 pages. DOI: 10.1145/3424953.3426627
Adam V. Maltese, Joseph A. Harsh, and Dubravka Svetina. 2015. Data Visualization Literacy: Investigating Data Interpretation Along the Novice—Expert Continuum. Journal of College Science Teaching 45, 1 (2015), 84–90. [link]
Tamara Munzner. 2014. Visualization analysis and design. CRC press.
Saugat Pandey and Alvitta Ottley. 2023. Mini-VLAT: A Short and Effective Measure of Visualization Literacy. Computer Graphics Forum 42, 3 (2023), 1–11. DOI: 10.1111/cgf.14809 arXiv: [link]
Patrícia Martinková and Adéla Drabinová. 2018. ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal 10, 2 (2018), 503–515. DOI:
Ariane Moraes Bueno Rodrigues, Gabriel Diniz Junqueira Barbosa, Hélio Côrtes Vieira Lopes, and Simone Diniz Junqueira Barbosa. 2021. What questions reveal about novices’ attempts to make sense of data visualizations: Patterns and misconceptions. Computers & Graphics 94 (2021), 32–42. DOI: //doi.org/10.1016/j.cag.2020.09.015
Ariane Moraes Bueno Rodrigues, Gabriel Diniz Junqueira Barbosa, Hélio Côrtes Vieira Lopes, and Simone Diniz Junqueira Barbosa. 2022. Investigating whether people identify how suitable a data visualization is for answering specific analysis questions. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’22). Association for Computing Machinery, New York, NY, USA, Article 22, 11 pages. DOI: 10.1145/3554364.3560904
Puripant Ruchikachorn and Klaus Mueller. 2015. Learning visualizations by analogy: Promoting visual literacy through visualization morphing. IEEE Transactions on Visualization and Computer Graphics 21, 9 (2015), 1028–1044.
Bahador Saket, Arjun Srinivasan, Eric D. Ragan, and Alex Endert. 2018. Evaluating Interactive Graphical Encodings for Data Visualization. IEEE Transactions on Visualization and Computer Graphics 24, 3 (March 2018), 1316–1330. DOI: 10.1109/TVCG.2017.2680452. Conference Name: IEEE Transactions on Visualization and Computer Graphics.
John R. Thompson, Jesse J. Martinez, Alper Sarikaya, Edward Cutrell, and Bongshin Lee. 2023. Chart Reader: Accessible Visualization Experiences Designed with Screen Reader Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 802, 18 pages.
Robert M Thorndike, George K Cunningham, Robert Ladd Thorndike, and Elizabeth P Hagen. 1991. Measurement and evaluation in psychology and education. Macmillan Publishing Co, Inc, 175 5th Avenue New York, NY 10010 United States.
Edward R Tufte and Peter R Graves-Morris. 1986. The visual display of quantitative information. Information Design Journal 4, 3 (Jan. 1986), 235–236. DOI: 10.1075/idj.4.3.12cos
José M. F. Vieira and Luciana A. M. Zaina. 2020. Representation, navigation and exploration: a three layered approach on learning trajectories. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’20). Association for Computing Machinery, New York, NY, USA, Article 31, 10 pages. DOI: 10.1145/3424953.3426629
Zezhong Wang, Lovisa Sundin, Dave Murray-Rust, and Benjamin Bach. 2020. Cheat Sheets for Data Visualization Techniques. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1–13. DOI: 10.1145/3313831.3376271
Colin Ware. 2019. Information visualization: perception for design. Morgan Kaufmann.
Publicado
07/11/2024
Como Citar
RODRIGUES, Ariane Moraes Bueno; BARBOSA, Gabriel Diniz Junqueira; LOPES, Hélio Côrtes Vieira; BARBOSA, Simone Diniz Junqueira.
Assessing Data Visualization Literacy: Design, Implementation, and Analysis of a Comprehensive Test. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 23. , 2024, Brasília/DF.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 594-606.
