Mind the Gap Between UX Data and Visualization Proposals: An Approach Emerged From the Grey Literature to Support the Analysis of User Dissatisfaction
Abstract
Software teams usually perform data analysis about user experience data (i.e., UX data) to extract positive and negative issues of users' interaction with the software. The scientific literature has presented forms of visualizing UX data. However, these visualization forms often are not evaluated by software professionals (i.e., designers and developers) in practice to check whether such visualizations can provide enough information to understand users' satisfaction with the product. In addition, software professionals have adapted the visualizations proposed in the literature considering their own learning that emerged from their practice. These professionals' lessons learned have resulted in a significant amount of practical knowledge on UX data visualizations published in UX blogs and websites (known as grey literature). Considering the scattered knowledge available in UX blogs and websites, we searched for publications in the grey literature that contained definitions of UX data and also of visualizations used to explore it. From the 144 articles explored, we found three legs that can support the investigation of causes of users' dissatisfaction reports: the visualization approach, the purpose of using the visualization, and UX data per se. We evaluated with 31 software professionals as a proof of concept on the relevance of the three-leg approach (i.e., visualization, purposes, and UX data) to help software teams identify insights about events that triggered the users' dissatisfaction reports. We selected four users' dissatisfaction reports that emerged from the grey literature analysis and constructed five visualizations to explore UX data related to a mobile app for airline tickets. We chose the airline tickets domain due to the fact that it is a common sense domain that avoids the efforts of interpreting the application domain. From an online questionnaire, we collected the participants' interpretations of the UX data available from the five visualizations. Our primary concern was evaluating how well the visualizations helped participants identify aspects of the mobile application that resulted in reports of user dissatisfaction. Data on participants' interpretation of visualizations were analyzed using descriptive statistics, and open-ended questions on definitions of UX data were analyzed using open coding. The participants evaluated positively the use of the three-leg approach to analyze UX data (i.e., an average of 78% of positive feedback). The findings revealed that most participants recognized that data on product use (e.g., logs and screen recording) was the only suitable relevant UX data source. In addition to evaluating the visualizations positively, a good number of participants (i.e., 84%) pointed out that the proposed visualizations could be appropriated to support their daily work.
References
Andrea Batch, Yipeng Ji, Mingming Fan, Jian Zhao, and Niklas Elmqvist. 2023. uxSense: Supporting User Experience Analysis with Visualization and Computer Vision. IEEE Transactions on Visualization and Computer Graphics (2023), 1–15. DOI: 10.1109/TVCG.2023.3241581
Paolo Buono, Danilo Caivano, Maria Francesca Costabile, Giuseppe Desolda, and Rosa Lanzilotti. 2020. Towards the Detection of UX Smells: The Support of Visualizations. IEEE Access 8 (2020), 6901–6914. DOI: 10.1109/ACCESS.2019.2961768.
Stuart Card, Jock Mackinlay, and Ben Shneiderman. 1999. Readings in Information Visualization: Using Vision To Think. Academic Press. DOI: doi/10.5555/300679
Gregorio Convertino and Nancy Frishberg. 2017. Why agile teams fail without UX research. Commun. ACM 60, 9 (2017), 35–37. DOI: 10.1145/3126156
Roberto Yuri Da Silva Franco, Alexandre Abreu De Freitas, Rodrigo Santos Do Amor Divino Lima, Marcelle Pereira Mota, Carlos Gustavo Resque Dos Santos, and Bianchi Serique Meiguins. 2019. UXmood A Tool to Investigate the User Experience (UX) Based on Multimodal Sentiment Analysis and Information Visualization (InfoVis). In 2019 23rd International Conference Information Visualisation (IV). 175–180. DOI: 10.1109/IV.2019.00038
Svenja Dittrich, Ferdinand Hof, and Alexander Wiethoff. 2019. InteracDiff: Visualizing and Interacting with UX-Data. In Proceedings of Mensch Und Computer 2019 (Hamburg, Germany) (MuC ’19). Association for Computing Machinery, New York, NY, USA, 583–587. DOI: 10.1145/3340764.3344463
Mike Fritz and Paul D. Berger. 2015. Improving the User Experience through Practical Data Analytics: Gain Meaningful Insight and Increase Your Bottom Line. Elsevier Science. DOI: 10.1016/C2013-0-18588-1
Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. 2016. The Need for Multivocal Literature Reviews in Software Engineering: Complementing Systematic Literature Reviews with Grey Literature. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering (Limerick, Ireland) (EASE ’16). Association for Computing Machinery, New York, NY, USA, Article 26, 6 pages. DOI: 10.1145/2915970.2916008
Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology 106 (2019), 101–121. DOI: 10.1016/j.infsof.2018.09.006
R.L. Glass and T. DeMarco. 2006. Software Creativity 2.0. Developer.* Books.
Marc Hassenzahl. 2018. The Thing and I (Summer of ’17 Remix). Springer International Publishing. 17–31 pages. DOI: 10.1007/978-3-319-68213-6_2
Laura Koesten and Elena Simperl. 2021. UX of Data: Making Data Available Doesn’t Make It Usable. Interactions 28, 2 (mar 2021), 97–99. DOI: 10.1145/3448888
Heidi Lam, Enrico Bertini, Petra Isenberg, Catherine Plaisant, and Sheelagh Carpendale. 2012. Empirical Studies in Information Visualization: Seven Scenarios. IEEE Transactions on Visualization and Computer Graphics 18, 9 (2012), 1520–1536. DOI: 10.1109/TVCG.2011.279
Laura Luther, Victor Tiberius, and Alexander Brem. 2020. User Experience (UX) in Business, Management, and Psychology: A Bibliometric Mapping of the Current State of Research. Multimodal Technologies and Interaction 4, 2 (May 2020), 18. DOI: 10.3390/mti4020018
Suéllen Martinelli, Larissa Lopes, and Luciana Zaina. 2022. UX research in the software industry: an investigation of long-term UX practices. In Anais do XXI Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (Diamantina). SBC, Porto Alegre, RS, Brasil. [link]
Tamara Munzner. 2014. Visualization analysis and design. A K Peters.
Don Norman and Jakob Nielsen. 2018. Nielsen Norman Group The definition of user experience (UX). [link].
Yanzhang Tong, Yang Xiang, Irena Spasic, Yulia Hicks, Huicong Hu, and Ying Liu. 2022. A Data-Driven Approach for Integrating Hedonic Quality and Pragmatic Quality in User Experience Modeling. Journal of Computing and Information Science in Engineering (2022). DOI: 10.1115/1.4054155
Colin Ware. 2012. Information Visualization: perception for design (3 ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
Luciana A.M. Zaina, Helen Sharp, and Leonor Barroca. 2021. UX information in the daily work of an agile team: A distributed cognition analysis. International Journal of Human-Computer Studies 147 (2021), 102574. DOI: 10.1016/j.ijhcs.2020.102574
