UX-MAPPER: A User eXperience Method to Analyze App Store Reviews
Resumo
The mobile app market has grown over the last decades. With the rise of app stores, users can easily choose an app from thousands, making them less tolerant of low-quality apps. More than ever, users are looking for apps that provide not only valuable functionalities but pleasurable experiences. Hence, User eXperience (UX) became the differential to stand out from competitors. By understanding what factors affect UX, practitioners could focus on factors that lead to positive UX while mitigating those that affect UX negatively. In this context, reviews from app stores emerged as a valuable source of information to investigate such factors. However, analyzing millions of reviews is costly and time-consuming. This paper presents UX-MAPPER, an approach to analyzing app store reviews and supporting practitioners in identifying factors affecting UX. We applied the Design Science Research method to design UX-MAPPER iteratively and grounded on a solid theoretical background. We performed exploratory studies to investigate the problem, a systematic mapping study to identify factors that affect UX, and an empirical study with 14 participants with experience in requirements engineering to determine the relevance and acceptance of our proposal from practitioners’ perspectives. The participants considered it useful to improve the quality of existing apps and explore the reviews of competing apps to identify functionalities and features that users are requesting, liking, or hating. They were also willing to use it when it became available, highlighting our proposal’s usefulness and relevance in software development.
Palavras-chave:
user experience, user reviews, machine learning, app stores
Referências
Fouad Nasser A Al Omran and Christoph Treude. 2017. Choosing an NLP library for analyzing software documentation: a systematic literature review and a series of experiments. In 2017 IEEE/ACM 14th international conference on mining software repositories (MSR). IEEE, 187–197.
Rui Alves, Pedro Valente, and Nuno Jardim Nunes. 2014. The state of user experience evaluation practice. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. 93–102.
Elsa Bakiu and Emitza Guzman. 2017. Which feature is unusable? Detecting usability and user experience issues from user reviews. In 2017 IEEE 25th international requirements engineering conference workshops (REW). IEEE, 182–187.
Julia Ayumi Bopp, Elisa D Mekler, and Klaus Opwis. 2016. Negative emotion, positive experience? Emotionally moving moments in digital games. In Proceedings of the 2016 CHI conference on human factors in computing systems. 2996–3006.
Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49–59.
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785–794.
Jacek Dąbrowski, Emmanuel Letier, Anna Perini, and Angelo Susi. 2020. Mining user opinions to support requirement engineering: an empirical study. In Advanced Information Systems Engineering: 32nd International Conference, CAiSE 2020, Grenoble, France, June 8–12, 2020, Proceedings 32. Springer, 401–416.
Giulia de Andrade Cardieri and Luciana Martinez Zaina. 2018. Analyzing user experience in mobile web, native and progressive web applications: a user and HCI specialist perspectives. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems. 1–11.
Adailton Ferreira de Araújo and Ricardo Marcondes Marcacini. 2021. Re-bert: automatic extraction of software requirements from app reviews using bert language model. In Proceedings of the 36th Annual ACM Symposium on Applied Computing. 1321–1327.
Vinicius HS Durelli, Rafael S Durelli, Andre T Endo, Elder Cirilo, Washington Luiz, and Leonardo Rocha. 2018. Please please me: does the presence of test cases influence mobile app users’ satisfaction?. In Proceedings of the XXXII Brazilian Symposium on Software Engineering. 132–141.
Emitza Guzman and Walid Maalej. 2014. How do users like this feature? a fine grained sentiment analysis of app reviews. In 2014 IEEE 22nd international requirements engineering conference (RE). Ieee, 153–162.
Emitza Guzman, Luís Oliveira, Yves Steiner, Laura C Wagner, and Martin Glinz. 2018. User feedback in the app store: a cross-cultural study. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Society. 13–22.
Emitza Guzman and Andres Paredes Rojas. 2019. Gender and user feedback: An exploratory study. In 2019 IEEE 27th international requirements engineering conference (RE). IEEE, 381–385.
Marc Hassenzahl. 2007. The hedonic/pragmatic model of user experience. Towards a UX manifesto 10 (2007), 2007.
Steffen Hedegaard and Jakob Grue Simonsen. 2014. Mining until it hurts: automatic extraction of usability issues from online reviews compared to traditional usability evaluation. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. 157–166.
Alan Hevner and Samir Chatterjee. 2010. Design research in information systems. Theory and practice. Springer.
Alan R Hevner. 2007. A three cycle view of design science research. Scandinavian journal of information systems 19, 2 (2007), 4.
Jincheul Jang and Mun Yong Yi. 2017. Modeling user satisfaction from the extraction of user experience elements in online product reviews. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 1718–1725.
Timo Johann, Christoph Stanik, Walid Maalej, 2017. Safe: A simple approach for feature extraction from app descriptions and app reviews. In 2017 IEEE 25th international requirements engineering conference (RE). IEEE, 21–30.
Effie Lai-Chong Law, Paul Van Schaik, and Virpi Roto. 2014. Attitudes towards user experience (UX) measurement. International Journal of Human-Computer Studies 72, 6 (2014), 526–541.
Walid Maalej, Zijad Kurtanović, Hadeer Nabil, and Christoph Stanik. 2016. On the automatic classification of app reviews. Requirements Engineering 21 (2016), 311–331.
Leonardo Marques, Patrícia Gomes Matsubara, Walter Takashi Nakamura, Bruna Moraes Ferreira, Igor Scaliante Wiese, Bruno Freitas Gadelha, Luciana Martinez Zaina, David Redmiles, and Tayana Uchôa Conte. 2021. Understanding UX better: A new technique to go beyond emotion assessment. Sensors 21, 21 (2021), 7183.
Stuart McIlroy, Nasir Ali, Hammad Khalid, and Ahmed E. Hassan. 2016. Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empirical Software Engineering 21 (2016), 1067–1106.
Walter T Nakamura, Edson Cesar de Oliveira, Elaine HT de Oliveira, David Redmiles, and Tayana Conte. 2022. What factors affect the UX in mobile apps? a systematic mapping study on the analysis of app store reviews. Journal of Systems and Software (2022), 111462.
Walter T Nakamura, Elaine HT de Oliveira, and Tayana Conte. 2019. Negative emotions, positive experience: What are we doing wrong when evaluating the ux?. In Extended abstracts of the 2019 chi conference on human factors in computing systems. 1–6.
Walter Takashi Nakamura, José Cezar de Souza, Lígia Márcia Teixeira, Afonso Silva, Rogenis da Silva, Bruno Gadelha, and Tayana Conte. 2021. Requirements Behind Reviews: How do Software Practitioners See App User Reviews to Think of Requirements?. In XX Brazilian Symposium on Software Quality. 1–9.
Walter Takashi Nakamura, Leonardo C Marques, Bruna Ferreira, Simone DJ Barbosa, and Tayana Conte. 2020. To inspect or to test? What approach provides better results when it comes to usability and UX?. In ICEIS (2). 487–498.
Walter T Nakamura, Leonardo C Marques, David Redmiles, Elaine HT de Oliveira, and Tayana Conte. 2022. Investigating the Influence of Different Factors on the UX Evaluation of a Mobile Application. International Journal of Human–Computer Interaction (2022), 1–21.
Walter Takashi Nakamura, Leonardo Carneiro Marques, Luis Rivero, Elaine HT de Oliveira, and Tayana Conte. 2019. Are scale-based techniques enough for learners to convey their UX when using a Learning Management System?Revista Brasileira de Informática na Educação 27, 01 (2019), 104.
Jakub Nalepa and Michal Kawulok. 2019. Selecting training sets for support vector machines: a review. Artificial Intelligence Review 52, 2 (2019), 857–900.
Fabio Palomba, Pasquale Salza, Adelina Ciurumelea, Sebastiano Panichella, Harald Gall, Filomena Ferrucci, and Andrea De Lucia. 2017. Recommending and localizing change requests for mobile apps based on user reviews. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 106–117.
Sebastiano Panichella, Andrea Di Sorbo, Emitza Guzman, Corrado A Visaggio, Gerardo Canfora, and Harald C Gall. 2015. How can i improve my app? classifying user reviews for software maintenance and evolution. In 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 281–290.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).
Jason D Rennie, Lawrence Shih, Jaime Teevan, and David R Karger. 2003. Tackling the poor assumptions of naive bayes text classifiers. In Proceedings of the 20th international conference on machine learning (ICML-03). 616–623.
Luis Rivero and Tayana Conte. 2017. A systematic mapping study on research contributions on UX evaluation technologies. In Proceedings of the XVI Brazilian symposium on human factors in computing systems. 1–10.
Martin Schrepp, Andreas Hinderks, and Jörg Thomaschewski. 2017. Design and evaluation of a short version of the user experience questionnaire (UEQ-S). International Journal of Interactive Multimedia and Artificial Intelligence, 4 (6), 103-108. (2017).
Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis Vlahavas. 2011. On the stratification of multi-label data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22. Springer, 145–158.
Viswanath Venkatesh and Hillol Bala. 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences 39, 2 (2008), 273–315.
Roel J Wieringa. 2014. Design science methodology for information systems and software engineering. Springer.
Rui Alves, Pedro Valente, and Nuno Jardim Nunes. 2014. The state of user experience evaluation practice. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. 93–102.
Elsa Bakiu and Emitza Guzman. 2017. Which feature is unusable? Detecting usability and user experience issues from user reviews. In 2017 IEEE 25th international requirements engineering conference workshops (REW). IEEE, 182–187.
Julia Ayumi Bopp, Elisa D Mekler, and Klaus Opwis. 2016. Negative emotion, positive experience? Emotionally moving moments in digital games. In Proceedings of the 2016 CHI conference on human factors in computing systems. 2996–3006.
Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49–59.
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785–794.
Jacek Dąbrowski, Emmanuel Letier, Anna Perini, and Angelo Susi. 2020. Mining user opinions to support requirement engineering: an empirical study. In Advanced Information Systems Engineering: 32nd International Conference, CAiSE 2020, Grenoble, France, June 8–12, 2020, Proceedings 32. Springer, 401–416.
Giulia de Andrade Cardieri and Luciana Martinez Zaina. 2018. Analyzing user experience in mobile web, native and progressive web applications: a user and HCI specialist perspectives. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems. 1–11.
Adailton Ferreira de Araújo and Ricardo Marcondes Marcacini. 2021. Re-bert: automatic extraction of software requirements from app reviews using bert language model. In Proceedings of the 36th Annual ACM Symposium on Applied Computing. 1321–1327.
Vinicius HS Durelli, Rafael S Durelli, Andre T Endo, Elder Cirilo, Washington Luiz, and Leonardo Rocha. 2018. Please please me: does the presence of test cases influence mobile app users’ satisfaction?. In Proceedings of the XXXII Brazilian Symposium on Software Engineering. 132–141.
Emitza Guzman and Walid Maalej. 2014. How do users like this feature? a fine grained sentiment analysis of app reviews. In 2014 IEEE 22nd international requirements engineering conference (RE). Ieee, 153–162.
Emitza Guzman, Luís Oliveira, Yves Steiner, Laura C Wagner, and Martin Glinz. 2018. User feedback in the app store: a cross-cultural study. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Society. 13–22.
Emitza Guzman and Andres Paredes Rojas. 2019. Gender and user feedback: An exploratory study. In 2019 IEEE 27th international requirements engineering conference (RE). IEEE, 381–385.
Marc Hassenzahl. 2007. The hedonic/pragmatic model of user experience. Towards a UX manifesto 10 (2007), 2007.
Steffen Hedegaard and Jakob Grue Simonsen. 2014. Mining until it hurts: automatic extraction of usability issues from online reviews compared to traditional usability evaluation. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. 157–166.
Alan Hevner and Samir Chatterjee. 2010. Design research in information systems. Theory and practice. Springer.
Alan R Hevner. 2007. A three cycle view of design science research. Scandinavian journal of information systems 19, 2 (2007), 4.
Jincheul Jang and Mun Yong Yi. 2017. Modeling user satisfaction from the extraction of user experience elements in online product reviews. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 1718–1725.
Timo Johann, Christoph Stanik, Walid Maalej, 2017. Safe: A simple approach for feature extraction from app descriptions and app reviews. In 2017 IEEE 25th international requirements engineering conference (RE). IEEE, 21–30.
Effie Lai-Chong Law, Paul Van Schaik, and Virpi Roto. 2014. Attitudes towards user experience (UX) measurement. International Journal of Human-Computer Studies 72, 6 (2014), 526–541.
Walid Maalej, Zijad Kurtanović, Hadeer Nabil, and Christoph Stanik. 2016. On the automatic classification of app reviews. Requirements Engineering 21 (2016), 311–331.
Leonardo Marques, Patrícia Gomes Matsubara, Walter Takashi Nakamura, Bruna Moraes Ferreira, Igor Scaliante Wiese, Bruno Freitas Gadelha, Luciana Martinez Zaina, David Redmiles, and Tayana Uchôa Conte. 2021. Understanding UX better: A new technique to go beyond emotion assessment. Sensors 21, 21 (2021), 7183.
Stuart McIlroy, Nasir Ali, Hammad Khalid, and Ahmed E. Hassan. 2016. Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empirical Software Engineering 21 (2016), 1067–1106.
Walter T Nakamura, Edson Cesar de Oliveira, Elaine HT de Oliveira, David Redmiles, and Tayana Conte. 2022. What factors affect the UX in mobile apps? a systematic mapping study on the analysis of app store reviews. Journal of Systems and Software (2022), 111462.
Walter T Nakamura, Elaine HT de Oliveira, and Tayana Conte. 2019. Negative emotions, positive experience: What are we doing wrong when evaluating the ux?. In Extended abstracts of the 2019 chi conference on human factors in computing systems. 1–6.
Walter Takashi Nakamura, José Cezar de Souza, Lígia Márcia Teixeira, Afonso Silva, Rogenis da Silva, Bruno Gadelha, and Tayana Conte. 2021. Requirements Behind Reviews: How do Software Practitioners See App User Reviews to Think of Requirements?. In XX Brazilian Symposium on Software Quality. 1–9.
Walter Takashi Nakamura, Leonardo C Marques, Bruna Ferreira, Simone DJ Barbosa, and Tayana Conte. 2020. To inspect or to test? What approach provides better results when it comes to usability and UX?. In ICEIS (2). 487–498.
Walter T Nakamura, Leonardo C Marques, David Redmiles, Elaine HT de Oliveira, and Tayana Conte. 2022. Investigating the Influence of Different Factors on the UX Evaluation of a Mobile Application. International Journal of Human–Computer Interaction (2022), 1–21.
Walter Takashi Nakamura, Leonardo Carneiro Marques, Luis Rivero, Elaine HT de Oliveira, and Tayana Conte. 2019. Are scale-based techniques enough for learners to convey their UX when using a Learning Management System?Revista Brasileira de Informática na Educação 27, 01 (2019), 104.
Jakub Nalepa and Michal Kawulok. 2019. Selecting training sets for support vector machines: a review. Artificial Intelligence Review 52, 2 (2019), 857–900.
Fabio Palomba, Pasquale Salza, Adelina Ciurumelea, Sebastiano Panichella, Harald Gall, Filomena Ferrucci, and Andrea De Lucia. 2017. Recommending and localizing change requests for mobile apps based on user reviews. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 106–117.
Sebastiano Panichella, Andrea Di Sorbo, Emitza Guzman, Corrado A Visaggio, Gerardo Canfora, and Harald C Gall. 2015. How can i improve my app? classifying user reviews for software maintenance and evolution. In 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 281–290.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).
Jason D Rennie, Lawrence Shih, Jaime Teevan, and David R Karger. 2003. Tackling the poor assumptions of naive bayes text classifiers. In Proceedings of the 20th international conference on machine learning (ICML-03). 616–623.
Luis Rivero and Tayana Conte. 2017. A systematic mapping study on research contributions on UX evaluation technologies. In Proceedings of the XVI Brazilian symposium on human factors in computing systems. 1–10.
Martin Schrepp, Andreas Hinderks, and Jörg Thomaschewski. 2017. Design and evaluation of a short version of the user experience questionnaire (UEQ-S). International Journal of Interactive Multimedia and Artificial Intelligence, 4 (6), 103-108. (2017).
Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis Vlahavas. 2011. On the stratification of multi-label data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22. Springer, 145–158.
Viswanath Venkatesh and Hillol Bala. 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences 39, 2 (2008), 273–315.
Roel J Wieringa. 2014. Design science methodology for information systems and software engineering. Springer.
Publicado
16/10/2023
Como Citar
NAKAMURA, Walter T.; C. DE OLIVEIRA, Edson César; H. T. DE OLIVEIRA, Elaine; CONTE, Tayana.
UX-MAPPER: A User eXperience Method to Analyze App Store Reviews. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
