Evolution may come with a price: analyzing user reviews to understand the impact of updates on mobile apps accessibility
Resumo
Mobile applications are constantly updated to adapt to evolving user and environment requirements. As changes are successively implemented to promote user satisfaction, the complexity of mobile apps increases, and overlooked quality aspects (e.g. privacy, security, power consumption) may decline if counter-measures are not adopted. In this paper, we analyzed accessibility reviews of mobile apps to show evidence that updates may introduce barriers that make the app less accessible than its previous version according to users' perceptions. Our results show that accessibility barriers reported by users mostly include incompatibility with screen readers, the removal of accessibility features (e.g. color scheme or font customization), small font sizes and widgets, and incompatibility with the device's accessibility configuration. The accessibility barriers impact the users considering different levels: i) perception: an inability or difficulty to see, read or distinguish interface elements due to their small size or color; ii) understanding: the inability or difficulty to navigate, access and interpret information; iii) operation: the inability of difficulty to perform tasks such as adding items to the cart, reading or sending messages, and booking a ride; iv) and physical reactions, such as eyestrain, vertigo, and headache. The sentiments expressed by users are generally negative, including frustration, disappointment, and sometimes a sense of discrimination against people with disabilities. The results of our study raise an alert to organizations and developers that they should implement measures to avoid introducing accessibility barriers while they add new features to their mobile apps.
Palavras-chave:
Accessibility, mobile, app, update, evolution, user reviews
Referências
Wajdi Aljedaani, Mohammed Alkahtani, Stephanie Ludi, Mohamed Wiem Mkaouer, Marcelo M. Eler, Marouane Kessentini, and Ali Ouni. 2023. The State of Accessibility in Blackboard: Survey and User Reviews Case Study. In Proceedings of the 20th International Web for All Conference (Austin, TX, USA) (W4A ’23). Association for Computing Machinery, New York, NY, USA, 84–95. DOI: 10.1145/3587281.3587291
Wajdi Aljedaani, Mohamed Wiem Mkaouer, Stephanie Ludi, and Yasir Javed. 2022. Automatic Classification of Accessibility User Reviews in Android Apps. In 2022 7th International Conference on Data Science and Machine Learning Applications (CDMA). IEEE, 133–138.
Wajdi Aljedaani, Mohamed Wiem Mkaouer, Stephanie Ludi, Ali Ouni, and Ilyes Jenhani. 2022. On the identification of accessibility bug reports in open source systems. In Proceedings of the 19th International Web for All Conference. 1–11.
Wajdi Aljedaani, Furqan Rustam, Stephanie Ludi, Ali Ouni, and Mohamed Wiem Mkaouer. 2021. Learning sentiment analysis for accessibility user reviews. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). IEEE, 239–246.
Eman Abdullah AlOmar, Wajdi Aljedaani, Murtaza Tamjeed, Mohamed Wiem Mkaouer, and Yasmine N El-Glaly. 2021. Finding the needle in a haystack: On the automatic identification of accessibility user reviews. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–15.
Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020. Accessibility issues in android apps: state of affairs, sentiments, and ways forward. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 1323–1334.
Tingting Bi, Xin Xia, David Lo, John Grundy, Thomas Zimmermann, and Denae Ford. 2021. Accessibility in software practice: A practitioner’s perspective. ACM Transactions on Software Engineering and Methodology (2021).
Courtni Byun, Piper Vasicek, and Kevin Seppi. 2023. Dispensing with Humans in Human-Computer Interaction Research. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). Association for Computing Machinery, New York, NY, USA, Article 413, 26 pages. DOI: 10.1145/3544549.3582749
Sen Chen, Chunyang Chen, Lingling Fan, Mingming Fan, Xian Zhan, and Yang Liu. 2022. Accessible or Not? An Empirical Investigation of Android App Accessibility. IEEE Transactions on Software Engineering 48, 10 (2022), 3954–3968. DOI: 10.1109/TSE.2021.3108162
Adelina Ciurumelea, Andreas Schaufelbühl, Sebastiano Panichella, and Harald C Gall. 2017. Analyzing reviews and code of mobile apps for better release planning. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 91–102.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46.
Marcelo Medeiros Eler, Leandro Orlandin, and Alberto Dumont Alves Oliveira. 2019. Do Android app users care about accessibility? an analysis of user reviews on the Google play store. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems. 1–11.
Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, 1981. The measurement of interrater agreement. Statistical methods for rates and proportions 2, 212-236 (1981), 22–23.
Jun Gao, Li Li, Tegawendé F. Bissyandé, and Jacques Klein. 2019. On the Evolution of Mobile App Complexity. In 2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS). 200–209. DOI: 10.1109/ICECCS.2019.00029
C. Iacob and R. Harrison. 2013. Retrieving and analyzing mobile apps feature requests from online reviews. In 2013 10th Working Conference on Mining Software Repositories (MSR). 41–44. DOI: 10.1109/MSR.2013.6624001
Frederic B. Kraft and Charles L. Martin. 2001. Customer Compliments as More than Complementary Feedback. Journal of Consumer Satisfaction, Dissatisfaction & Complaining Behavior 14 (March 2001), 1–13. [link]
M.M. Lehman. 1980. Programs, life cycles, and laws of software evolution. Proc. IEEE 68, 9 (1980), 1060–1076. DOI: 10.1109/PROC.1980.11805
M. M. Lehman. 1996. Laws of software evolution revisited. In Software Process Technology, Carlo Montangero (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 108–124.
Manoel Victor Rodrigues Leite, Lilian Passos Scatalon, André Pimenta Freire, and Marcelo Medeiros Eler. 2021. Accessibility in the mobile development industry in Brazil: Awareness, knowledge, adoption, motivations and barriers. Journal of Systems and Software 177 (2021), 110942.
Tianyang Liu, Chong Wang, Kun Huang, Peng Liang, Beiqi Zhang, Maya Daneva, and Marten van Sinderen. 2023. RoseMatcher: Identifying the impact of user reviews on app updates. Information and Software Technology (2023), 107261. DOI: 10.1016/j.infsof.2023.107261
Stuart McIlroy, Nasir Ali, and Ahmed E. Hassan. 2016. Fresh apps: an empirical study of frequently-updated mobile apps in the Google play store. Empirical Software Engineering 21, 3 (June 2016), 1346–1370. DOI: 10.1007/s10664-015-9388-2
Maleknaz Nayebi, Konstantin Kuznetsov, Paul Chen, Andreas Zeller, and Guenther Ruhe. 2018. Anatomy of Functionality Deletion: An Exploratory Study on Mobile Apps. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New York, NY, USA, 243–253. DOI: 10.1145/3196398.3196410
Alberto Dumont Alves Oliveira, Paulo Sérgio Henrique Dos Santos, Wilson Estécio Marcílio Júnior, Wajdi M Aljedaani, Danilo Medeiros Eler, and Marcelo Medeiros Eler. 2023. Analyzing Accessibility Reviews Associated with Visual Disabilities or Eye Conditions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 37, 14 pages. DOI: 10.1145/3544548.3581315
Fabio Palomba, Mario Linares-Vásquez, Gabriele Bavota, Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk, and Andrea De Lucia. 2018. Crowdsourcing user reviews to support the evolution of mobile apps. Journal of Systems and Software 137 (2018), 143 – 162. DOI: 10.1016/j.jss.2017.11.043
Rahul Potharaju, Mizanur Rahman, and Bogdan Carbunar. 2017. A Longitudinal Study of Google Play. IEEE Transactions on Computational Social Systems PP (08 2017), 1–15. DOI: 10.1109/TCSS.2017.2732167
Emma Sabzalieva and Arianna Valentini. 2023. ChatGPT and artificial intelligence in higher education: quick start guide. UNESCO International Institute for Higher Education in Latin America and the Caribbean 5 (April 2023). [link]
Mara Taynar Santiago and Anna Beatriz Marques. 2022. Are User Reviews Useful for Identifying Accessibility Issues That Autistic Users Face? An Exploratory Study. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’22). Association for Computing Machinery, New York, NY, USA, Article 6, 11 pages. DOI: 10.1145/3554364.3559114
Vincent F. Taylor and Ivan Martinovic. 2017. To Update or Not to Update: Insights From a Two-Year Study of Android App Evolution. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (Abu Dhabi, United Arab Emirates) (ASIA CCS ’17). Association for Computing Machinery, New York, NY, USA, 45–57. DOI: 10.1145/3052973.3052990
Shunguo Yan and PG Ramachandran. 2019. The current status of accessibility in mobile apps. ACM Transactions on Accessible Computing (TACCESS) 12, 1 (2019), 1–31.
Aidan Z. H. Yang, Safwat Hassan, Ying Zou, and Ahmed E. Hassan. 2022. An empirical study on release notes patterns of popular apps in the Google Play Store. Empirical Software Engineering 27, 2 (March 2022), 55. DOI: 10.1007/s10664-021-10086-2
Wajdi Aljedaani, Mohamed Wiem Mkaouer, Stephanie Ludi, and Yasir Javed. 2022. Automatic Classification of Accessibility User Reviews in Android Apps. In 2022 7th International Conference on Data Science and Machine Learning Applications (CDMA). IEEE, 133–138.
Wajdi Aljedaani, Mohamed Wiem Mkaouer, Stephanie Ludi, Ali Ouni, and Ilyes Jenhani. 2022. On the identification of accessibility bug reports in open source systems. In Proceedings of the 19th International Web for All Conference. 1–11.
Wajdi Aljedaani, Furqan Rustam, Stephanie Ludi, Ali Ouni, and Mohamed Wiem Mkaouer. 2021. Learning sentiment analysis for accessibility user reviews. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). IEEE, 239–246.
Eman Abdullah AlOmar, Wajdi Aljedaani, Murtaza Tamjeed, Mohamed Wiem Mkaouer, and Yasmine N El-Glaly. 2021. Finding the needle in a haystack: On the automatic identification of accessibility user reviews. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–15.
Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020. Accessibility issues in android apps: state of affairs, sentiments, and ways forward. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 1323–1334.
Tingting Bi, Xin Xia, David Lo, John Grundy, Thomas Zimmermann, and Denae Ford. 2021. Accessibility in software practice: A practitioner’s perspective. ACM Transactions on Software Engineering and Methodology (2021).
Courtni Byun, Piper Vasicek, and Kevin Seppi. 2023. Dispensing with Humans in Human-Computer Interaction Research. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). Association for Computing Machinery, New York, NY, USA, Article 413, 26 pages. DOI: 10.1145/3544549.3582749
Sen Chen, Chunyang Chen, Lingling Fan, Mingming Fan, Xian Zhan, and Yang Liu. 2022. Accessible or Not? An Empirical Investigation of Android App Accessibility. IEEE Transactions on Software Engineering 48, 10 (2022), 3954–3968. DOI: 10.1109/TSE.2021.3108162
Adelina Ciurumelea, Andreas Schaufelbühl, Sebastiano Panichella, and Harald C Gall. 2017. Analyzing reviews and code of mobile apps for better release planning. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 91–102.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46.
Marcelo Medeiros Eler, Leandro Orlandin, and Alberto Dumont Alves Oliveira. 2019. Do Android app users care about accessibility? an analysis of user reviews on the Google play store. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems. 1–11.
Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, 1981. The measurement of interrater agreement. Statistical methods for rates and proportions 2, 212-236 (1981), 22–23.
Jun Gao, Li Li, Tegawendé F. Bissyandé, and Jacques Klein. 2019. On the Evolution of Mobile App Complexity. In 2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS). 200–209. DOI: 10.1109/ICECCS.2019.00029
C. Iacob and R. Harrison. 2013. Retrieving and analyzing mobile apps feature requests from online reviews. In 2013 10th Working Conference on Mining Software Repositories (MSR). 41–44. DOI: 10.1109/MSR.2013.6624001
Frederic B. Kraft and Charles L. Martin. 2001. Customer Compliments as More than Complementary Feedback. Journal of Consumer Satisfaction, Dissatisfaction & Complaining Behavior 14 (March 2001), 1–13. [link]
M.M. Lehman. 1980. Programs, life cycles, and laws of software evolution. Proc. IEEE 68, 9 (1980), 1060–1076. DOI: 10.1109/PROC.1980.11805
M. M. Lehman. 1996. Laws of software evolution revisited. In Software Process Technology, Carlo Montangero (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 108–124.
Manoel Victor Rodrigues Leite, Lilian Passos Scatalon, André Pimenta Freire, and Marcelo Medeiros Eler. 2021. Accessibility in the mobile development industry in Brazil: Awareness, knowledge, adoption, motivations and barriers. Journal of Systems and Software 177 (2021), 110942.
Tianyang Liu, Chong Wang, Kun Huang, Peng Liang, Beiqi Zhang, Maya Daneva, and Marten van Sinderen. 2023. RoseMatcher: Identifying the impact of user reviews on app updates. Information and Software Technology (2023), 107261. DOI: 10.1016/j.infsof.2023.107261
Stuart McIlroy, Nasir Ali, and Ahmed E. Hassan. 2016. Fresh apps: an empirical study of frequently-updated mobile apps in the Google play store. Empirical Software Engineering 21, 3 (June 2016), 1346–1370. DOI: 10.1007/s10664-015-9388-2
Maleknaz Nayebi, Konstantin Kuznetsov, Paul Chen, Andreas Zeller, and Guenther Ruhe. 2018. Anatomy of Functionality Deletion: An Exploratory Study on Mobile Apps. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New York, NY, USA, 243–253. DOI: 10.1145/3196398.3196410
Alberto Dumont Alves Oliveira, Paulo Sérgio Henrique Dos Santos, Wilson Estécio Marcílio Júnior, Wajdi M Aljedaani, Danilo Medeiros Eler, and Marcelo Medeiros Eler. 2023. Analyzing Accessibility Reviews Associated with Visual Disabilities or Eye Conditions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 37, 14 pages. DOI: 10.1145/3544548.3581315
Fabio Palomba, Mario Linares-Vásquez, Gabriele Bavota, Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk, and Andrea De Lucia. 2018. Crowdsourcing user reviews to support the evolution of mobile apps. Journal of Systems and Software 137 (2018), 143 – 162. DOI: 10.1016/j.jss.2017.11.043
Rahul Potharaju, Mizanur Rahman, and Bogdan Carbunar. 2017. A Longitudinal Study of Google Play. IEEE Transactions on Computational Social Systems PP (08 2017), 1–15. DOI: 10.1109/TCSS.2017.2732167
Emma Sabzalieva and Arianna Valentini. 2023. ChatGPT and artificial intelligence in higher education: quick start guide. UNESCO International Institute for Higher Education in Latin America and the Caribbean 5 (April 2023). [link]
Mara Taynar Santiago and Anna Beatriz Marques. 2022. Are User Reviews Useful for Identifying Accessibility Issues That Autistic Users Face? An Exploratory Study. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems (Diamantina, Brazil) (IHC ’22). Association for Computing Machinery, New York, NY, USA, Article 6, 11 pages. DOI: 10.1145/3554364.3559114
Vincent F. Taylor and Ivan Martinovic. 2017. To Update or Not to Update: Insights From a Two-Year Study of Android App Evolution. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (Abu Dhabi, United Arab Emirates) (ASIA CCS ’17). Association for Computing Machinery, New York, NY, USA, 45–57. DOI: 10.1145/3052973.3052990
Shunguo Yan and PG Ramachandran. 2019. The current status of accessibility in mobile apps. ACM Transactions on Accessible Computing (TACCESS) 12, 1 (2019), 1–31.
Aidan Z. H. Yang, Safwat Hassan, Ying Zou, and Ahmed E. Hassan. 2022. An empirical study on release notes patterns of popular apps in the Google Play Store. Empirical Software Engineering 27, 2 (March 2022), 55. DOI: 10.1007/s10664-021-10086-2
Publicado
16/10/2023
Como Citar
DOS SANTOS, Paulo Sérgio Henrique; OLIVEIRA, Alberto Dumont Alves; DE JESUS, Thais Bonjorni Nobre; ALJEDAANI, Wajdi; ELER, Marcelo Medeiros.
Evolution may come with a price: analyzing user reviews to understand the impact of updates on mobile apps accessibility. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
