Accessibility of Mobile Apps for Visually Impaired Users: Problems Encountered by User Evaluation, Inspections and Automated Tools
Resumo
Mobile apps should be usable and accessible to everyone, including visually impaired people. Accessibility evaluation techniques are essential to identify accessibility problems and help to avoid problems that hinder access by people with disabilities. Different accessibility evaluation methods may be used in different stages of software development. However, knowing the types of accessibility problems encountered by each method is vital for developers, testers, and designers to allocate the different methods appropriately in different stages of the development cycle. This study aimed to compare the outcomes of automated evaluations, manual inspection and user evaluation methods applied to mobile apps focusing on visually impaired users. The study compared the results from different methods by comparing results from a systematic mapping of the literature and the analysis of user evaluations performed in a previous study with two mobile apps. The study compared the results from the usability evaluations of Saraiva and Receita Federal by nine users with visual impairment and 189 instances of problems in two applications. These results were compared to the automated evaluation of the two apps by the tools MATE (Mobile Accessibility Testing) and Accessibility Scanner. Next, the apps were inspected by two groups of professionals: 17 experts in different areas of software development (full-stack developers, testers and front-end developers) and ten specialists in Human-Computer Interaction (HCI) with previous experience with accessibility. The results showed a difference between the different accessibility evaluation methods. Inspections by HCI specialists and tests with users with disabilities found a greater diversity of types of problems. Automated tools showed limited performance in the detection of types of problems. In accessibility inspections, there is a difference between specialists in software development and specialists in HCI. The results showed a difference in the number of violations that specialists in software development find compared to HCI specialists. This study contributes to the understanding of how different methods can contribute to the accessibility evaluation of mobile apps, helping different stakeholders make decisions about when and how to apply different methods.
Palavras-chave:
Mobile Accessibility, User Evaluation, Inspections
Referências
Julio Abascal, Myriam Arrue, and Xabier Valencia. 2019. Tools for web accessibility evaluation. Web accessibility: a foundation for research (2019), 479–503.
Associação Brasileira de Normas Técnicas (ABNT). 2022. ABNT NBR 17060 - Acessibilidade em aplicativos de dispositivos móveis - Requisitos (Accessibility in Mobile Apps - requirements). shorturl.at/lvwJ2
British Broadcasting Corporation BBC. 2014. HTML Accessibility Standards v2.0.BBC. Retrieved February 15, 2021 from [link].
Giorgio Brajnik. 2008. A comparative test of web accessibility evaluation methods. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility. 113–120.
Giorgio Brajnik, Markel Vigo, Yeliz Yesilada, and Simon Harper. 2016. Group vs individual web accessibility evaluations: effects with novice evaluators. Interacting with Computers 28, 6 (2016), 843–861.
Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2010. Testability and validity of WCAG 2.0: the expertise effect. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. 43–50.
Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2011. The expertise effect on web accessibility evaluation methods. Human–Computer Interaction 26, 3 (2011), 246–283.
British Broadcasting Corporation - BBC. 2014. Mobile Accessibility Standards and Guidelines v1.0. BBC. Retrieved February 15, 2021 from [link].
Michael Crystian Nepomuceno Carvalho, Felipe Silva Dias, Aline Grazielle Silva Reis, and André Pimenta Freire. 2018. Accessibility and Usability Problems Encountered on Websites and Applications in Mobile Devices by Blind and Normal-Vision Users. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing (Pau, France) (SAC ’18). Association for Computing Machinery, New York, NY, USA, 2022–2029.
Felipe Silva Dias. 2018. Análise de causas técnicasde problemas de acessibilidade encontrados em aplicativos nativos para dispositivos móveis. UFLA, Lavras.
Marcelo Medeiros Eler, José Miguel Rojas, Yan Ge, and Gordon Fraser. 2018. Automated accessibility testing of mobile apps. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 116–126.
André Pimenta Freire. 2012. Disabled people and the Web: User-based measurement of accessibility. Ph. D. Dissertation. University of York.
Google. 2015. Accessibility Scanner. [link]
Governo Brasileiro. 2015. Lei Brasileira de Inclusão da Pessoa com Deficiência. Retrieved January 31, 2023 from [link].
ISO - International Organization for Standardization. 2018. ISO 9241-11. Usability: Definitions and concepts. Retrieved January 31, 2023 from [link].
Paul T Jaeger. 2006. Assessing Section 508 compliance on federal e-government Web sites: A multi-method, user-centered evaluation of accessibility for persons with disabilities. Government Information Quarterly 23, 2 (2006), 169–190.
Jennifer Mankoff, Holly Fait, and Tu Tran. 2005. Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind. In Proceedings of the SIGCHI conference on Human factors in computing systems. 41–50.
Delvani Antônio Mateus, Simone Bacellar Leal Ferreira, Maurício Ronny de Almeida Souza, and André Pimenta Freire. 2023. Accessibility Inspections of Mobile Applications by Professionals with Different Expertise Levels: an Empirical Study Comparing with User Evaluations. In Human-Computer Interaction – INTERACT 2023 19th IFIP TC 13 International Conference. 1–23.
Delvani Antônio Mateus, Carlos Alberto Silva, Marcelo Medeiros Eler, and André Pimenta Freire. 2020. Accessibility of mobile applications: evaluation by users with visual impairment and by automated tools. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems. 1–10.
Christopher Power, André Freire, Helen Petrie, and David Swallow. 2012. Guidelines are only half of the story: accessibility problems encountered by blind users on the web. In Proceedings of the SIGCHI conference on human factors in computing systems. 433–442.
Christopher Power and Helen Petrie. 2019. Working with participants. Web Accessibility: A Foundation for Research (2019), 153–168.
Dagfinn Rømen and Dag Svanæs. 2012. Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users. Universal Access in the Information Society 11, 4 (2012), 375–385.
Statista. 2022. Market share of mobile operating systems in Brazil from January 2019 to August 2022. Retrieved 20 May 2023 from [link].
Statista. 2022. Mobile OS share in North America 2018-2023. Retrieved 20 May 2023 from [link].
Beat Vollenwyder, Serge Petralito, Glena H Iten, Florian Brühlmann, Klaus Opwis, and Elisa D Mekler. 2023. How compliance with web accessibility standards shapes the experiences of users with and without disabilities. International Journal of Human-Computer Studies 170 (2023), 102956.
W3C. 1999. Web Content Accessibility Guidelines 1.0, Web Accessibility Initiative. Retrieved January 31, 2023 from [link]. .
W3C. 2008. W3C – World Wide Web Consortium. Web Content Accessibility Guidelines (WCAG) 2.0. Retrieved January 31, 2023 from [link].
W3C. 2018. W3C. Web Content Accessibility Guidelines (WCAG) 2.1. Retrieved January 31, 2023 from [link].
W3C. 2020. W3C. Web Content Accessibility Guidelines (WCAG) 2.2. Retrieved January 31, 2023 from [link].
W3C. 2021. W3C. Web Content Accessibility Guidelines (WCAG) 3.0. Retrieved January 31, 2023 from [link].
Associação Brasileira de Normas Técnicas (ABNT). 2022. ABNT NBR 17060 - Acessibilidade em aplicativos de dispositivos móveis - Requisitos (Accessibility in Mobile Apps - requirements). shorturl.at/lvwJ2
British Broadcasting Corporation BBC. 2014. HTML Accessibility Standards v2.0.BBC. Retrieved February 15, 2021 from [link].
Giorgio Brajnik. 2008. A comparative test of web accessibility evaluation methods. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility. 113–120.
Giorgio Brajnik, Markel Vigo, Yeliz Yesilada, and Simon Harper. 2016. Group vs individual web accessibility evaluations: effects with novice evaluators. Interacting with Computers 28, 6 (2016), 843–861.
Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2010. Testability and validity of WCAG 2.0: the expertise effect. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. 43–50.
Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2011. The expertise effect on web accessibility evaluation methods. Human–Computer Interaction 26, 3 (2011), 246–283.
British Broadcasting Corporation - BBC. 2014. Mobile Accessibility Standards and Guidelines v1.0. BBC. Retrieved February 15, 2021 from [link].
Michael Crystian Nepomuceno Carvalho, Felipe Silva Dias, Aline Grazielle Silva Reis, and André Pimenta Freire. 2018. Accessibility and Usability Problems Encountered on Websites and Applications in Mobile Devices by Blind and Normal-Vision Users. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing (Pau, France) (SAC ’18). Association for Computing Machinery, New York, NY, USA, 2022–2029.
Felipe Silva Dias. 2018. Análise de causas técnicasde problemas de acessibilidade encontrados em aplicativos nativos para dispositivos móveis. UFLA, Lavras.
Marcelo Medeiros Eler, José Miguel Rojas, Yan Ge, and Gordon Fraser. 2018. Automated accessibility testing of mobile apps. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 116–126.
André Pimenta Freire. 2012. Disabled people and the Web: User-based measurement of accessibility. Ph. D. Dissertation. University of York.
Google. 2015. Accessibility Scanner. [link]
Governo Brasileiro. 2015. Lei Brasileira de Inclusão da Pessoa com Deficiência. Retrieved January 31, 2023 from [link].
ISO - International Organization for Standardization. 2018. ISO 9241-11. Usability: Definitions and concepts. Retrieved January 31, 2023 from [link].
Paul T Jaeger. 2006. Assessing Section 508 compliance on federal e-government Web sites: A multi-method, user-centered evaluation of accessibility for persons with disabilities. Government Information Quarterly 23, 2 (2006), 169–190.
Jennifer Mankoff, Holly Fait, and Tu Tran. 2005. Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind. In Proceedings of the SIGCHI conference on Human factors in computing systems. 41–50.
Delvani Antônio Mateus, Simone Bacellar Leal Ferreira, Maurício Ronny de Almeida Souza, and André Pimenta Freire. 2023. Accessibility Inspections of Mobile Applications by Professionals with Different Expertise Levels: an Empirical Study Comparing with User Evaluations. In Human-Computer Interaction – INTERACT 2023 19th IFIP TC 13 International Conference. 1–23.
Delvani Antônio Mateus, Carlos Alberto Silva, Marcelo Medeiros Eler, and André Pimenta Freire. 2020. Accessibility of mobile applications: evaluation by users with visual impairment and by automated tools. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems. 1–10.
Christopher Power, André Freire, Helen Petrie, and David Swallow. 2012. Guidelines are only half of the story: accessibility problems encountered by blind users on the web. In Proceedings of the SIGCHI conference on human factors in computing systems. 433–442.
Christopher Power and Helen Petrie. 2019. Working with participants. Web Accessibility: A Foundation for Research (2019), 153–168.
Dagfinn Rømen and Dag Svanæs. 2012. Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users. Universal Access in the Information Society 11, 4 (2012), 375–385.
Statista. 2022. Market share of mobile operating systems in Brazil from January 2019 to August 2022. Retrieved 20 May 2023 from [link].
Statista. 2022. Mobile OS share in North America 2018-2023. Retrieved 20 May 2023 from [link].
Beat Vollenwyder, Serge Petralito, Glena H Iten, Florian Brühlmann, Klaus Opwis, and Elisa D Mekler. 2023. How compliance with web accessibility standards shapes the experiences of users with and without disabilities. International Journal of Human-Computer Studies 170 (2023), 102956.
W3C. 1999. Web Content Accessibility Guidelines 1.0, Web Accessibility Initiative. Retrieved January 31, 2023 from [link]. .
W3C. 2008. W3C – World Wide Web Consortium. Web Content Accessibility Guidelines (WCAG) 2.0. Retrieved January 31, 2023 from [link].
W3C. 2018. W3C. Web Content Accessibility Guidelines (WCAG) 2.1. Retrieved January 31, 2023 from [link].
W3C. 2020. W3C. Web Content Accessibility Guidelines (WCAG) 2.2. Retrieved January 31, 2023 from [link].
W3C. 2021. W3C. Web Content Accessibility Guidelines (WCAG) 3.0. Retrieved January 31, 2023 from [link].
Publicado
16/10/2023
Como Citar
MATEUS, Delvani Antônio; SOUZA, Maurício Ronny De Almeida; FREIRE, André Pimenta.
Accessibility of Mobile Apps for Visually Impaired Users: Problems Encountered by User Evaluation, Inspections and Automated Tools. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
