Accessibility of Mobile Apps for Visually Impaired Users: Problems Encountered by User Evaluation, Inspections and Automated Tools - Mateus, Delvani Antônio; Souza, Maurício Ronny De Almeida; Freire, André Pimenta (Anais do Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (IHC), 2023)
Mobile apps should be usable and accessible to everyone, including visually impaired people. Accessibility evaluation techniques are essential to identify accessibility problems and help to avoid problems that hinder access by people with disabilities. Different accessibility evaluation methods may be used in different stages of software development. However, knowing the types of accessibility problems encountered by each method is vital for developers, testers, and designers to allocate the different methods appropriately in different stages of the development cycle. This study aimed to compare the outcomes of automated evaluations, manual inspection and user evaluation methods applied to mobile apps focusing on visually impaired users. The study compared the results from different methods by comparing results from a systematic mapping of the literature and the analysis of user evaluations performed in a previous study with two mobile apps. The study compared the results from the usability evaluations of Saraiva and Receita Federal by nine users with visual impairment and 189 instances of problems in two applications. These results were compared to the automated evaluation of the two apps by the tools MATE (Mobile Accessibility Testing) and Accessibility Scanner. Next, the apps were inspected by two groups of professionals: 17 experts in different areas of software development (full-stack developers, testers and front-end developers) and ten specialists in Human-Computer Interaction (HCI) with previous experience with accessibility. The results showed a difference between the different accessibility evaluation methods. Inspections by HCI specialists and tests with users with disabilities found a greater diversity of types of problems. Automated tools showed limited performance in the detection of types of problems. In accessibility inspections, there is a difference between specialists in software development and specialists in HCI. The results showed a difference in the number of violations that specialists in software development find compared to HCI specialists. This study contributes to the understanding of how different methods can contribute to the accessibility evaluation of mobile apps, helping different stakeholders make decisions about when and how to apply different methods.