Accessibility Inspections Using the Web Content Accessibility Guidelines by Novice Evaluators: an Experience Report

• André Pimenta Freire Federal University of Lavras https://orcid.org/0000-0001-7894-9740
• Paulo Afonso Parreira Junior Federal University of Lavras https://orcid.org/0000-0002-8877-2931
• Silvia Helena Rigatto Federal University of Lavras https://orcid.org/0000-0002-5937-5611
• Tiago Silva da Silva Federal University of São Paulo https://orcid.org/0000-0001-8459-7833
• Bruna Poletto Salton Federal Institute of Rio Grande do Sul https://orcid.org/0000-0003-4915-9883
• Lael Nervis Federal Institute of Rio Grande do Sul https://orcid.org/0000-0001-8425-7628
• Eduardo Augusto Ribeiro Federal University of Lavras https://orcid.org/0009-0002-8680-2487
• Gabriel Aguiar Alves e Silva Federal University of Lavras https://orcid.org/0009-0003-0495-4792
• Jamilly Danielle Antonio Federal University of Lavras https://orcid.org/0009-0000-0156-9715
• João Victor de Oliveira Rosa Federal University of Lavras https://orcid.org/0009-0006-5066-7243
• Julia Ribeiro Souza Federal University of Lavras https://orcid.org/0009-0006-7143-7981
• Paulo Henrique dos Anjos Silveira Federal University of Lavras https://orcid.org/0009-0001-2128-3738
• Ruan Pablo Gomes Rocha Federal University of Lavras https://orcid.org/0009-0009-8049-2407
• Saimon Gabriel de Andrade Federal University of Lavras https://orcid.org/0009-0009-3053-2837

Abstract

The use of Web Accessibility standards is fundamental as a first step in evaluating the accessibility of digital resources. The Web Content Accessibility Guidelines (WCAG) have become the leading worldwide reference for accessibility standards and are used by many governmental regulations in numerous countries. However, many studies have revealed difficulties novice evaluators encounter in conducting accessibility inspections using WCAG. In Brazil, there is ongoing work to adopt WCAG 2.2 as the basis of a national standard by the Brazilian Technical Standards Association (Associação Brasileira de Normas Técnicas - ABNT), which will include a checklist to simplify the use of the guidelines. This experience report describes the experiences of two groups of students engaged in an outreach digital accessibility project, aimed at providing detailed inspection reports to entities representing people with disabilities when filing official complaints concerning digital accessibility. The first group of students evaluated websites using the WCAG 2.2 documentation alone. The second group had their first experience conducting accessibility inspections using the checklist and guidance from the ABNT checklist under construction. The report provides insights into the main difficulties encountered by the students in both scenarios and the strategies they used to overcome them. The experience report provides a detailed account of the types of success criteria that presented the most difficulties and how the new checklist provided better guidance to them. The reflections in the report provide lessons learned concerning using the ABNT checklist and possibilities for improvements for both the checklist and future versions of WCAG.

Keywords: Accessibility inspections, Novice Evaluators, Web Content Accessibility Guidelines

References

Leonelo Dell Anhol Almeida and Maria Cecília Calani Baranauskas. 2012. Accessibility in rich internet applications: people and research. In Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems. 3–12.

Fernando Alonso, José L Fuertes, Ángel L González, and Loïc Martínez. 2010. Evaluating conformance to WCAG 2.0: open challenges. In Computers Helping People with Special Needs: 12th International Conference, ICCHP 2010, Vienna, Austria, July 14-16, 2010. Proceedings 12. Springer, 417–424.

Fernando Alonso, José Luis Fuertes, Ángel Lucas González, and Loïc Martínez. 2010. On the testability of WCAG 2.0 for beginners. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A). 1–9.

Christopher Bailey, Elaine Pearson, and Voula Gkatzidou. 2014. Measuring and comparing the reliability of the structured walkthrough evaluation method with novices and experts. In Proceedings of the 11th Web for All Conference. 1–10.

Giorgio Brajnik, Markel Vigo, Yeliz Yesilada, and Simon Harper. 2016. Group vs individual web accessibility evaluations: effects with novice evaluators. Interacting with Computers 28, 6 (2016), 843–861.

Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2010. Testability and validity of WCAG 2.0: the expertise effect. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. 43–50.

Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2011. The expertise effect on web accessibility evaluation methods. Human–Computer Interaction 26, 3 (2011), 246–283.

Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2012. Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0. ACM Transactions on Accessible Computing (TACCESS) 4, 2 (2012), 1–28.

Governo Brasileiro. 2010. e-MAG Checklist de Acessibilidade Manual para o Desenvolvedor. Disponível online em [link], acesso em 6 de junho de 2024

Governo Brasileiro. 2014. eMAG Modelo de Acessibilidade em Governo Eletrônico. Disponível em [link], acessado em 6 de junho de 2024

Ben Caldwell, Loretta Guarino Reid, and Gregg Vanderheiden. 2023. Web Content Accessibility Guidelines 2.0 (WCAG 2.0). Web Accessibility Initiative. World Wide Web Consortium (W3C). Disponível online em [link], acessado em 6 de junho de 2024.

Alastair Campbell, Chuck Adams, Rachael Bradley Montgomery, Michael Cooper, and Andrew Kirkpatrick. 2023. Web Content Accessibility Guidelines 2.2 (WCAG 2.2). Web Accessibility Initiative. World Wide Web Consortium (W3C). Disponível online em [link], acessado em 6 de junho de 2024.

Wendy Chisholm, Gregg Vanderheiden, and Ian Jacobs. 1999. Web Content Accessibility Guidelines 1.0 (WCAG 1.0). Web Accessibility Initiative. World Wide Web Consortium (W3C). Disponível online em [link], acessado em 6 de junho de 2024.

Leandro Agostini do Amaral, Renata Pontin de Mattos Fortes, and Thiago Jabur Bittar. 2018. A4U-an approach to evaluation considering accessibility and usability guidelines. In Proceedings of the 24th Brazilian Symposium on Multimedia and the Web. 295–298.

Francisco Iniesto and Covadonga Rodrigo. 2024. The use of WCAG and automatic tools by computer science students: a case study evaluating MOOC accessibility. Journal of Universal Computer Science (2024), In–Press.

Andrew Kirkpatrick, Joshue O Connor, Alastair Campbell, and Michael Cooper. 2023. Web Content Accessibility Guidelines 2.1 (WCAG 2.1). Web Accessibility Initiative. World Wide Web Consortium (W3C). Disponível online em [link], acessado em 6 de junho de 2024.

Jonathan Lazar. 2019. Web accessibility policy and law. Web Accessibility: A Foundation for Research (2019), 247–261.

Delvani Antônio Mateus, Simone Bacellar Leal Ferreira, Maurício Ronny de Almeida Souza, and André Pimenta Freire. 2023. Accessibility Inspections of Mobile Applications by Professionals with Different Expertise Levels: an Empirical Study Comparing with User Evaluations. In IFIP Conference on Human-Computer Interaction. Springer, 135–154.

Delvani Antônio Mateus, Carlos Alberto Silva, Arthur F. B. A. de Oliveira, Heitor Costa, and André Pimenta Freire. 2021. A Systematic Mapping of Accessibility Problems Encountered on Websites and Mobile Apps: A Comparison Between Automated Tests, Manual Inspections and User Evaluations. Journal on Interactive Systems 12, 1 (Nov. 2021), 145–171. DOI: 10.5753/jis.2021.1778

Delvani Antônio Mateus, Carlos Alberto Silva, Marcelo Medeiros Eler, and André Pimenta Freire. 2020. Accessibility of mobile applications: evaluation by users with visual impairment and by automated tools. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems. 1–10.

Delvani Antônio Mateus, Fábio Aparecido Cândido da Silva, Tiago Silva da Silva, and André Pimenta Freire. 2022. Evaluation methods in legal procedures concerning digital accessibility in Brazil: an analysis of cases investigated by the federal public ministry. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems. 1–12.

Elaine Pearson, Chrstopher Bailey, and Steve Green. 2011. A tool to support the web accessibility evaluation process for novices. In Proceedings of the 16th annual joint conference on Innovation and technology in computer science education. 28–32.

Talita Cristina Pagani Britto Pichiliani and Ednaldo Brigante Pizzolato. 2021. Cognitive disabilities and web accessibility: a survey into the Brazilian web development community. Journal on Interactive Systems 12, 1 (Dec. 2021), 308–327. DOI: 10.5753/jis.2021.982

Mara Taynar Santiago and Anna Beatriz Marques. 2023. Exploring user reviews to identify accessibility problems in applications for autistic users. Journal on Interactive Systems 14, 1 (Jul. 2023), 317–330. DOI: 10.5753/jis.2023.3238