Can AI Judge Usability? A Comparative Analysis of Generative Tools on Climate Conference Websites
Abstract
Introduction: This paper analyzes the usability of COP30 (Brazil) and COP29 (Azerbaijan) websites using four Generative Artificial Intelligences (GAIs): ChatGPT, DeepSeek, Gemini, and Microsoft 365 Copilot. Objective: To examine the role of GAIs as support tools in usability evaluations. Methodology: Following ISO 9241-210, each GAI received a standardized prompt, and outputs were analyzed through Directed Categorical Content Analysis. Results: GAIs identified issues such as deep navigation, absent accessibility mechanisms, and terminological inconsistencies, but showed limits in cultural sensitivity and technical accessibility. The study highlights opportunities and cautions for their application in real projects.
Keywords:
Usability, Generative Artifcial Intelligence, Human-Centered Design, COP Conferences
References
Alsadi, A. e Miller, D. (2023). Exploring the impact of artificial intelligence language model chatgpt on the user experience. International Journal of Technology, Innovation and Management (IJTIM), 1:2023.
Araújo, M. F. B. d., Mota, M. P., e Seruffo, M. C. d. R. (2024). Combinando inteligência artificial generativa e inspeção humana: Uma análise da usabilidade do site da greenpeace brasil. In Anais Estendidos do XXIII Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (IHC), pages 81–85. Sociedade Brasileira de Computação.
Bardin, L. (2011). Análise de conteúdo. ed. Revista e Ampliada. São Paulo: Edições, 70.
Bevan, N., Carter, J., e Harker, S. (2015). Iso 9241-11 revised: What have we learnt about usability since 1998? In Human-Computer Interaction: Design and Evaluation: 17th International Conference, HCI International 2015, Los Angeles, CA, USA, August 2-7, 2015, Proceedings, Part I 17, pages 143–151. Springer.
Bleichner, A. e Hermansson, N. (2023). Investigating the usefulness of a generative ai when designing user interfaces. Master’s thesis, Uppsala University.
Borges, J. M. e Araújo, R. D. (2024). Experiences and challenges of a redesign process with the support of an ai assistant on an educational platform. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, IHC ’24, New York, NY, USA. Association for Computing Machinery.
Brazilian Federal Government (2024). COP 30 in Brazil. [link]. Accessed: April 25, 2025.
Brooke, J. (1996). Sus: A ’quick and dirty’ usability scale. In Jordan, P. W., Thomas, B., Weerdmeester, B. A., e McClelland, I. L., editors, Usability Evaluation in Industry, pages 189–194. Taylor & Francis, London.
Castells, M. (2008). The new public sphere: Global civil society, communication networks, and global governance. In The aNNalS of the american academy of Political and Social Science, pages 78–93. Publishing Press.
Cyr, D., Head, M., e Ivanov, A. (2006). Design aesthetics leading to m-loyalty in mobile commerce. Information & management, 43(8):950–963.
Fischer, M. e Lanquillon, C. (2024). Evaluation of generative ai-assisted software design and engineering: A user-centered approach. In Degen, H. e Ntoa, S., editors, Artifcial Intelligence in HCI, pages 31–47, Cham. Springer Nature Switzerland.
Hassenzahl, M. e Tractinsky, N. (2006). User experience-a research agenda. Behaviour & information technology, 25(2):91–97.
International Organization for Standardization (2010). ISO 9241-210: Ergonomics of human-system interaction – Part 210: Human-centred design for interactive systems. [link]. Accessed: 2025-04-24.
Ivory, M. Y. e Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv., 33(4):470–516.
Kuric, E., Demcak, P., Krajcovic, M., e Lang, J. (2025). Systematic literature review of automation and artificial intelligence in usability issue detection.
Li, D. (2024). Exploration of user experience design optimization for the campus library information management system. Journal of Education, Humanities and Social Sciences, 37:6–15.
Liu, F. (2021). International usability testing: Why you need it. [link]. Accessed: 2025-04-24.
Miraz, M. H., Ali, M., e Excell, P. S. (2017). Multilingual website usability analysis based on an international user survey. CoRR, abs/1708.05085.
Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J. e Mack, R. L., editors, Usability Inspection Methods, pages 25–62. John Wiley & Sons, New York.
Nielsen, J. (1999). Designing web usability: The practice of simplicity. New riders publishing.
Norman Donald, A. (2013). The design of everyday things. MIT Press.
Oliveira, L. F. P. d. e Ferreira, S. B. L. (2022). Usabilidade e acessibilidade: Um estudo de caso com a plataforma instagram. [link]. Trabalho de Conclusão de Curso, Universidade Federal do Estado do Rio de Janeiro (UNIRIO).
Pereira, R., Darin, T., e Silveira, M. S. (2024). Grandihc-br: Grand research challenges in human-computer interaction in brazil for 2025-2035. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, pages 1–24.
Serra, L., Carvalho, L., Ferreira, L., Vaz, J., e Freire, A. (2015). Accessibility evaluation of e-government mobile applications in brazil. Procedia Computer Science, 67:348–357.
Shneiderman, B. e Plaisant, C. (2010). Designing the user interface: strategies for effective human-computer interaction. Pearson Education India.
Vatrapu, R. e Pérez-Quiñones, M. A. (2004). Culture and international usability testing: The effects of culture in structured interviews. CoRR, cs.HC/0405045.
Araújo, M. F. B. d., Mota, M. P., e Seruffo, M. C. d. R. (2024). Combinando inteligência artificial generativa e inspeção humana: Uma análise da usabilidade do site da greenpeace brasil. In Anais Estendidos do XXIII Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (IHC), pages 81–85. Sociedade Brasileira de Computação.
Bardin, L. (2011). Análise de conteúdo. ed. Revista e Ampliada. São Paulo: Edições, 70.
Bevan, N., Carter, J., e Harker, S. (2015). Iso 9241-11 revised: What have we learnt about usability since 1998? In Human-Computer Interaction: Design and Evaluation: 17th International Conference, HCI International 2015, Los Angeles, CA, USA, August 2-7, 2015, Proceedings, Part I 17, pages 143–151. Springer.
Bleichner, A. e Hermansson, N. (2023). Investigating the usefulness of a generative ai when designing user interfaces. Master’s thesis, Uppsala University.
Borges, J. M. e Araújo, R. D. (2024). Experiences and challenges of a redesign process with the support of an ai assistant on an educational platform. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, IHC ’24, New York, NY, USA. Association for Computing Machinery.
Brazilian Federal Government (2024). COP 30 in Brazil. [link]. Accessed: April 25, 2025.
Brooke, J. (1996). Sus: A ’quick and dirty’ usability scale. In Jordan, P. W., Thomas, B., Weerdmeester, B. A., e McClelland, I. L., editors, Usability Evaluation in Industry, pages 189–194. Taylor & Francis, London.
Castells, M. (2008). The new public sphere: Global civil society, communication networks, and global governance. In The aNNalS of the american academy of Political and Social Science, pages 78–93. Publishing Press.
Cyr, D., Head, M., e Ivanov, A. (2006). Design aesthetics leading to m-loyalty in mobile commerce. Information & management, 43(8):950–963.
Fischer, M. e Lanquillon, C. (2024). Evaluation of generative ai-assisted software design and engineering: A user-centered approach. In Degen, H. e Ntoa, S., editors, Artifcial Intelligence in HCI, pages 31–47, Cham. Springer Nature Switzerland.
Hassenzahl, M. e Tractinsky, N. (2006). User experience-a research agenda. Behaviour & information technology, 25(2):91–97.
International Organization for Standardization (2010). ISO 9241-210: Ergonomics of human-system interaction – Part 210: Human-centred design for interactive systems. [link]. Accessed: 2025-04-24.
Ivory, M. Y. e Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv., 33(4):470–516.
Kuric, E., Demcak, P., Krajcovic, M., e Lang, J. (2025). Systematic literature review of automation and artificial intelligence in usability issue detection.
Li, D. (2024). Exploration of user experience design optimization for the campus library information management system. Journal of Education, Humanities and Social Sciences, 37:6–15.
Liu, F. (2021). International usability testing: Why you need it. [link]. Accessed: 2025-04-24.
Miraz, M. H., Ali, M., e Excell, P. S. (2017). Multilingual website usability analysis based on an international user survey. CoRR, abs/1708.05085.
Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J. e Mack, R. L., editors, Usability Inspection Methods, pages 25–62. John Wiley & Sons, New York.
Nielsen, J. (1999). Designing web usability: The practice of simplicity. New riders publishing.
Norman Donald, A. (2013). The design of everyday things. MIT Press.
Oliveira, L. F. P. d. e Ferreira, S. B. L. (2022). Usabilidade e acessibilidade: Um estudo de caso com a plataforma instagram. [link]. Trabalho de Conclusão de Curso, Universidade Federal do Estado do Rio de Janeiro (UNIRIO).
Pereira, R., Darin, T., e Silveira, M. S. (2024). Grandihc-br: Grand research challenges in human-computer interaction in brazil for 2025-2035. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, pages 1–24.
Serra, L., Carvalho, L., Ferreira, L., Vaz, J., e Freire, A. (2015). Accessibility evaluation of e-government mobile applications in brazil. Procedia Computer Science, 67:348–357.
Shneiderman, B. e Plaisant, C. (2010). Designing the user interface: strategies for effective human-computer interaction. Pearson Education India.
Vatrapu, R. e Pérez-Quiñones, M. A. (2004). Culture and international usability testing: The effects of culture in structured interviews. CoRR, cs.HC/0405045.
Published
2025-09-08
How to Cite
LIMA, Danilo Teixeira; MEDEIROS, Alana M.; PAULINO, Rita de Cássia R.; SERUFFO, Marcos César R..
Can AI Judge Usability? A Comparative Analysis of Generative Tools on Climate Conference Websites. In: BRAZILIAN SYMPOSIUM ON HUMAN FACTORS IN COMPUTATIONAL SYSTEMS (IHC), 24. , 2025, Belo Horizonte/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 497-517.
DOI: https://doi.org/10.5753/ihc.2025.10805.
