A Method for Evaluating Usability Mechanisms in Software Systems
Resumo
Usability is an essential quality in software, as it allows straightforward and intuitive interactions between the user and the system. Consequently, software developers must prioritize addressing usability issues to ensure the system is easy to use. Thus, developing a usability evaluation method that also functions as a learning tool for students through hands-on evaluation practice could be a helpful resource. This document proposes a method for evaluating usability mechanisms in software systems. The method includes both an evaluation process and a questionnaire. These were validated through a case study, and their guiding procedures were applied to evaluate two serious digital games and a web application. Then, interviews with the software evaluators were conducted to gather insights into their perceptions of the method's effectiveness. Although there are some reservations about using case studies for evaluations, the results indicate that our evaluation method effectively identifies usability problems in software. In addition, it can serve as a valuable learning resource, as the evaluation process leads to discussions among the evaluators about the implementation of usability mechanisms in software, thereby enhancing their understanding of these concepts.
Palavras-chave:
Evaluation, usability mechanisms, serious games, web systems
Referências
Ailamar A. Guimarães, André Barros de Sales, Bruna Almeida Santos, and Eduardo Gabriel Queiroz Palmeira. 2022. Avaliação de Características de Usabilidade em Jogos Sérios em Interação Humano-Computador. Sociedade Brasileira de Computacao SB, 505–516. DOI: 10.5753/sbie.2022.225155
Alexandra De Raadt, Matthijs J. Warrens, Roel J. Bosker, and Henk A.L. Kiers. 2019. Kappa Coefficients for Missing Data. Educational and Psychological Measurement 79 (2019). Issue 3. DOI: 10.1177/0013164418823249
André Barros de Sales, Bruna Almeida Santos, and Eduardo Gabriel Queiroz Palmeira. 2022. Feedback Mechanisms in Human-Computer Interaction Serious Games. In Iberian Conference on Information Systems and Technologies, CISTI, Vol. 2022-June. IEEE Computer Society. DOI: 10.23919/CISTI54924.2022.9820368
Andréa Pereira Mendonça, Evandro de Barros Costa, and Dalton Dario Serey Guerrero. 2008. Elicitação de Requisitos Evidências de uma Problemática na Formação dos Estudantes de Computação. Fórum de Educação em Engenharia de Software (2008).
Antonio Carlos Gil. 2002. Como elaborar projetos de pesquisa. Vol. 5. Editora Atlas, São Paulo, SP.
Caio Almeida Abreu, Jean Clemisson Santos Rosa, and Ecivaldo De Souza Matos. 2018. Usabilidade de aplicativos móveis educacionais infantis: design e avaliação de interação do Fantastic Pirates. RENOTE 16 (2018). Issue 1. DOI: 10.22456/1679-1916.86042
Cauê Mateus Oliveira and Lucas Midlhey Cardoso Naves. 2022. Universidade das Heurísticas: Um jogo para o apoio a aprendizagem das heurísticas de Nielsen.
Clodis Boscarioli, Milene Silveira, Raquel Prates, Sílvia Bim, and Simone Barbosa. 2014. Charting the Landscape of HCI Education in Brazil. In Human-Computer Interaction. Theories, Methods, and Tools. 177–186. DOI: 10.1007/978-3-319-07233-3_17
Fábio Avigo de Castro Pinto , Anarosa Alves Franco Brandão, and Fábio Levy Siqueira. 2022. Design Thinking and Non-Functional Requirements Elicitation: A Survey. In Workshop de Engenharia de Requisitos.
Florian Huber, Tobias Eigler, Georg Hagel, and Christian Wolff. 2023. From Difficulties to Functional Requirements Deriving Requirements from Literature about Tool-supported Teaching of UML Diagrams in Software Engineering Education. In Proceedings of the 5th European Conference on Software Engineering Education (Seeon/Bavaria, Germany) (ECSEE ’23). Association for Computing Machinery, New York, NY, USA, 184–188. DOI: 10.1145/3593663.3593672
Francisco Heronildo Sousa Santos and João Vitor Ferreira Alves. 2022. Avaliação e Melhoria de aplicação WEB utilizando Mecanismos de Usabilidade.
Francy Diomar Rodríguez Tibocha. 2014. Obtención y Uso de Patrones para la Implementación de Funcionalidades de Usabilidad en Aplicaciones Web. Ph. D. Dissertation. Universidad Politécnica de Madrid. DOI: 10.20868/UPM.thesis.33528
Gabriel de Jesus Carvalho. 2022. IA-QVT: Ferramenta para coleta e tratamento de dados sobre qualidade de vida no trabalho, com foco em usabilidade e acessibilidade.
Giani Petri, Christiane Gresse Von Wangenheim, and Adriano Ferreti Borgatto. 2019. MEEGA+: Um Modelo para a Avaliação de Jogos Educacionais para o ensino de Computação. Revista Brasileira de Informática na Educação 27 (2019). Issue 03.
ISO 9241-11:2018 2018. Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. Standard. International Organization for Standardization.
Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37–46. DOI: 10.1177/001316446002000104
J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174. DOI: 10.2307/2529310
Jakob Nielsen. 1992. In Finding Usability Problems through Heuristic Evaluation (CHI ’92). Association for Computing Machinery, New York, NY, USA, 373–380. DOI: 10.1145/142750.142834
Jakob Nielsen. 1994. Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco, CA.
Jakob Nielsen. 1994. Usability inspection methods. Conference on Human Factors in Computing Systems Proceedings 1994-April. DOI: 10.1145/259963.260531
Jakob Nielsen. 2012. Usability 101: Introduction to Usability. [link]. Acesso em: 19-12-2022.
Jakob Nielsen and Rolf Molich. 1990. Heuristic Evaluation of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’90). Association for Computing Machinery, New York, NY, USA, 249–256. DOI: 10.1145/97243.97281
José Valderlei Da Silva and Filipe Roseiro Côgo. 2014. A Flexibilização Curricular da Disciplina Interação Humano Computador (IHC) para Enfrentar Desafios Emergentes. IHC ’14: Proceedings of the 13th Brazilian Symposium on Human Factors in Computing Systems (2014). [link]
Magda Maria Ventura. 2007. O estudo de caso como modalidade de pesquisa. Revista SoCERJ 20, 5 (2007), 383–386.
Maria Amelia Eliseo, Bárbara Soares Casac, and Gustavo Rorato Gentil. 2017. A comparative study of video content user interfaces based on heuristic evaluation. In 2017 12th Iberian Conference on Information Systems and Technologies (CISTI). 1–6. DOI: 10.23919/CISTI.2017.7975820
Matheus Souza, Pedro Oliveira, Rafael Ribeiro, Ricardo Britto, and Pedro Santos Neto. 2015. UseSkill: uma ferramenta de apoio a avaliação de usabilidade de sistemas Web. In Anais do XIV Simpósio Brasileiro de Qualidade de Software (Manaus). SBC, Porto Alegre, RS, Brasil, 180–194. DOI: 10.5753/sbqs.2015.15222
Natalia Juristo, Ana Moreno, and Maria-Isabel Sanchez-Segura. 2006. Usability Elicitation Patterns (USEPs). [link]. Acesso em: 27-05-2022.
Natalia Juristo, Ana Maria Moreno, and Maria-Isabel Sanchez-Segura. 2007. Guidelines for eliciting usability functionalities. IEEE Transactions on Software Engineering 33 (2007). Issue 11. DOI: 10.1109/TSE.2007.70741
Pablo Moreno-Ger, Javier Torrente, Yichuan Grace Hsieh, and William T. Lester. 2012. Usability testing for serious games: Making informed design decisions with user data. Advances in Human-Computer Interaction 2012 (2012). DOI: 10.1155/2012/369637
Pearl Brereton, Barbara Kitchenham, David Budgen, and Zhi Li. 2008. Using a protocol template for case study planning. Proceedings of EASE 2008 (01 2008).
Rafael Savi, Christiane Gresse von Wangenheim, and Adriano Ferreti Borgatto. 2011. Um Modelo de Avaliação de Jogos Educacionais na Engenharia de Software. 25th Brazilian Symposium on Software Engineering.
Rossicler Rodrigues Pires Júnior. 2021. PersonaDesignGame (PDG): Um Jogo Educacional sobre Personas.
Simone Bacellar Leal Ferreira and Ricardo Nunes Rodrigues. 2008. E-Usabilidade. Grupo Gen-LTC, São Paulo, SP.
Simone Diniz Junqueira Barbosa and Bruno Santana da Silva. 2010. Interação Humano-Computador. Elsevier Editora Ltda., Rio de Janeiro, RJ.
Simone Diniz Junqueira Barbosa, Silvia Amélia Bim, Clodis Boscarioli, and Milene S. Silveira. 2014. Desafios do ensinar para enfrentar os desafios de IHC. In Proceedings of the 13th Brazilian Symposium on Human Factors in Computing Systems (Foz do Iguaçu, Brazil) (IHC ’14). Sociedade Brasileira de Computação, BRA, 461–464.
Tatiana Engel Gerhardt and Denise Tolfo Silveira. 2009. Métodos de Pesquisa. UFRGS Editora, Rio Grande do Sul, RS.
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3 (2006), 77–101. DOI: 10.1191/1478088706qp063oa
Alexandra De Raadt, Matthijs J. Warrens, Roel J. Bosker, and Henk A.L. Kiers. 2019. Kappa Coefficients for Missing Data. Educational and Psychological Measurement 79 (2019). Issue 3. DOI: 10.1177/0013164418823249
André Barros de Sales, Bruna Almeida Santos, and Eduardo Gabriel Queiroz Palmeira. 2022. Feedback Mechanisms in Human-Computer Interaction Serious Games. In Iberian Conference on Information Systems and Technologies, CISTI, Vol. 2022-June. IEEE Computer Society. DOI: 10.23919/CISTI54924.2022.9820368
Andréa Pereira Mendonça, Evandro de Barros Costa, and Dalton Dario Serey Guerrero. 2008. Elicitação de Requisitos Evidências de uma Problemática na Formação dos Estudantes de Computação. Fórum de Educação em Engenharia de Software (2008).
Antonio Carlos Gil. 2002. Como elaborar projetos de pesquisa. Vol. 5. Editora Atlas, São Paulo, SP.
Caio Almeida Abreu, Jean Clemisson Santos Rosa, and Ecivaldo De Souza Matos. 2018. Usabilidade de aplicativos móveis educacionais infantis: design e avaliação de interação do Fantastic Pirates. RENOTE 16 (2018). Issue 1. DOI: 10.22456/1679-1916.86042
Cauê Mateus Oliveira and Lucas Midlhey Cardoso Naves. 2022. Universidade das Heurísticas: Um jogo para o apoio a aprendizagem das heurísticas de Nielsen.
Clodis Boscarioli, Milene Silveira, Raquel Prates, Sílvia Bim, and Simone Barbosa. 2014. Charting the Landscape of HCI Education in Brazil. In Human-Computer Interaction. Theories, Methods, and Tools. 177–186. DOI: 10.1007/978-3-319-07233-3_17
Fábio Avigo de Castro Pinto , Anarosa Alves Franco Brandão, and Fábio Levy Siqueira. 2022. Design Thinking and Non-Functional Requirements Elicitation: A Survey. In Workshop de Engenharia de Requisitos.
Florian Huber, Tobias Eigler, Georg Hagel, and Christian Wolff. 2023. From Difficulties to Functional Requirements Deriving Requirements from Literature about Tool-supported Teaching of UML Diagrams in Software Engineering Education. In Proceedings of the 5th European Conference on Software Engineering Education (Seeon/Bavaria, Germany) (ECSEE ’23). Association for Computing Machinery, New York, NY, USA, 184–188. DOI: 10.1145/3593663.3593672
Francisco Heronildo Sousa Santos and João Vitor Ferreira Alves. 2022. Avaliação e Melhoria de aplicação WEB utilizando Mecanismos de Usabilidade.
Francy Diomar Rodríguez Tibocha. 2014. Obtención y Uso de Patrones para la Implementación de Funcionalidades de Usabilidad en Aplicaciones Web. Ph. D. Dissertation. Universidad Politécnica de Madrid. DOI: 10.20868/UPM.thesis.33528
Gabriel de Jesus Carvalho. 2022. IA-QVT: Ferramenta para coleta e tratamento de dados sobre qualidade de vida no trabalho, com foco em usabilidade e acessibilidade.
Giani Petri, Christiane Gresse Von Wangenheim, and Adriano Ferreti Borgatto. 2019. MEEGA+: Um Modelo para a Avaliação de Jogos Educacionais para o ensino de Computação. Revista Brasileira de Informática na Educação 27 (2019). Issue 03.
ISO 9241-11:2018 2018. Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. Standard. International Organization for Standardization.
Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37–46. DOI: 10.1177/001316446002000104
J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174. DOI: 10.2307/2529310
Jakob Nielsen. 1992. In Finding Usability Problems through Heuristic Evaluation (CHI ’92). Association for Computing Machinery, New York, NY, USA, 373–380. DOI: 10.1145/142750.142834
Jakob Nielsen. 1994. Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco, CA.
Jakob Nielsen. 1994. Usability inspection methods. Conference on Human Factors in Computing Systems Proceedings 1994-April. DOI: 10.1145/259963.260531
Jakob Nielsen. 2012. Usability 101: Introduction to Usability. [link]. Acesso em: 19-12-2022.
Jakob Nielsen and Rolf Molich. 1990. Heuristic Evaluation of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’90). Association for Computing Machinery, New York, NY, USA, 249–256. DOI: 10.1145/97243.97281
José Valderlei Da Silva and Filipe Roseiro Côgo. 2014. A Flexibilização Curricular da Disciplina Interação Humano Computador (IHC) para Enfrentar Desafios Emergentes. IHC ’14: Proceedings of the 13th Brazilian Symposium on Human Factors in Computing Systems (2014). [link]
Magda Maria Ventura. 2007. O estudo de caso como modalidade de pesquisa. Revista SoCERJ 20, 5 (2007), 383–386.
Maria Amelia Eliseo, Bárbara Soares Casac, and Gustavo Rorato Gentil. 2017. A comparative study of video content user interfaces based on heuristic evaluation. In 2017 12th Iberian Conference on Information Systems and Technologies (CISTI). 1–6. DOI: 10.23919/CISTI.2017.7975820
Matheus Souza, Pedro Oliveira, Rafael Ribeiro, Ricardo Britto, and Pedro Santos Neto. 2015. UseSkill: uma ferramenta de apoio a avaliação de usabilidade de sistemas Web. In Anais do XIV Simpósio Brasileiro de Qualidade de Software (Manaus). SBC, Porto Alegre, RS, Brasil, 180–194. DOI: 10.5753/sbqs.2015.15222
Natalia Juristo, Ana Moreno, and Maria-Isabel Sanchez-Segura. 2006. Usability Elicitation Patterns (USEPs). [link]. Acesso em: 27-05-2022.
Natalia Juristo, Ana Maria Moreno, and Maria-Isabel Sanchez-Segura. 2007. Guidelines for eliciting usability functionalities. IEEE Transactions on Software Engineering 33 (2007). Issue 11. DOI: 10.1109/TSE.2007.70741
Pablo Moreno-Ger, Javier Torrente, Yichuan Grace Hsieh, and William T. Lester. 2012. Usability testing for serious games: Making informed design decisions with user data. Advances in Human-Computer Interaction 2012 (2012). DOI: 10.1155/2012/369637
Pearl Brereton, Barbara Kitchenham, David Budgen, and Zhi Li. 2008. Using a protocol template for case study planning. Proceedings of EASE 2008 (01 2008).
Rafael Savi, Christiane Gresse von Wangenheim, and Adriano Ferreti Borgatto. 2011. Um Modelo de Avaliação de Jogos Educacionais na Engenharia de Software. 25th Brazilian Symposium on Software Engineering.
Rossicler Rodrigues Pires Júnior. 2021. PersonaDesignGame (PDG): Um Jogo Educacional sobre Personas.
Simone Bacellar Leal Ferreira and Ricardo Nunes Rodrigues. 2008. E-Usabilidade. Grupo Gen-LTC, São Paulo, SP.
Simone Diniz Junqueira Barbosa and Bruno Santana da Silva. 2010. Interação Humano-Computador. Elsevier Editora Ltda., Rio de Janeiro, RJ.
Simone Diniz Junqueira Barbosa, Silvia Amélia Bim, Clodis Boscarioli, and Milene S. Silveira. 2014. Desafios do ensinar para enfrentar os desafios de IHC. In Proceedings of the 13th Brazilian Symposium on Human Factors in Computing Systems (Foz do Iguaçu, Brazil) (IHC ’14). Sociedade Brasileira de Computação, BRA, 461–464.
Tatiana Engel Gerhardt and Denise Tolfo Silveira. 2009. Métodos de Pesquisa. UFRGS Editora, Rio Grande do Sul, RS.
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3 (2006), 77–101. DOI: 10.1191/1478088706qp063oa
Publicado
07/11/2024
Como Citar
SANTOS, Bruna Almeida; PALMEIRA, Eduardo Gabriel Queiroz; MENDES, Fabiana Freitas; SALES, André Barros de.
A Method for Evaluating Usability Mechanisms in Software Systems. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 23. , 2024, Brasília/DF.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 236-246.
