Um Estudo Exploratório sobre Métodos de Avaliação de User Experience em Chatbots
Resumo
As empresas investem cada vez mais em projetar, desenvolver e avaliar agentes conversacionais baseados em texto. Apesar do mercado potencial, ainda há pouco conhecimento para avaliar a qualidade dos chatbots do ponto de vista da experiência do usuário (UX). O objetivo deste estudo é investigar a aplicabilidade, viabilidade de uso e aceitação de três métodos de avaliação de UX para avaliar chatbots (AttrakDiff, Think Aloud e MAX). Para isso, foi conduzido um estudo exploratório para avaliar a UX do chatbot chamado ANA. Com base nos resultados, acredita-se que a combinação dos três métodos de UX e fundamental para capturar toda a experiência do usuário, enquanto estes interagem com chatbot.
Referências
Alhadreti, Obead e Mayhew, P. (2018). Rethinking thinking aloud: A comparison of three think-aloud protocols. In CHI, pages 1–12.
Barbosa, M., Nakamura, W. T., Valle, P., Guerino, G. C., Finger, A. F., Lunardi, G. M., and Silva, W. (2022). Ux of chatbots: An exploratory study on acceptance of user experience evaluation methods. In ICEIS (2), pages 355–363.
Cavalcante, Emanuelle e Rivero, L. e. C. T. (2015). Max: A method for evaluating the post-use user experience through cards and a board. In SEKE 2015, pages 495–500.
Fernandes, U. d. S., Prates, R. O., Chagas, B. A., and Barbosa, G. A. (2021). Analyzing molic’s applicability to model the interaction of conversational agents: A case study on ana chatbot. In IHC 2021, pages 1–7.
Fiore, Dario e Baldauf, M. e. T. C. (2019). “forgot your password again?” acceptance and user experience of a chatbot for in-company it support. In MUM 2019, pages 1–11
Følstad, A. and Skjuve, M. (2019). Chatbots for customer service: user experience and motivation. In CUI 2019, pages 1–9.
Guerino, G. C., Silva, W. A. F., Coleti, T. A., and Valentim, N. M. C. (2021). Assessing a technology for usability and user experience evaluation of conversational systems: An exploratory study. In ICEIS 2021, volume 2, pages 461–471.
Hassenzahl, Marc e Burmester, M. e. K. F. (2003). Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. In Mensch & computer 2003, pages 187–196. Springer.
ISO 9241-210 (2011). ISO/IEC 9241-210: Ergonomics of human-system interaction – part 210: Human-centred design for interactive systems.
Jain, M., Kumar, P., Kota, R., and Patel, S. N. (2018). Evaluating and informing the design of chatbots. In Designing Interactive Systems Conference, pages 895–906.
Jaspers, M. W., Steen, T., Van Den Bos, C., and Geenen, M. (2004). The think aloud method: a guide to user interface design. Int. Journal of Medical Informatics, 73(11-12):781–795.
Lewis, James R e Sauro, J. (2021). Usability and user experience: Design and evaluation. Handbook of Human Factors and Ergonomics, pages 972–1015.
Luger, E. and Sellen, A. (2016). “like having a really bad pa” the gulf between user expectation and experience of conversational agents. In CHI 2016, pages 5286–5297.
Marques, Leonardo C e Nakamura, W. T. e. V. N. M. C. e. R. L. e. C. T. (2018). Do scale type techniques identify problems that affect user experience? user experience evaluation of a mobile application (s). In SEKE, pages 451–450.
Nielsen, J. (2000). Why you only need to test with 5 users. Disponível em: [link], Acessado em: 06/09/21.
Rapp, A., Curti, L., and Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, page 102630.
Rivero, L. and Conte, T. (2017). A systematic mapping study on research contributions on ux evaluation technologies. In IHC 2017, pages 1–10.
Sivaji, A. and Tzuaan, S. S. (2012). Website user experience (ux) testing tool development using open source software (oss). In SEANES, pages 1–6.
Smestad, Tuva Lunde e Volden, F. (2018). Chatbot personalities matters. In International Conference on Internet Science, pages 170–181. Springer.
Valentim, N. M. C., Rabelo, J., Silva, W., Coutinho, W., Mota, Á., and Conte, T. (2014). Avaliando a qualidade de um aplicativo web móvel através de um teste de usabilidade: um relato de experiência. In SBQS 2015, pages 256–263. SBC.
Barbosa, M., Nakamura, W. T., Valle, P., Guerino, G. C., Finger, A. F., Lunardi, G. M., and Silva, W. (2022). Ux of chatbots: An exploratory study on acceptance of user experience evaluation methods. In ICEIS (2), pages 355–363.
Cavalcante, Emanuelle e Rivero, L. e. C. T. (2015). Max: A method for evaluating the post-use user experience through cards and a board. In SEKE 2015, pages 495–500.
Fernandes, U. d. S., Prates, R. O., Chagas, B. A., and Barbosa, G. A. (2021). Analyzing molic’s applicability to model the interaction of conversational agents: A case study on ana chatbot. In IHC 2021, pages 1–7.
Fiore, Dario e Baldauf, M. e. T. C. (2019). “forgot your password again?” acceptance and user experience of a chatbot for in-company it support. In MUM 2019, pages 1–11
Følstad, A. and Skjuve, M. (2019). Chatbots for customer service: user experience and motivation. In CUI 2019, pages 1–9.
Guerino, G. C., Silva, W. A. F., Coleti, T. A., and Valentim, N. M. C. (2021). Assessing a technology for usability and user experience evaluation of conversational systems: An exploratory study. In ICEIS 2021, volume 2, pages 461–471.
Hassenzahl, Marc e Burmester, M. e. K. F. (2003). Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. In Mensch & computer 2003, pages 187–196. Springer.
ISO 9241-210 (2011). ISO/IEC 9241-210: Ergonomics of human-system interaction – part 210: Human-centred design for interactive systems.
Jain, M., Kumar, P., Kota, R., and Patel, S. N. (2018). Evaluating and informing the design of chatbots. In Designing Interactive Systems Conference, pages 895–906.
Jaspers, M. W., Steen, T., Van Den Bos, C., and Geenen, M. (2004). The think aloud method: a guide to user interface design. Int. Journal of Medical Informatics, 73(11-12):781–795.
Lewis, James R e Sauro, J. (2021). Usability and user experience: Design and evaluation. Handbook of Human Factors and Ergonomics, pages 972–1015.
Luger, E. and Sellen, A. (2016). “like having a really bad pa” the gulf between user expectation and experience of conversational agents. In CHI 2016, pages 5286–5297.
Marques, Leonardo C e Nakamura, W. T. e. V. N. M. C. e. R. L. e. C. T. (2018). Do scale type techniques identify problems that affect user experience? user experience evaluation of a mobile application (s). In SEKE, pages 451–450.
Nielsen, J. (2000). Why you only need to test with 5 users. Disponível em: [link], Acessado em: 06/09/21.
Rapp, A., Curti, L., and Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, page 102630.
Rivero, L. and Conte, T. (2017). A systematic mapping study on research contributions on ux evaluation technologies. In IHC 2017, pages 1–10.
Sivaji, A. and Tzuaan, S. S. (2012). Website user experience (ux) testing tool development using open source software (oss). In SEANES, pages 1–6.
Smestad, Tuva Lunde e Volden, F. (2018). Chatbot personalities matters. In International Conference on Internet Science, pages 170–181. Springer.
Valentim, N. M. C., Rabelo, J., Silva, W., Coutinho, W., Mota, Á., and Conte, T. (2014). Avaliando a qualidade de um aplicativo web móvel através de um teste de usabilidade: um relato de experiência. In SBQS 2015, pages 256–263. SBC.
Publicado
18/10/2022
Como Citar
BARBOSA, Marcus; VALLE, Pedro; NAKAMURA, Walter; GUERINO, Guilherme; FINGER, Alice; LUNARDI, Gabriel; SILVA, Williamson.
Um Estudo Exploratório sobre Métodos de Avaliação de User Experience em Chatbots. In: ESCOLA REGIONAL DE ENGENHARIA DE SOFTWARE (ERES), 6. , 2022, Blumenau.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 21-30.
DOI: https://doi.org/10.5753/eres.2022.227723.
