What Do We Know About Usability Evaluation for Chatbots?: A Systematic Mapping Study

  • Lucas Vieira de Carvalho UNIPAMPA
  • Pedro Henrique Dias Valle UFJF
  • Bhruno Roan Leifheit UNIPAMPA
  • Luís Enrique Rivero Cabrejos UFMA
  • Walter Nakamura UTFPR
  • Guilherme Corredato Guerino UNESPAR
  • Renato De Souza Garcia UNIPAMPA
  • Williamson Silva UNIPAMPA

Resumo


Context: Given the current scenario of technological advances, it is undeniable that the software industry is increasingly investing in designing, developing, and evaluating text-based conversational agents called chatbots. Chatbots are rising because they provide efficient and scalable technological support and user interaction. Problem: Although there is a promising market for these technologies, there is still a need for more knowledge to evaluate the quality of chatbots from the usability perspective, which results in a lack of consensus on the best practices and evaluation criteria to be applied. Method: This paper aims to describe a Systematic Mapping Study (SMS) seeking to characterize studies that address usability evaluation technologies (methods, techniques, approaches, tools) designed to assist software professionals in improving chatbots’ quality from the usability perspective. IS Theory: Not applicable. Summary of Results: We identified 27 relevant studies, of which 4 are systematic literature reviews. Questionnaires, metrics, methods, and evaluation techniques are among the identified technologies. Contributions and Impact in the IS area: Based on these studies and others selected, it was possible to carry out a comprehensive analysis and characterization of the assessment technologies used, which helped in a more in-depth understanding, presenting valuable insights that not only fill gaps in the current knowledge of these systems but also provide direction in a careful choice of the most appropriate evaluation methods is significant.

Palavras-chave: Chatbots, Conversational Agents, Systematic Mapping Study, Usability Evaluation Methods
Publicado
20/05/2024
CARVALHO, Lucas Vieira de; VALLE, Pedro Henrique Dias; LEIFHEIT, Bhruno Roan; CABREJOS, Luís Enrique Rivero; NAKAMURA, Walter; GUERINO, Guilherme Corredato; GARCIA, Renato De Souza; SILVA, Williamson. What Do We Know About Usability Evaluation for Chatbots?: A Systematic Mapping Study. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 20. , 2024, Juiz de Fora/MG. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2024 .

Artigos mais lidos do(s) mesmo(s) autor(es)

<< < 1 2 3 4 > >>