What Do We Know About Usability Evaluation for Chatbots?: A Systematic Mapping Study
Resumo
Context: Given the current scenario of technological advances, it is undeniable that the software industry is increasingly investing in designing, developing, and evaluating text-based conversational agents called chatbots. Chatbots are rising because they provide efficient and scalable technological support and user interaction. Problem: Although there is a promising market for these technologies, there is still a need for more knowledge to evaluate the quality of chatbots from the usability perspective, which results in a lack of consensus on the best practices and evaluation criteria to be applied. Method: This paper aims to describe a Systematic Mapping Study (SMS) seeking to characterize studies that address usability evaluation technologies (methods, techniques, approaches, tools) designed to assist software professionals in improving chatbots’ quality from the usability perspective. IS Theory: Not applicable. Summary of Results: We identified 27 relevant studies, of which 4 are systematic literature reviews. Questionnaires, metrics, methods, and evaluation techniques are among the identified technologies. Contributions and Impact in the IS area: Based on these studies and others selected, it was possible to carry out a comprehensive analysis and characterization of the assessment technologies used, which helped in a more in-depth understanding, presenting valuable insights that not only fill gaps in the current knowledge of these systems but also provide direction in a careful choice of the most appropriate evaluation methods is significant.
