Investigating user experience with a customer service chatbot: a case with a telecommunications company’s software

• Ana Paula Pereira Lopes de Carvalho USP
• Leo Natan Paschoal USP

DOI: https://doi.org/10.5753/washes.2024.2635

Abstract

In the business sector, chatbots are widely employed to provide customer service. When customers cannot achieve a satisfactory experience with the service provided by the chatbot, they may become dissatisfied and even abandon the use of the company’s services. Therefore, the user experience with the chatbot is of paramount importance. This study examined the experience of 15 users who interacted with the customer service chatbot of one of the leading mobile phone operators in Brazil. The results of this study indicate a neutral evaluation of users’ experience when interacting with the chatbot, demonstrating that the chatbot did not hinder the service experience. Some opportunities for improving the interaction were identified and may provide support to chatbot developers.

References

Allouch, M., Azaria, A., and Azoulay, R. (2021). Conversational agents: Goals, technologies, vision and challenges. Sensors, 21(24):8448.

Barbosa, M., Nakamura, W. T., Valle, P., Guerino, G. C., Finger, A. F., Lunardi, G. M., and Silva, W. (2022). Ux of chatbots: An exploratory study on acceptance of user experience evaluation methods. In International Conference on Enterprise Information Systems, pages 355–363.

Basili, V. R., Caldeira, G., and Rombach, H. D. (1994). Goal question metric paradigm. Encyclopedia of software engineering, 1(528-532):6.

Cavalcante, E., Rivero, L., and Conte, T. (2015). Max: A method for evaluating the postuse user experience through cards and a board. In 27th International Conference on Software Engineering and Knowledge Engineering (SEKE 2015), pages 495–500.

Cheng, X., Bao, Y., Zarifis, A., Gong, W., and Mou, J. (2021). Exploring consumers’ response to text-based chatbots in e-commerce: the moderating role of task complexity and chatbot disclosure. Internet Research, 32(2):496–517.

Følstad, A. and Brandtzaeg, P. B. (2020). Users’ experiences with chatbots: findings from a questionnaire study. Quality and User Experience, 5(1):3.

Følstad, A., Nordheim, C. B., and Bjørkli, C. A. (2018). What makes users trust a chatbot for customer service? an exploratory interview study. In Internet Science: 5th International Conference, pages 194–208.

Følstad, A. and Skjuve, M. (2019). Chatbots for customer service: user experience and motivation. In International conference on conversational user interfaces, pages 1–9.

GARTNER (2019). Market guide for virtual customer assistants. Technical report. Disponível em: [link]. Acesso em: 22 mar. 2023.

Hassenzahl, M. (2008). User experience (ux) towards an experiential perspective on product quality. In Proceedings of the 20th Conference on l’Interaction Homme-Machine, pages 11–15.

Hassenzahl, M. (2018). The thing and i: understanding the relationship between user and product. Funology 2: from usability to enjoyment, pages 301–313.

Hassenzahl, M., Burmester, M., and Koller, F. (2003). Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. Mensch & Computer 2003: Interaktion in Bewegung, pages 187–196.

Hassenzahl, M., Law, E. L.-C., and Hvannberg, E. T. (2006). User experience-towards a unified view. Ux Ws Nordichi, 6:1–3.

Haugeland, I. K. F., Følstad, A., Taylor, C., and Bjørkli, C. A. (2022). Understanding the user experience of customer service chatbots: An experimental study of chatbot interaction design. International Journal of Human-Computer Studies, 161:102788.

Kvale, K., Freddi, E., Hodnebrog, S., Sell, O. A., and Følstad, A. (2020). Understanding the user experience of customer service chatbots: what can we learn from customer satisfaction surveys? In International Workshop on Chatbot Research and Design, pages 205–218.

Misischia, C. V., Poecze, F., and Strauss, C. (2022). Chatbots in customer service: Their relevance and impact on service quality. Procedia Computer Science, 201:421–428.

Sauro, J. (2015). Supr-q: A comprehensive measure of the quality of the website user experience. Journal of usability studies, 10(2).

Schankin, A., Budde, M., Riedel, T., and Beigl, M. (2022). Psychometric properties of the user experience questionnaire (ueq). In Conference on human factors in computing systems, pages 1–11.

Schloss, D., Gnewuch, U., and Maedche, A. (2022). Towards designing a conversation mining system for customer service chatbots. In International Conference on Information Systems.

van der Goot, M. J., Hafkamp, L., and Dankfort, Z. (2020). Customer service chatbots: A qualitative interview study into the communication journey of customers. In International Workshop on Chatbot Research and Design, pages 190–204.