Human Factors in the Design of Chatbot Interactions: Conversational Design Practices
Resumo
Context: Chatbots are intelligent agents that mimic human behavior to carry on meaningful conversations. The conversational nature of chatbots poses challenges to designers since their development is different from other software and requires investigating new practices in the context of human-AI interaction and their impact on user experience. Since human dialogue involves several variables beyond verbalizing words, it is vital to design well-thought dialogues for chatbots to provide a humanized and optimal interaction. Objective: The main objective of this work is to unveil textual, visual, or interactive design practices from text-based chatbot interactions and how they can potentiate or weaken some perceptions and feelings of users, such as satisfaction, engagement, and trust, for the creation of the Guidelines for Chatbot Conversational Design (GCCD) guide. Method: We used multiple research methods to generate and validate the guide. First, we conducted a Systematic Literature Review (SRL) to identify conversational design practices and their impacts. These practices were inserted into the GCCD guide through qualitative analysis and coding of SLR results. Then, the guide was validated quantitatively through a survey and qualitatively through a case study. The survey aimed to assess the guide's clarity and usefulness based on the reading of the guide by the participants and their responses to a questionnaire adapted from the Technology Acceptance Model. The case study aimed to assess the guide's usefulness based on its practical application by participants in a situation that simulates a real scenario and follow-up interviews. Results: The survey showed that software developers with different levels of experience strongly agreed that the guide could induce greater user satisfaction and engagement. Furthermore, they also strongly agreed that the guide is clear, understandable, flexible, and easy to use. Although participants suggested some improvements, they reported that the guide's main strengths are objectivity and clarity. The case study confirmed the survey findings, as participants reported positive feelings toward the guide and an intention to use it. Their extensive perceptions given through the conducted interviews unveiled that their previous experiences with chatbots and in specific software development positions influenced their design and adoption of practices. Conclusion: The guide proved to be useful for developers with different levels of knowledge, with the potential to become a strong ally for developers in the conversational design process.
Referências
Saleema Amershi, Daniel S. Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi T. Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for HumanAI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Stephen A. Brewster, Geraldine Fitzpatrick, Anna L. Cox, and Vassilis Kostakos (Eds.). ACM, Glasgow, Scotland, UK, 3. DOI: 10.1145/3290605.3300233
Z. Ashktorab, M. Jain, Q. Vera Liao, and J.D. Weisz. 2019. Resilient chatbots: Repair strategy preferences for conversational breakdowns. Conference on Human Factors in Computing Systems Proceedings (2019). DOI: 10.1145/3290605.3300484.
Kitchenham Barbara and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. Keele University, UK 9 (2007), 1–65.
Petter Bae Brandtzaeg and Asbjørn Følstad. 2017. Why people use chatbots. In International conference on internet science. Springer, 377–392.
J. Bührke, A.B. Brendel, S. Lichtenberg, M. Greve, and M. Mirbabaie. 2021. Is making mistakes human? On the perception of typing errors in chatbot communication. Proceedings of the Annual Hawaii International Conference on System Sciences 2020-January (2021), 4456–4465.
Heloisa Candello, Claudio Pinhanez, and Flavio Figueiredo. 2017. Typefaces and the Perception of Humanness in Natural Language Chatbots. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3476–3487. DOI: 10.1145/3025453.3025919
Ana Paula Chaves and Marco Aurelio Gerosa. 2021. How Should My Chatbot Interact? A Survey on Social Characteristics in Human–Chatbot Interaction Design. International Journal of Human–Computer Interaction 37, 8 (2021), 729–758. DOI: 10.1080/10447318.2020.1841438 arXiv: [link]
D Jean Clandinin and F Michael Connelly. 2004. Narrative inquiry: Experience and story in qualitative research. John Wiley & Sons
Jasper Feine, Ulrich Gnewuch, Stefan Morana, and Alexander Maedche. 2019. A Taxonomy of Social Cues for Conversational Agents. International Journal of Human-Computer Studies 132 (2019), 138–161. DOI: 10.1016/j.ijhcs.2019.07.009
Asbjørn Følstad and Petter Bae Brandtzæg. 2017. Chatbots and the new world of HCI. Interactions 24, 4 (2017), 38–42. DOI: 10.1145/3085558
Asbjørn Følstad, Cecilie Bertinussen Nordheim, and Cato Alexander Bjørkli. 2018. What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study. In Internet Science 5th International Conference, INSCI 2018 (Lecture Notes in Computer Science, Vol. 11193), Svetlana S. Bodrunova (Ed.). Springer, St. Petersburg, Russia, 194–208. DOI: 10.1007/978-3-030-01437-7_16
International Organization for Standardization. 2017. Systems and software engineering — Vocabulary. Standard. International Organization for Standardization, Geneva, CH.
Floyd J Fowler Jr. 2013. Survey research methods. Sage publications.
Andrew Freed. 2021. Conversational AI: Chatbots that work. Manning Publications, New York, NY.
Google. 2021. What is conversation design? [link]
Yunsan Guo, Jian Wang, Runfan Wu, Zeyu Li, and Lingyun Sun. 2022. Designing for trust: a set of design principles to increase trust in chatbot. CCF Transactions on Pervasive Computing and Interaction (2022), 1–8.
Paul Henman. 2020. Improving public services using artificial intelligence: possibilities, pitfalls, governance. Asia Pacific Journal of Public Administration 42, 4 (2020), 209–221.
Michelle E Kiger and Lara Varpio. 2020. Thematic analysis of qualitative data: AMEE Guide No. 131. Medical teacher 42, 8 (2020), 846–854.
Kazunori Komatani, Ryu Takeda, Keisuke Nakashima, and Mikio Nakano. 2022. Design guidelines for developing systems for dialogue system competitions. In Conversational AI for Natural Human-Centric Interaction. Springer, 161–177.
Christine Liebrecht, Lena Sander, and Charlotte van Hooijdonk. 2020. Too informal? How a chatbot’s communication style affects brand attitude and quality of interaction. In International Workshop on Chatbot Research and Design. Springer, 16–31.
Malu Mafra, Kennedy Nunes, Adailton Castro, Adriana Lopes, Ana Carolina Oran, Geraldo Braz Junior, João Almeida, Anselmo Paiva, Aristofanes Silva, Simara Rocha, et al . 2022. Defining Requirements for the Development of Useful and Usable Chatbots: An Analysis of Quality Attributes from Academy and Industry. In International Conference on Human-Computer Interaction. Springer, 479–493.
Michael F. McTear. 2020. Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots. Morgan & Claypool Publishers, Toronto. DOI: 10.2200/S01060ED1V01Y202010HLT048
Albert Meharabian. 1980. Silent messages: Implicit communication of emotions and attributes. Wadsworth, California (1980).
Jo Anne Ollerenshaw and John W. Creswell. 2002. Narrative Research: A Comparison of Two Restorying Data Analysis Approaches. Qualitative Inquiry 8, 3 (2002), 329–347. DOI: 10.1177/10778004008003008 arXiv: [link]
Amon Rapp, Lorenzo Curti, and Arianna Boldi. 2021. The human side of humanchatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies 151 (2021), 102630. DOI: 10.1016/j.ijhcs.2021.102630
J. Rhim, M. Kwak, Y. Gong, and G. Gweon. 2022. Application of humanization to survey chatbots: Change in chatbot perception, interaction experience, and survey data quality. Computers in Human Behavior 126 (2022). DOI: 10.1016/j.chb.2021.107034
Geovana Ramos Sousa Silva and Edna Dias Canedo. 2023. Supplementary Material for Human Factors in the Design of Chatbot Interactions: Conversational Design Practices. DOI: 10.5281/zenodo.7538681
Jeff Stanley, Ronna ten Brink, Alexandra Valiton, Trevor Bostic, and Becca Scollan. 2022. Chatbot Accessibility Guidance: A Review and Way Forward. In Proceedings of Sixth International Congress on Information and Communication Technology. Springer, 919–942.
Kyoko Sugisaki and Andreas Bleiker. 2020. Usability Guidelines and Evaluation Criteria for Conversational User Interfaces: A Heuristic and Linguistic Approach. In Proceedings of the Conference on Mensch Und Computer (Magdeburg, Germany) (MuC ’20). Association for Computing Machinery, New York, NY, USA, 309–319. DOI: 10.1145/3404983.3405505
Joseph Weizenbaum. 1966. ELIZA a computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36–45. DOI: 10.1145/365153.365168
Joost F Wolfswinkel, Elfi Furtmueller, and Celeste P M Wilderom. 2013. Using grounded theory as a method for rigorously reviewing literature. European Journal of Information Systems 22, 1 (2013), 45–55. DOI: 10.1057/ejis.2011.51 arXiv: [link]
Xi Yang and Marco Aurisicchio. 2021. Designing Conversational Agents: A SelfDetermination Theory Approach. In CHI ’21: CHI Conference on Human Factors in Computing Systems, Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, Takeo Igarashi, Pernille Bjørn, and Steven Mark Drucker (Eds.). ACM, Virtual Event / Yokohama, Japan, May 8-13, 2021, 256:1–256:16. DOI: 10.1145/3411764.3445445
Robert K Yin. 2009. Case study research: Design and methods. Vol. 5. sage.
Jennifer Zamora. 2017. I’m Sorry, Dave, I’m Afraid I Can’t Do That: Chatbot Perception and Expectations. In Proceedings of the 5th International Conference on Human Agent Interaction, HAI 2017, Britta Wrede, Yukie Nagai, Takanori Komatsu, Marc Hanheide, and Lorenzo Natale (Eds.). ACM, Bielefeld, Germany, 253–260. DOI: 10.1145/3125739.3125766
Geovana Silva and Edna Canedo. 2022. Requirements Engineering Challenges and Techniques in Building Chatbots. In Proceedings of the 14th International Conference on Agents and Artificial Intelligence Volume 1: ICAART,. INSTICC, SciTePress, 180–187. DOI: 10.5220/0010801800003116
Geovana Ramos Sousa Silva and Edna Dias Canedo. 2023. Unveiling Quality in Chatbot Conversations: Quantitative Analysis of Chatbot Requirements. In Proceedings of the XXII Brazilian Symposium on Software Quality (conf-loc, cityBrasília/city, countryBrazil/country, /conf-loc) (SBQS ’23). Association for Computing Machinery, New York, NY, USA, 148–157. DOI: 10.1145/3629479.3629481
Geovana Ramos Sousa Silva and Edna Dias Canedo. 2024. Designing from experience: on the adoption of chatbot conversational design practices by practitioners. Personal and Ubiquitous Computing (2024), 1–16.
Geovana Ramos Sousa Silva and Edna Dias Canedo. 2024. Towards user-centric guidelines for chatbot conversational design. International Journal of Human– Computer Interaction 40, 2 (2024), 98–120
Geovana Ramos Sousa Silva, Genaína Nunes Rodrigues, and Edna Dias Canedo. 2023. A Modeling Strategy for the Verification of Context-Oriented Chatbot Conversational Flows via Model Checking. Journal of Universal Computer Science 29, 7 (2023), 805.
