Is my agent good enough? Evaluating Embodied Conversational Agents with Long and Short-term interactions
Resumo
The use of digital resources has been increasing in every instance of today’s society, being it in business or even ludic purposes. Despite such ever increasing use of technologies as interfaces, in all fields, it seems that it lacks the importance of users perception in this context. This work aims to present a case study about the evaluation of ECAs. We propose a Long-Term Interaction (LTI) to evaluate our conversational agent effectiveness through the user perception and compare it with Short-Term Interactions (STIs), performed by three users. Results show that many different aspects of users perception about the chosen ECA (i.e. Arthur) could be evaluated in our case study, in particular that LTI and STI are both important in order to have a better understanding of ECA impact in UX.
Palavras-chave:
Embodied Conversational Agent, Virtual Agent, Long-term Interaction, User Experience
Referências
B. Wimmer, B. Wockl, M. Leitner, and M. Tscheligi, “Measuring the dynamics of user experience in short interaction sequences”, in Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 2010, pp. 825–828.
Y. Chen, A. Naveed, and R. Porzel, “Behavior and preference in minimal personality: A study on embodied conversational agents”, in International conference on multimodal interfaces and the workshop on machine learning for multimodal interaction, 2010, pp. 1–4.
P. Knob, W. S. Dias, N. Kuniechick, J. Moraes, and S. R. Musse, “Arthur: a new eca that uses memory to improve communication”, in 2021 IEEE 15th International Conference on Semantic Computing (ICSC). IEEE, 2021, pp. 163–170
Z. Ruttkay, C. Dormann, and H. Noot, “Embodied conversational agents on a common ground: A framework for design and evaluation,” in From brows to trust: evaluating embodied conversational agents. Kluwer, 2004, pp. 27–66.
Y. Zhang, X. Gao, S. Lee, C. Brockett, M. Galley, J. Gao, and B. Dolan, “Consistent dialogue generation with self-supervised feature learning”, arXiv preprint arXiv:1903.05759, 2019.
H. Zhou, M. Huang, T. Zhang, X. Zhu, and B. Liu, “Emotional chatting machine: Emotional conversation generation with internal and external memory", in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
O. N. Yalc¸?n, “Empathy framework for embodied conversational agents", Cognitive Systems Research, vol. 59, pp. 123–132, 2020.
P. Morville, “Experience design unplugged,” in ACM SIGGRAPH 2005 Web Program, ser. SIGGRAPH ’05. New York, NY, USA: Association for Computing Machinery, 2005, p. 10.
J. Nielsen, “Enhancing the explanatory power of usability heuristics”, in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 1994, pp. 152–158.
N. Jakob, “Severity ratings for usability problems,” Papers and Essays, vol. 54, pp. 1–2, 1995.
J. W. Castro, R. Ren, S. T. Acuna, and J. d. Lara, “Usability of chatbots: A systematic mapping study”, Repositorio Academico Institucional, Universidad de Atacama, 2019.
W. Albert and T. Tullis, Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes, 2013.
T. Bickmore, D. Schulman, and L. Yin, “Maintaining engagement in long-term interventions with relational agents”, Applied Artificial Intelligence, vol. 24, no. 6, pp. 648–666, 2010.
S. Babu, S. Schmugge, T. Barnes, and L. F. Hodges, ““what would you like to talk about?” an evaluation of social conversations with a virtual receptionist”, in International Workshop on Intelligent Virtual Agents, Springer, 2006, pp. 169–180.
D. Benyon and O. Mival, “From human-computer interactions to human-companion relationships,” in Proceedings of the First International Conference on Intelligent Interactive Technologies and Multimedia, ser. IITM ’10. New York, NY, USA: Association for Computing Machinery, 2011, p. 1–9.
T. Castle-Green, S. Reeves, J. E. Fischer, and B. Koleva, “Decision trees as sociotechnical objects in chatbot design,” in Proceedings of the 2nd Conference on Conversational User Interfaces, 2020, pp. 1–3.
T. Kanda, R. Sato, N. Saiwaki, and H. Ishiguro, “A two-month field trial in an elementary school for long-term human–robot interaction”, IEEE Transactions on robotics, vol. 23, no. 5, pp. 962–971, 2007.
Y. Chen, A. Naveed, and R. Porzel, “Behavior and preference in minimal personality: A study on embodied conversational agents”, in International conference on multimodal interfaces and the workshop on machine learning for multimodal interaction, 2010, pp. 1–4.
P. Knob, W. S. Dias, N. Kuniechick, J. Moraes, and S. R. Musse, “Arthur: a new eca that uses memory to improve communication”, in 2021 IEEE 15th International Conference on Semantic Computing (ICSC). IEEE, 2021, pp. 163–170
Z. Ruttkay, C. Dormann, and H. Noot, “Embodied conversational agents on a common ground: A framework for design and evaluation,” in From brows to trust: evaluating embodied conversational agents. Kluwer, 2004, pp. 27–66.
Y. Zhang, X. Gao, S. Lee, C. Brockett, M. Galley, J. Gao, and B. Dolan, “Consistent dialogue generation with self-supervised feature learning”, arXiv preprint arXiv:1903.05759, 2019.
H. Zhou, M. Huang, T. Zhang, X. Zhu, and B. Liu, “Emotional chatting machine: Emotional conversation generation with internal and external memory", in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
O. N. Yalc¸?n, “Empathy framework for embodied conversational agents", Cognitive Systems Research, vol. 59, pp. 123–132, 2020.
P. Morville, “Experience design unplugged,” in ACM SIGGRAPH 2005 Web Program, ser. SIGGRAPH ’05. New York, NY, USA: Association for Computing Machinery, 2005, p. 10.
J. Nielsen, “Enhancing the explanatory power of usability heuristics”, in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 1994, pp. 152–158.
N. Jakob, “Severity ratings for usability problems,” Papers and Essays, vol. 54, pp. 1–2, 1995.
J. W. Castro, R. Ren, S. T. Acuna, and J. d. Lara, “Usability of chatbots: A systematic mapping study”, Repositorio Academico Institucional, Universidad de Atacama, 2019.
W. Albert and T. Tullis, Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes, 2013.
T. Bickmore, D. Schulman, and L. Yin, “Maintaining engagement in long-term interventions with relational agents”, Applied Artificial Intelligence, vol. 24, no. 6, pp. 648–666, 2010.
S. Babu, S. Schmugge, T. Barnes, and L. F. Hodges, ““what would you like to talk about?” an evaluation of social conversations with a virtual receptionist”, in International Workshop on Intelligent Virtual Agents, Springer, 2006, pp. 169–180.
D. Benyon and O. Mival, “From human-computer interactions to human-companion relationships,” in Proceedings of the First International Conference on Intelligent Interactive Technologies and Multimedia, ser. IITM ’10. New York, NY, USA: Association for Computing Machinery, 2011, p. 1–9.
T. Castle-Green, S. Reeves, J. E. Fischer, and B. Koleva, “Decision trees as sociotechnical objects in chatbot design,” in Proceedings of the 2nd Conference on Conversational User Interfaces, 2020, pp. 1–3.
T. Kanda, R. Sato, N. Saiwaki, and H. Ishiguro, “A two-month field trial in an elementary school for long-term human–robot interaction”, IEEE Transactions on robotics, vol. 23, no. 5, pp. 962–971, 2007.
Publicado
18/10/2021
Como Citar
DOS SANTOS, Juliane B. S.; KNOB, Paulo Ricardo; SCHERER, Victor Putrich; MUSSE, Soraia Raupp.
Is my agent good enough? Evaluating Embodied Conversational Agents with Long and Short-term interactions. In: TRILHA DE COMPUTAÇÃO – ARTIGOS CURTOS - SIMPÓSIO BRASILEIRO DE JOGOS E ENTRETENIMENTO DIGITAL (SBGAMES), 20. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 324-328.
DOI: https://doi.org/10.5753/sbgames_estendido.2021.19661.
