Evaluation of AI-Based Virtual Assistants for Psychological Counseling Simulations
Abstract
This paper addresses mental health virtual assistants, or chatbots, that simulate psychotherapeutic care. These chatbots are based on Large Lan-guage Models, such as ChatGPT, and are trained with data from real therapy sessions. They have been widely adopted due to their cost, ease of access, avail-ability, anonymity, and other factors that make them an alternative to a psy-chologist. The theoretical foundation and a review of the literature focusing on their creation, evaluation, and specific objectives are presented. The positive aspects and limitations are discussed. Finally, seven selected applications are detailed according to their popularity, objectives, ease of use, and user reviews.
References
Bill, D., Eriksson, T. (2023). “Fine-tuning a LLM using Reinforcement Learning from Human Feedback for a Therapy Chatbot Application”. Trab. conclusão de curso. Escola de Engenharia Elétrica e Ciência da Computação. Instituto Real de Tecnologia.
Coghlan, S., Leins, K., Sheldrick, S., D’Alfonso, S. (2023). “To chat or bot to chat: Ethical issues with using chatbots in mental health”. Digital Health. 9: 11.
Cooper, J. O., Heron, T. E., Heward, W. L. (2020) “Applied Behavior Analysis”. Pearson.
Demontis, E., Lorena, G., Merlin, F., Basso, A., Chiavarino, C. (2021). “Chatbots for the clinical training of psychologists”. Tec., Mind, and Behavior. TMS Proceedings 2021.
Dokukina, I., Gumanova, J. (2020) “The rise of chatbots – new personal assistants in foreign language learning”. Procedia Computer Science, v. 169, p. 542–546, 2020.
Faceli, K., Lorena, A. C., Gama, J., Almeida, T. A., & de Carvalho, A. C. P. L. F. (2021).
“Inteligência Artificial: Uma Abordagem de Aprendizado de Máquina” (2ª ed.). LTC.
Jurafsky, D., Martin, J. H. (2008). Speech and Language Processing: An Intro. to Nat.
Lang. Processing, Computational Linguistics, and Speech Recognition. Prentice Hall.
Lin, S. et al. (2023) “Empathy-Based communication Framework for Chatbots: A Mental Health Chatbot Application and Evaluation”. In: Proc. 11th Int. Conf. on HumanAgent Interaction. Association for Computing Machinery.
Lorena, G., Demontis, E. (2023) “Autonomous Systems for education: use of chatbots for psychologists training”. Duazary, v. 20, p. 9.
Maroengsit, W. et al (2019). A Survey on Evaluation Methods for Chatbots. Proc.7th Int. Conf. on Information and Education Technology, 111–119.
Matthews, P. (2024) ‘Empathic’ Mental Health Chatbots. [link], Acesso em agosto de 2024.
Morrison, J. Q., Puzio, H. (2017). “Utilization of virtual simulation and other technologybased educational tools in school psychology training programs”. Contemporary School Psychology, v. 21, n. 3, p. 287-295.
Organização Pan-Americana da Saúde (2022). OMS destaca necessidade urgente de transformar a saúde mental e a atenção. [link], Acesso: ago. 2024.
Palanica, A. et al. (2019) “Physicians’ Perceptions of Chatbots in Health Care: Cross-Sectional Web-Based Survey”. J. Med. Internet Res. 21, e12887.
Park, S. et al. (2019). “Designing a Chatbot for a Brief Motivational Interview on Stress Management: Qualitative Case Study”. J. Medical Internet Research 21, e12231.
Rojewska, K. et al (2022). “Natural Language Processing and Machine Learning Supporting the Work of a Psychologist and Its Evaluation on the Example of Support for Psychological Diagnosis of Anorexia”. Applied Sciences 12, 4702.
Romero Velázquez, M., Casadevante de la Fuente, C., Montoro, H., (2020). Cómo Construir um Psicólogo-chatbot. Papeles del Psicólogo - Psychologist Papers, 41.
Rooney, Anne (2016). A história da psicologia. São Paulo: M Book.
Shan, Y., Zhang, J., Li, Z., Feng, Y., & Zhou, J. (2022). Mental Health Assessment for the Chatbots. ArXiv, abs/2201.05382.
Shevat, A. (2017) Designing Bots: Creating Conversational Experiences. O'Reilly Media.
Svikhnushina, E., Pu, P. (2023) “Approximating Online Human Evaluation of Social Chatbots with Prompting”. ArXiv, abs/2304.05253v1.
Tidy, J. (2024) “No divã com IA: os jovens que fazem terapia com bots de IA”. [link], Acesso em agosto de 2024.
Woebot Health (2024). AI Core Principles. [link], Acesso em julho de 2024.
WHO (2020). The World Health Report: Mental Health Atlas 2020. [link], Acesso em junho de 2024.
Coghlan, S., Leins, K., Sheldrick, S., D’Alfonso, S. (2023). “To chat or bot to chat: Ethical issues with using chatbots in mental health”. Digital Health. 9: 11.
Cooper, J. O., Heron, T. E., Heward, W. L. (2020) “Applied Behavior Analysis”. Pearson.
Demontis, E., Lorena, G., Merlin, F., Basso, A., Chiavarino, C. (2021). “Chatbots for the clinical training of psychologists”. Tec., Mind, and Behavior. TMS Proceedings 2021.
Dokukina, I., Gumanova, J. (2020) “The rise of chatbots – new personal assistants in foreign language learning”. Procedia Computer Science, v. 169, p. 542–546, 2020.
Faceli, K., Lorena, A. C., Gama, J., Almeida, T. A., & de Carvalho, A. C. P. L. F. (2021).
“Inteligência Artificial: Uma Abordagem de Aprendizado de Máquina” (2ª ed.). LTC.
Jurafsky, D., Martin, J. H. (2008). Speech and Language Processing: An Intro. to Nat.
Lang. Processing, Computational Linguistics, and Speech Recognition. Prentice Hall.
Lin, S. et al. (2023) “Empathy-Based communication Framework for Chatbots: A Mental Health Chatbot Application and Evaluation”. In: Proc. 11th Int. Conf. on HumanAgent Interaction. Association for Computing Machinery.
Lorena, G., Demontis, E. (2023) “Autonomous Systems for education: use of chatbots for psychologists training”. Duazary, v. 20, p. 9.
Maroengsit, W. et al (2019). A Survey on Evaluation Methods for Chatbots. Proc.7th Int. Conf. on Information and Education Technology, 111–119.
Matthews, P. (2024) ‘Empathic’ Mental Health Chatbots. [link], Acesso em agosto de 2024.
Morrison, J. Q., Puzio, H. (2017). “Utilization of virtual simulation and other technologybased educational tools in school psychology training programs”. Contemporary School Psychology, v. 21, n. 3, p. 287-295.
Organização Pan-Americana da Saúde (2022). OMS destaca necessidade urgente de transformar a saúde mental e a atenção. [link], Acesso: ago. 2024.
Palanica, A. et al. (2019) “Physicians’ Perceptions of Chatbots in Health Care: Cross-Sectional Web-Based Survey”. J. Med. Internet Res. 21, e12887.
Park, S. et al. (2019). “Designing a Chatbot for a Brief Motivational Interview on Stress Management: Qualitative Case Study”. J. Medical Internet Research 21, e12231.
Rojewska, K. et al (2022). “Natural Language Processing and Machine Learning Supporting the Work of a Psychologist and Its Evaluation on the Example of Support for Psychological Diagnosis of Anorexia”. Applied Sciences 12, 4702.
Romero Velázquez, M., Casadevante de la Fuente, C., Montoro, H., (2020). Cómo Construir um Psicólogo-chatbot. Papeles del Psicólogo - Psychologist Papers, 41.
Rooney, Anne (2016). A história da psicologia. São Paulo: M Book.
Shan, Y., Zhang, J., Li, Z., Feng, Y., & Zhou, J. (2022). Mental Health Assessment for the Chatbots. ArXiv, abs/2201.05382.
Shevat, A. (2017) Designing Bots: Creating Conversational Experiences. O'Reilly Media.
Svikhnushina, E., Pu, P. (2023) “Approximating Online Human Evaluation of Social Chatbots with Prompting”. ArXiv, abs/2304.05253v1.
Tidy, J. (2024) “No divã com IA: os jovens que fazem terapia com bots de IA”. [link], Acesso em agosto de 2024.
Woebot Health (2024). AI Core Principles. [link], Acesso em julho de 2024.
WHO (2020). The World Health Report: Mental Health Atlas 2020. [link], Acesso em junho de 2024.
Published
2024-10-17
How to Cite
ZANQUETTA, Bruno; OLIVEIRA, Matheus V. M. de; OLIVEIRA, Leandro R. de; CINTRA, Marcos E.; FERNANDES, Aline F. Campagna.
Evaluation of AI-Based Virtual Assistants for Psychological Counseling Simulations. In: REGIONAL SCHOOL OF INFORMATICS OF ESPÍRITO SANTO (ERI-ES), 9. , 2024, Vitória/ES.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 61-70.
DOI: https://doi.org/10.5753/eries.2024.244414.
