Enhancing Empathic Communication in Healthcare Education Through Virtual Conversations: Leveraging Large Language Models for Real-Time Feedback
Resumo
Virtual conversations are increasingly utilized in healthcare education to enhance verbal empathic communication skills through tailored feedback on trainees’ responses. These conversations, supported by modalities such as speech, animation, and gestures, are highly customizable and accessible via the internet, bypassing the need for head-mounted displays (HMDs). However, training models to accurately evaluate empathic responses and generate human-like language with actionable suggestions remains a challenge. The advent of large language models (LLMs) provides new solutions to these challenges. This paper examines the impact of GPT-4-generated feedback on the empathic expressions of health professions trainees during virtual conversations. We enrolled 72 students from nursing and dental disciplines in an Interprofessional Collaborative Care course at the University of Florida. Participants were randomly assigned to one of two groups: one received feedback and suggestions from GPT-4 during conversations, while the other did not. We collected data on the perceived accuracy and helpfulness of the feedback from the intervention group. Using the Empathic Communication Coding System (ECCS) and GPT-4 Turbo, we assessed and compared the empathy levels of participants between the two groups. A Mann-Whitney U test was used to determine differences in average empathy levels. Results showed that participants receiving GPT-4 feedback had significantly higher median empathy levels than those without feedback. Feedback’s accuracy and utility were also affirmed by the participants. This study highlights the effectiveness of integrating LLMs like GPT-4 into virtual conversations for enhancing training outcomes in healthcare education.
Palavras-chave:
virtual humans, verbal empathic communication training, large language models
Referências
Abdellatif, A., Badran, K., Costa, D. E., & Shihab, E. (2021). A comparison of natural language understanding platforms for chatbots in software engineering. IEEE Transactions on Software Engineering, 48(8), 3087–3102.
Altman, D. G. (1990). Practical statistics for medical research. Chapman and Hall/CRC.
Askew, S., & Lodge, C. (2004). Gifts, ping-pong and loops–linking feedback and learning. In Feedback for learning. Routledge, 1–18.
Balmer, M. C. (2008). A dental undergraduate course for the management of medical emergencies in dental practice. European Journal of Dental Education, 12(4), 239–246.
Benedict, N. (2010). Virtual patients and problem-based learning in advanced therapeutics. American Journal of Pharmaceutical Education, 74(8), 143.
Bickmore, T., & Cassell, J. (2005). Social dialogue with embodied conversational agents. Advances in Natural Multimodal Dialogue Systems, 23–54.
Borish, M., Cordar, A., Foster, A., Kim, T., Murphy, J., & Lok, B. (2014). Utilizing real-time human-assisted virtual humans to increase real-world interaction empathy. Kansei Engineering & Emotion Research (KEER’14), 15.
Borish, M., & Lok, B. (2016). Rapid low-cost virtual human bootstrapping via the crowd. ACM Transactions on Intelligent Systems and Technology (TIST), 7(4), 1–20.
Burgess, A., van Diggele, C., Roberts, C., & Mellis, C. (2020). Feedback in the clinical setting. BMC Medical Education, 20, 1–5.
Bylund, C. L., & Makoul, G. (2002). Empathic communication and gender in the physician–patient encounter. Patient Education and Counseling, 48(3), 207–216.
Bylund, C. L., & Makoul, G. (2005). Examining empathy in medical encounters: an observational study using the empathic communication coding system. Health Communication, 18(2), 123–140.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213.
Curran, V., Reid, A., Reis, P., Doucet, S., Price, S., Alcock, L., & Fitzgerald, S. (2015). The use of information and communications technologies in the delivery of interprofessional education: A review of evaluation outcome levels. Journal of Interprofessional Care, 29(6), 541–550.
Derksen, F., Bensing, J., & Lagro-Janssen, A. (2013). Effectiveness of empathy in general practice: a systematic review. British Journal of General Practice, 63(606), e76–e84.
Evans, S. M., Ward, C., & Reeves, S. (2019). Online interprofessional education facilitation: A scoping review. Medical Teacher, 41(2), 215–222.
Fleiss, J. L., Levin, B., & Paik, M. C. (2013). Statistical methods for rates and proportions. John Wiley & Sons.
Foster, A., Chaudhary, N., Kim, T., Waller, J. L., Wong, J. L., Borish, M., Cordar, A., Lok, B., & Buckley, P. F. (2016). Using virtual patients to teach empathy: a randomized controlled study to enhance medical students’ empathic communication. Simulation in Healthcare, 11(3), 181–189.
Foster, A., Harms, J., Ange, B., Rossen, B., Lok, B., Lind, S., & Palladino, C. (2014). Empathic communication in medical students’ interactions with mental health virtual patient scenarios: a descriptive study using the Empathic Communication Coding System. Austin Journal of Psychiatry and Behavioral Sciences, 1(3), 6.
Graziano, W. G., Habashi, M. M., Sheese, B. E., & Tobin, R. M. (2007). Agreeableness, empathy, and helping: a person× situation perspective. Journal of Personality and Social Psychology, 93(4), 583.
Guilera, T., Batalla, I., & Soler-González, J. (2018). Empathy and specialty preference in medical students. Follow-up study and feedback. Educación Médica, 19, 153–161.
Hair, J. F. Jr., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis. In Multivariate data analysis. 62–68.
Kenny, P., Hartholt, A., Gratch, J., Swartout, W., Traum, D., Marsella, S., & Piepol, D. (2007). Building interactive virtual humans for training environments. In Proceedings of I/ITSEC, Vol. 174, 911–916.
Kenny, P., Parsons, T. D., Gratch, J., & Rizzo, A. A. (2008). Evaluation of Justina: a virtual patient with PTSD. In Intelligent Virtual Agents: 8th International Conference, IVA 2008, Tokyo, Japan, September 1-3, 2008. Proceedings 8. Springer, 394–408.
Kerasidou, A., Bærøe, K., Berger, Z., & Brown, A. E. (2021). The need for empathetic healthcare systems. Journal of Medical Ethics, 47(12), e27–e27.
Kleinsmith, A., Rivera-Gutierrez, D., Finney, G., Cendan, J., & Lok, B. (2015). Understanding empathy training with virtual patients. Computers in Human Behavior, 52, 151–158.
Lee, Y. K., Lee, I., Shin, M., Bae, S., & Hahn, S. (2023). Chain of empathy: Enhancing empathetic response of large language models based on psychotherapy models. arXiv preprint arXiv:2311.04915.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Peifer, C., Schönfeld, P., Wolters, G., Aust, F., & Margraf, J. (2020). Well done! Effects of positive feedback on perceived self-efficacy, flow and performance in a mental arithmetic task. Frontiers in Psychology, 11, 487115.
Riess, H., Kelley, J. M., Bailey, R. W., Dunn, E. J., & Phillips, M. (2012). Empathy training for resident physicians: a randomized controlled trial of a neuroscience-informed curriculum. Journal of General Internal Medicine, 27(10), 1280–1286.
Rivera-Gutierrez, D., Kleinsmith, A., Childs, G., Pileggi, R., & Lok, B. (2016). Self-assessment through interactive in-action reflections to improve interpersonal skills training. In 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT). IEEE, 143–147.
Sabharwal, N., & Agrawal, A. (2020). Introduction to Google Dialogflow. Apress, Berkeley, CA, 13–54.
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023). Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1), 46–57.
Sharma, A., Miner, A. S., Atkins, D. C., & Althoff, T. (2020). A computational approach to understanding empathy expressed in text-based mental health support. arXiv preprint arXiv:2009.08441.
Stanik-Hutt, J., Newhouse, R. P., White, K. M., Johantgen, M., Bass, E. B., Zangaro, G., Wilson, R., Fountain, L., Steinwachs, D. M., Heindel, L., et al. (2013). The quality and effectiveness of care provided by nurse practitioners. The Journal for Nurse Practitioners, 9(8), 492–500.
Sullivan, E. J., Handley, S. M., & Connors, H. (1994). The role of nurses in primary care: managing alcohol-abusing patients. Alcohol Health and Research World, 18(2), 158.
Volante, M., Babu, S. V., Chaturvedi, H., Newsome, N., Ebrahimi, E., Roy, T., Daily, S. B., & Fasolino, T. (2016). Effects of virtual human appearance fidelity on emotion contagion in affective inter-personal simulations. IEEE Transactions on Visualization and Computer Graphics, 22(4), 1326–1335.
Webster, D. (2014). Using standardized patients to teach therapeutic communication in psychiatric nursing. Clinical Simulation in Nursing, 10(2), e81–e86.
Winter, R., Issa, E., Roberts, N., Norman, R. I., & Howick, J. (2020). Assessing the effect of empathy-enhancing interventions in health education and training: a systematic review of randomized controlled trials. BMJ Open, 10(9), e036471.
Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 487662.
Yao, H., de Siqueira, A. G., Bafna, A., Peterkin, D., Richards, J., Rogers, M. L., Foster, A., Galynker, I., & Lok, B. (2022). A virtual human interaction using scaffolded ping-pong feedback for healthcare learners to practice empathy skills. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents (IVA ’22). Association for Computing Machinery, New York, NY, USA, Article 2, 8 pages.
Yao, H., de Siqueira, A. G., Foster, A., Galynker, I., & Lok, B. (2020). Toward automated evaluation of empathetic responses in virtual human interaction systems for mental health scenarios. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (IVA ’20). Association for Computing Machinery, New York, NY, USA, Article 57, 8 pages.
Yao, H., de Siqueira, A. G., Rogers, M. L., Bloch-Elkouby, S., Lawrence, O., Sarli, G., Foster, A., Mitelman, S. A., Galynker, I., & Lok, B. (2024). The impact of scaffolded and non-scaffolded suicidal virtual human interaction training on clinician emotional self-awareness, empathic communication, and clinical efficacy. BMC Medical Education, 24(1), 413.
Yu, C. C., Tan, L., Le, M. K., Tang, B., Liaw, S. Y., Tierney, T., Ho, Y. Y., Lim, B. E. E., Lim, D., Ng, R., et al. (2022). The development of empathy in the healthcare setting: a qualitative approach. BMC Medical Education, 22(1), 245.
Altman, D. G. (1990). Practical statistics for medical research. Chapman and Hall/CRC.
Askew, S., & Lodge, C. (2004). Gifts, ping-pong and loops–linking feedback and learning. In Feedback for learning. Routledge, 1–18.
Balmer, M. C. (2008). A dental undergraduate course for the management of medical emergencies in dental practice. European Journal of Dental Education, 12(4), 239–246.
Benedict, N. (2010). Virtual patients and problem-based learning in advanced therapeutics. American Journal of Pharmaceutical Education, 74(8), 143.
Bickmore, T., & Cassell, J. (2005). Social dialogue with embodied conversational agents. Advances in Natural Multimodal Dialogue Systems, 23–54.
Borish, M., Cordar, A., Foster, A., Kim, T., Murphy, J., & Lok, B. (2014). Utilizing real-time human-assisted virtual humans to increase real-world interaction empathy. Kansei Engineering & Emotion Research (KEER’14), 15.
Borish, M., & Lok, B. (2016). Rapid low-cost virtual human bootstrapping via the crowd. ACM Transactions on Intelligent Systems and Technology (TIST), 7(4), 1–20.
Burgess, A., van Diggele, C., Roberts, C., & Mellis, C. (2020). Feedback in the clinical setting. BMC Medical Education, 20, 1–5.
Bylund, C. L., & Makoul, G. (2002). Empathic communication and gender in the physician–patient encounter. Patient Education and Counseling, 48(3), 207–216.
Bylund, C. L., & Makoul, G. (2005). Examining empathy in medical encounters: an observational study using the empathic communication coding system. Health Communication, 18(2), 123–140.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213.
Curran, V., Reid, A., Reis, P., Doucet, S., Price, S., Alcock, L., & Fitzgerald, S. (2015). The use of information and communications technologies in the delivery of interprofessional education: A review of evaluation outcome levels. Journal of Interprofessional Care, 29(6), 541–550.
Derksen, F., Bensing, J., & Lagro-Janssen, A. (2013). Effectiveness of empathy in general practice: a systematic review. British Journal of General Practice, 63(606), e76–e84.
Evans, S. M., Ward, C., & Reeves, S. (2019). Online interprofessional education facilitation: A scoping review. Medical Teacher, 41(2), 215–222.
Fleiss, J. L., Levin, B., & Paik, M. C. (2013). Statistical methods for rates and proportions. John Wiley & Sons.
Foster, A., Chaudhary, N., Kim, T., Waller, J. L., Wong, J. L., Borish, M., Cordar, A., Lok, B., & Buckley, P. F. (2016). Using virtual patients to teach empathy: a randomized controlled study to enhance medical students’ empathic communication. Simulation in Healthcare, 11(3), 181–189.
Foster, A., Harms, J., Ange, B., Rossen, B., Lok, B., Lind, S., & Palladino, C. (2014). Empathic communication in medical students’ interactions with mental health virtual patient scenarios: a descriptive study using the Empathic Communication Coding System. Austin Journal of Psychiatry and Behavioral Sciences, 1(3), 6.
Graziano, W. G., Habashi, M. M., Sheese, B. E., & Tobin, R. M. (2007). Agreeableness, empathy, and helping: a person× situation perspective. Journal of Personality and Social Psychology, 93(4), 583.
Guilera, T., Batalla, I., & Soler-González, J. (2018). Empathy and specialty preference in medical students. Follow-up study and feedback. Educación Médica, 19, 153–161.
Hair, J. F. Jr., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis. In Multivariate data analysis. 62–68.
Kenny, P., Hartholt, A., Gratch, J., Swartout, W., Traum, D., Marsella, S., & Piepol, D. (2007). Building interactive virtual humans for training environments. In Proceedings of I/ITSEC, Vol. 174, 911–916.
Kenny, P., Parsons, T. D., Gratch, J., & Rizzo, A. A. (2008). Evaluation of Justina: a virtual patient with PTSD. In Intelligent Virtual Agents: 8th International Conference, IVA 2008, Tokyo, Japan, September 1-3, 2008. Proceedings 8. Springer, 394–408.
Kerasidou, A., Bærøe, K., Berger, Z., & Brown, A. E. (2021). The need for empathetic healthcare systems. Journal of Medical Ethics, 47(12), e27–e27.
Kleinsmith, A., Rivera-Gutierrez, D., Finney, G., Cendan, J., & Lok, B. (2015). Understanding empathy training with virtual patients. Computers in Human Behavior, 52, 151–158.
Lee, Y. K., Lee, I., Shin, M., Bae, S., & Hahn, S. (2023). Chain of empathy: Enhancing empathetic response of large language models based on psychotherapy models. arXiv preprint arXiv:2311.04915.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Peifer, C., Schönfeld, P., Wolters, G., Aust, F., & Margraf, J. (2020). Well done! Effects of positive feedback on perceived self-efficacy, flow and performance in a mental arithmetic task. Frontiers in Psychology, 11, 487115.
Riess, H., Kelley, J. M., Bailey, R. W., Dunn, E. J., & Phillips, M. (2012). Empathy training for resident physicians: a randomized controlled trial of a neuroscience-informed curriculum. Journal of General Internal Medicine, 27(10), 1280–1286.
Rivera-Gutierrez, D., Kleinsmith, A., Childs, G., Pileggi, R., & Lok, B. (2016). Self-assessment through interactive in-action reflections to improve interpersonal skills training. In 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT). IEEE, 143–147.
Sabharwal, N., & Agrawal, A. (2020). Introduction to Google Dialogflow. Apress, Berkeley, CA, 13–54.
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023). Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1), 46–57.
Sharma, A., Miner, A. S., Atkins, D. C., & Althoff, T. (2020). A computational approach to understanding empathy expressed in text-based mental health support. arXiv preprint arXiv:2009.08441.
Stanik-Hutt, J., Newhouse, R. P., White, K. M., Johantgen, M., Bass, E. B., Zangaro, G., Wilson, R., Fountain, L., Steinwachs, D. M., Heindel, L., et al. (2013). The quality and effectiveness of care provided by nurse practitioners. The Journal for Nurse Practitioners, 9(8), 492–500.
Sullivan, E. J., Handley, S. M., & Connors, H. (1994). The role of nurses in primary care: managing alcohol-abusing patients. Alcohol Health and Research World, 18(2), 158.
Volante, M., Babu, S. V., Chaturvedi, H., Newsome, N., Ebrahimi, E., Roy, T., Daily, S. B., & Fasolino, T. (2016). Effects of virtual human appearance fidelity on emotion contagion in affective inter-personal simulations. IEEE Transactions on Visualization and Computer Graphics, 22(4), 1326–1335.
Webster, D. (2014). Using standardized patients to teach therapeutic communication in psychiatric nursing. Clinical Simulation in Nursing, 10(2), e81–e86.
Winter, R., Issa, E., Roberts, N., Norman, R. I., & Howick, J. (2020). Assessing the effect of empathy-enhancing interventions in health education and training: a systematic review of randomized controlled trials. BMJ Open, 10(9), e036471.
Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 487662.
Yao, H., de Siqueira, A. G., Bafna, A., Peterkin, D., Richards, J., Rogers, M. L., Foster, A., Galynker, I., & Lok, B. (2022). A virtual human interaction using scaffolded ping-pong feedback for healthcare learners to practice empathy skills. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents (IVA ’22). Association for Computing Machinery, New York, NY, USA, Article 2, 8 pages.
Yao, H., de Siqueira, A. G., Foster, A., Galynker, I., & Lok, B. (2020). Toward automated evaluation of empathetic responses in virtual human interaction systems for mental health scenarios. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (IVA ’20). Association for Computing Machinery, New York, NY, USA, Article 57, 8 pages.
Yao, H., de Siqueira, A. G., Rogers, M. L., Bloch-Elkouby, S., Lawrence, O., Sarli, G., Foster, A., Mitelman, S. A., Galynker, I., & Lok, B. (2024). The impact of scaffolded and non-scaffolded suicidal virtual human interaction training on clinician emotional self-awareness, empathic communication, and clinical efficacy. BMC Medical Education, 24(1), 413.
Yu, C. C., Tan, L., Le, M. K., Tang, B., Liaw, S. Y., Tierney, T., Ho, Y. Y., Lim, B. E. E., Lim, D., Ng, R., et al. (2022). The development of empathy in the healthcare setting: a qualitative approach. BMC Medical Education, 22(1), 245.
Publicado
30/09/2024
Como Citar
YAO, Heng; DE SIQUEIRA, Alexandre Gomes; JOHNSON, Margeaux; PILEGGI, Roberta; BLUE, Amy; BUMBACH, Michael D.; LOVE, Rene; LOK, Benjamin.
Enhancing Empathic Communication in Healthcare Education Through Virtual Conversations: Leveraging Large Language Models for Real-Time Feedback. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 26. , 2024, Manaus/AM.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 41-50.
