LLM vs. Human Driven Conversation: A User Study in Virtual Reality Medical Simulation
Resumo
Virtual humans are an important area of study in human-computer interaction, where artificial intelligence-driven agents have gained prominence with the emergence of large language models (LLMs). This work explores users’ preferences in conversations produced either by a large language model or by a human in a synthesized voice conversation. To address this question, we developed a scenario within a medical learning simulator. There, the LLM and a human participant alternately simulate the role of a patient in the medical scenario while the subject user takes the role of the doctor. We conducted a user experiment with a relevant population to assess the use of AI agents in realistic training environments, focusing on user perceptions of realism, empathy, and utility. The findings suggest that LLM-based virtual patients hold strong potential for scalable clinical training, but improvements in expressiveness and interaction flow are essential for broader adoption.
Palavras-chave:
LLM, virtual reality, virtual human, intelligent agent, voice interaction, health education
Referências
World Health Organization. Global strategy on human resources for health: Workforce 2030. 2016. Available: [link]
Joyner, B. and Young, D. L. Teaching medical students using role play: Twelve tips for successful role plays. Medical Teacher, 28(3), 225–229, 2009. DOI: 10.1080/01421590600711252
Nestel, D. and Tierney, T. Role-play for medical students learning about communication: Guidelines for maximising benefits. BMC Medical Education, 7(1), 3, 2007. DOI: 10.1186/1472-6920-7-3
Cook, D. A. and Triola, M. M. Virtual patients: a critical literature review and proposed next steps. Medical Education, 43(4), 303–311, 2009.
Consorti, F. Mancuso, R. Nocioni, M. and Piccolo, A. Efficacy of virtual patients in medical education: a meta-analysis of randomized studies. Computers & Education, 59(3), 1001–1008, 2012.
Li, Z. Zhang, H. Peng, C. and Peiris, R. Exploring large language model-driven agents for environment-aware spatial interactions and conversations in virtual reality role-play scenarios. 2025 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 1–11, 2025.
Freeman, D. Reeve, S. Robinson, A. Ehlers, A. Clark, D. Spanlang, B. and Slater, M. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychological Medicine, 47(14), 2393–2400, 2017.
Ke, L. B, L. S, G. D, J. E. S, G. and C, M. Virtual reality for stroke rehabilitation. Cochrane Database of Systematic Reviews, 11, 2017. DOI: 10.1002/14651858.CD008349.pub4
Chaby, L. Benamara, A. Pino, M. Prigent, E. Ravenet, B. Martin, J. C. Vanderstichel, H. Rigaud, A. S. and Chetouani, M. Embodied virtual patients as a simulation-based framework for training clinician-patient communication skills: An overview of their use in psychiatric and geriatric care. Frontiers in Virtual Reality, 3, 827312, 2022.
Jimenez, F. A. Can Virtual Patient Simulation Be Used in Substitution of Traditional Clinical Hours in Undergraduate Nursing Education? A Review of the Evidence. White Paper, 15, 2022.
Patel, D. B. Pei, Y. Vasoya, M. and Hershberger, P. J. Computer-Supported Experiential Learning-Based Tool for Healthcare Skills. IEEE Computer Graphics and Applications, 43(2), 57–68, 2023.
Bellucci, A. Jacucci, G. Trung, K. D. Das, P. K. Smirnov, S. V. Ahmed, I. and Lugrin, J. L. Immersive tailoring of embodied agents using large language models. 2025 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 392–400, 2025.
Lugrin, J. L. Latt, J. and Latoschik, M. E. Avatar embodiment realism and social interaction quality in virtual reality. 2015 IEEE Virtual Reality (VR), 225–226, 2015.
Cassell, J. Sullivan, J. Prevost, S. and Churchill, E. Embodied Conversational Agents. Cambridge, MA: MIT Press, 2000.
Cai, Y. Liu, H. and Wang, J. How does interactive virtual reality enhance learning outcomes via emotional and cognitive mechanisms? evidence from the COVID-19 pandemic. Frontiers in Psychology, 13, 1081372, 2022. DOI: 10.3389/fpsyg.2022.1081372
Ghandeharioun, A. McDuff, D. Czerwinski, M. and Picard, R. Physiological responses to facial expressions of emotion in a virtual social interaction task. IEEE Transactions on Affective Computing, 2019.
Morley, J. Machado, C. C. Burr, C. Cowls, J. Joshi, I. Taddeo, M. and Floridi, L. The ethics of AI in health care: A mapping review. Social Science & Medicine, 260, 113172, 2020. Available: [link]
Masoumian, H. M. Masoumain, H. T. and Qayumi, K. Integration of artificial intelligence in medical education: opportunities, challenges, and ethical considerations. Journal of Medical Education for Future Demands, 2023.
Harden, R. M. and Gleeson, F. A. Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education, 13(1), 41–54, 1979.
Mikropoulos, T. A. and Natsis, A. Educational virtual environments: A ten-year review of empirical research (1999–2009). Computers & Education, 56(3), 769–780, 2011. Available: [link]
Schmid Mast, M. Kindlimann, A. and Langewitz, W. Reciprocal influence in doctor–patient interaction: A dyadic analysis of doctor–patient communication. Social Science & Medicine, 60(5), 1105–1117, 2005.
Gao, Y. Dai, Y. Zhang, G. Guo, H. Mostajeran, F. Zheng, B. and Yu, T. Trust in virtual agents: Exploring the role of stylization and voice. IEEE Transactions on Visualization and Computer Graphics, 31(5), 3623–3633, 2025.
Connolly, R. Buck, L. Zordan, V. and McDonnell, R. The impact of navigation on proxemics in an immersive virtual environment with conversational agents. IEEE Transactions on Visualization and Computer Graphics, 31(5), 2787–2797, 2025.
Wan, H. Zhang, J. Suria, A. A. Yao, B. Wang, D. Coady, Y. and Prpa, M. Building LLM-based AI agents in social virtual reality. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24). ACM, 2024. DOI: 10.1145/3613905.3651026
Bovo, R. Abreu, S. Ahuja, K. Gonzalez, E. J. Cheng, L. T. and Gonzalez-Franco, M. Embardiment: an embodied AI agent for productivity in XR. 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), 708–717, 2025.
Basit, A. and Shafique, M. TinyDigiClones: A multi-modal LLM-based framework for edge-optimized personalized avatars. 2024 International Joint Conference on Neural Networks (IJCNN), 1–9, 2024.
Wu, S. Zhao, S. Huang, Q. Huang, K. Yasunaga, M. Cao, K. Ioannidis, V. N. Subbian, K. Leskove, J. and Zou, J. Avatar: Optimizing LLM agents for tool usage via contrastive reasoning. NeurIPS, 2024.
Rupprecht, T. Chang, S. E. Wu, Y. Lu, L. Nan, E. Li, C. H. Lai, C. Li, Z. Hu, Z. He, Y. Kaeli, D. and Wang, Y. Digital avatars: Framework development and their evaluation. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24), 8780–8783, 2024. DOI: 10.24963/ijcai.2024/1031
John, K. S. Roy, G. A. and P. S. B. LLM based 3D avatar assistant. 2024 1st International Conference on Trends in Engineering Systems and Technologies (ICTEST), 1–5, 2024.
Maslych, M. Pumarada, C. Ghasemaghaei, A. and L. J. J. Jr. Takeaways from applying LLM capabilities to multiple conversational avatars in a VR pilot study. arXiv preprint arXiv:2501.00168, 2025. Available: [link]
Qin, H. X. Jin, S. Gao, Z. Fan, M. and Hui, P. CharacterMeet: Supporting creative writers’ entire story character construction processes through conversation with LLM-powered chatbot avatars. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). ACM, 2024. DOI: 10.1145/3613904.3642105
Cook, D. A. Overgaard, J. Pankratz, V. S. Fiol, G. D. and Aakre, C. A. Virtual patients using large language models: Scalable, contextualized simulation of clinician-patient dialogue with feedback. Journal of Medical Internet Research, 27, e68486, 2025. Available: [link]
Thunstrom, A. O. Evaluating virtual reality and artificial intelligence as emerging digital tools for mental health care. Doctoral Thesis, University of Gothenburg, 2025.
Kim, H. Lee, J. Han, S. Yoon, Y. Park, S. Kim, J. Lee, J. Kang, J. and Oh, A. Adaptive-VP: A framework for LLM-based virtual patients that adapts to trainees’ dialogue to facilitate nurse communication training. Journal of Medical Internet Research, 27(1), e68486, 2025. Available: [link]
Slater, M. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549–3557, 2009.
Radford, A. Kim, J. W. Xu, T. Brockman, G. McLeavey, C. and Sutskever, I. Robust speech recognition via large-scale weak supervision. Proceedings of the 40th International Conference on Machine Learning (ICML’23), 2023.
Team, G. Kamath, A. Ferret, J. Pathak, S. Vieillard, N. Merhej, R. Perrin, S. Matejovicova, T. Rame, A. and Riviere, M. Gemma 3 Technical Report. arXiv preprint arXiv:2503.19786, 2025.
Coqui. XTTS v2: Cross-lingual text-to-speech. [link], 2024. Accessed: 2025-05-09.
Kelley, J. F. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems, 2(1), 26–41, 1984. DOI: 10.1145/357417.357420
Lazar, J. Feng, J. H. and Hochheiser, H. Research Methods in Human-Computer Interaction, 2nd ed. Morgan Kaufmann, 2017.
Joyner, B. and Young, D. L. Teaching medical students using role play: Twelve tips for successful role plays. Medical Teacher, 28(3), 225–229, 2009. DOI: 10.1080/01421590600711252
Nestel, D. and Tierney, T. Role-play for medical students learning about communication: Guidelines for maximising benefits. BMC Medical Education, 7(1), 3, 2007. DOI: 10.1186/1472-6920-7-3
Cook, D. A. and Triola, M. M. Virtual patients: a critical literature review and proposed next steps. Medical Education, 43(4), 303–311, 2009.
Consorti, F. Mancuso, R. Nocioni, M. and Piccolo, A. Efficacy of virtual patients in medical education: a meta-analysis of randomized studies. Computers & Education, 59(3), 1001–1008, 2012.
Li, Z. Zhang, H. Peng, C. and Peiris, R. Exploring large language model-driven agents for environment-aware spatial interactions and conversations in virtual reality role-play scenarios. 2025 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 1–11, 2025.
Freeman, D. Reeve, S. Robinson, A. Ehlers, A. Clark, D. Spanlang, B. and Slater, M. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychological Medicine, 47(14), 2393–2400, 2017.
Ke, L. B, L. S, G. D, J. E. S, G. and C, M. Virtual reality for stroke rehabilitation. Cochrane Database of Systematic Reviews, 11, 2017. DOI: 10.1002/14651858.CD008349.pub4
Chaby, L. Benamara, A. Pino, M. Prigent, E. Ravenet, B. Martin, J. C. Vanderstichel, H. Rigaud, A. S. and Chetouani, M. Embodied virtual patients as a simulation-based framework for training clinician-patient communication skills: An overview of their use in psychiatric and geriatric care. Frontiers in Virtual Reality, 3, 827312, 2022.
Jimenez, F. A. Can Virtual Patient Simulation Be Used in Substitution of Traditional Clinical Hours in Undergraduate Nursing Education? A Review of the Evidence. White Paper, 15, 2022.
Patel, D. B. Pei, Y. Vasoya, M. and Hershberger, P. J. Computer-Supported Experiential Learning-Based Tool for Healthcare Skills. IEEE Computer Graphics and Applications, 43(2), 57–68, 2023.
Bellucci, A. Jacucci, G. Trung, K. D. Das, P. K. Smirnov, S. V. Ahmed, I. and Lugrin, J. L. Immersive tailoring of embodied agents using large language models. 2025 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 392–400, 2025.
Lugrin, J. L. Latt, J. and Latoschik, M. E. Avatar embodiment realism and social interaction quality in virtual reality. 2015 IEEE Virtual Reality (VR), 225–226, 2015.
Cassell, J. Sullivan, J. Prevost, S. and Churchill, E. Embodied Conversational Agents. Cambridge, MA: MIT Press, 2000.
Cai, Y. Liu, H. and Wang, J. How does interactive virtual reality enhance learning outcomes via emotional and cognitive mechanisms? evidence from the COVID-19 pandemic. Frontiers in Psychology, 13, 1081372, 2022. DOI: 10.3389/fpsyg.2022.1081372
Ghandeharioun, A. McDuff, D. Czerwinski, M. and Picard, R. Physiological responses to facial expressions of emotion in a virtual social interaction task. IEEE Transactions on Affective Computing, 2019.
Morley, J. Machado, C. C. Burr, C. Cowls, J. Joshi, I. Taddeo, M. and Floridi, L. The ethics of AI in health care: A mapping review. Social Science & Medicine, 260, 113172, 2020. Available: [link]
Masoumian, H. M. Masoumain, H. T. and Qayumi, K. Integration of artificial intelligence in medical education: opportunities, challenges, and ethical considerations. Journal of Medical Education for Future Demands, 2023.
Harden, R. M. and Gleeson, F. A. Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education, 13(1), 41–54, 1979.
Mikropoulos, T. A. and Natsis, A. Educational virtual environments: A ten-year review of empirical research (1999–2009). Computers & Education, 56(3), 769–780, 2011. Available: [link]
Schmid Mast, M. Kindlimann, A. and Langewitz, W. Reciprocal influence in doctor–patient interaction: A dyadic analysis of doctor–patient communication. Social Science & Medicine, 60(5), 1105–1117, 2005.
Gao, Y. Dai, Y. Zhang, G. Guo, H. Mostajeran, F. Zheng, B. and Yu, T. Trust in virtual agents: Exploring the role of stylization and voice. IEEE Transactions on Visualization and Computer Graphics, 31(5), 3623–3633, 2025.
Connolly, R. Buck, L. Zordan, V. and McDonnell, R. The impact of navigation on proxemics in an immersive virtual environment with conversational agents. IEEE Transactions on Visualization and Computer Graphics, 31(5), 2787–2797, 2025.
Wan, H. Zhang, J. Suria, A. A. Yao, B. Wang, D. Coady, Y. and Prpa, M. Building LLM-based AI agents in social virtual reality. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24). ACM, 2024. DOI: 10.1145/3613905.3651026
Bovo, R. Abreu, S. Ahuja, K. Gonzalez, E. J. Cheng, L. T. and Gonzalez-Franco, M. Embardiment: an embodied AI agent for productivity in XR. 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), 708–717, 2025.
Basit, A. and Shafique, M. TinyDigiClones: A multi-modal LLM-based framework for edge-optimized personalized avatars. 2024 International Joint Conference on Neural Networks (IJCNN), 1–9, 2024.
Wu, S. Zhao, S. Huang, Q. Huang, K. Yasunaga, M. Cao, K. Ioannidis, V. N. Subbian, K. Leskove, J. and Zou, J. Avatar: Optimizing LLM agents for tool usage via contrastive reasoning. NeurIPS, 2024.
Rupprecht, T. Chang, S. E. Wu, Y. Lu, L. Nan, E. Li, C. H. Lai, C. Li, Z. Hu, Z. He, Y. Kaeli, D. and Wang, Y. Digital avatars: Framework development and their evaluation. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24), 8780–8783, 2024. DOI: 10.24963/ijcai.2024/1031
John, K. S. Roy, G. A. and P. S. B. LLM based 3D avatar assistant. 2024 1st International Conference on Trends in Engineering Systems and Technologies (ICTEST), 1–5, 2024.
Maslych, M. Pumarada, C. Ghasemaghaei, A. and L. J. J. Jr. Takeaways from applying LLM capabilities to multiple conversational avatars in a VR pilot study. arXiv preprint arXiv:2501.00168, 2025. Available: [link]
Qin, H. X. Jin, S. Gao, Z. Fan, M. and Hui, P. CharacterMeet: Supporting creative writers’ entire story character construction processes through conversation with LLM-powered chatbot avatars. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). ACM, 2024. DOI: 10.1145/3613904.3642105
Cook, D. A. Overgaard, J. Pankratz, V. S. Fiol, G. D. and Aakre, C. A. Virtual patients using large language models: Scalable, contextualized simulation of clinician-patient dialogue with feedback. Journal of Medical Internet Research, 27, e68486, 2025. Available: [link]
Thunstrom, A. O. Evaluating virtual reality and artificial intelligence as emerging digital tools for mental health care. Doctoral Thesis, University of Gothenburg, 2025.
Kim, H. Lee, J. Han, S. Yoon, Y. Park, S. Kim, J. Lee, J. Kang, J. and Oh, A. Adaptive-VP: A framework for LLM-based virtual patients that adapts to trainees’ dialogue to facilitate nurse communication training. Journal of Medical Internet Research, 27(1), e68486, 2025. Available: [link]
Slater, M. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549–3557, 2009.
Radford, A. Kim, J. W. Xu, T. Brockman, G. McLeavey, C. and Sutskever, I. Robust speech recognition via large-scale weak supervision. Proceedings of the 40th International Conference on Machine Learning (ICML’23), 2023.
Team, G. Kamath, A. Ferret, J. Pathak, S. Vieillard, N. Merhej, R. Perrin, S. Matejovicova, T. Rame, A. and Riviere, M. Gemma 3 Technical Report. arXiv preprint arXiv:2503.19786, 2025.
Coqui. XTTS v2: Cross-lingual text-to-speech. [link], 2024. Accessed: 2025-05-09.
Kelley, J. F. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems, 2(1), 26–41, 1984. DOI: 10.1145/357417.357420
Lazar, J. Feng, J. H. and Hochheiser, H. Research Methods in Human-Computer Interaction, 2nd ed. Morgan Kaufmann, 2017.
Publicado
30/09/2025
Como Citar
FERREIRA, Wesley Ferreira De; NEGRÃO, Matheus D.; MACIEL, Anderson; TORCHELSEN, Rafael Piccin; NEDEL, Luciana.
LLM vs. Human Driven Conversation: A User Study in Virtual Reality Medical Simulation. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 27. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 598-606.
