Enhancing Social Presence in VR with Emotionally Expressive, Knowledge-Grounded Avatars
Resumo
Large language models (LLMs) and generative AI are revolutionizing avatars, transforming them from static systems to intelligent, adaptive virtual agents capable of real-time interaction. This paper presents an advanced avatar system that brings Romeo, the iconic character from Shakespeare’s works, to life. The avatar leverages LLMs, Retrieval-Augmented Generation (RAG), and sentiment analysis to create a dynamic and emotionally expressive virtual representation. It generates contextually accurate responses based on a tailored database of Shakespeare’s works, adapting its tone and expressions to enhance user engagement. The system employs GPT-4o mini for dialogue generation, RAG for precise knowledge retrieval, and sentiment analysis to control Romeo’s facial expressions in response to emotional cues. Designed with Blender and animated using Mixamo, the avatar embodies a cartoon-inspired aesthetic that ensures approachability and supports lifelike speech-driven lip-sync and animations. This integration delivers fluid, context-aware interactions, making Romeo an innovative tool for education, entertainment, and cultural preservation. By combining advanced AI capabilities with immersive technologies, this Romeo avatar exemplifies the future of human-machine interaction. It redefines virtual agents by enabling intelligent, emotionally expressive, and highly interactive experiences, offering a transformative approach to engaging with Shakespeare’s timeless stories.
Palavras-chave:
Virtual Reality, Avatar Interaction, Artificial Intelligence, Virtual Agents
Referências
W. Sadowski and K. Stanney. Presence in virtual environments. In Handbook of Virtual Environments, CRC Press, 2002, 831–846.
M. Slater and S. Wilbur. A framework for immersive virtual environments (five) Speculations on the role of presence in virtual environments. Presence Teleoperators and Virtual Environments, 6(6):603–616, 1997.
G. Riva, B. K. Wiederhold, and F. Mantovani. Neuroscience of virtual reality From virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. 22(1):82–96, 2019.
J. Cassell. Embodied conversational agents representation and intelligence in user interfaces. AI Magazine, 22(4):67–67, 2001.
E. Adamopoulou and L. Moussiades. An overview of chatbot technology. In Artificial Intelligence Applications and Innovations 16th IFIP WG 12.5 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16. Springer, 2020, 373–383.
W. Zhu, X. Fan, and Y. Zhang. Applications and research trends of digital human models in the manufacturing industry. Virtual Reality & Intelligent Hardware, 1(6):558–579, 2019.
K. Loveys, M. Sagar, M. Billinghurst, N. Saffaryazdi, and E. Broadbent. Exploring empathy with digital humans. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 2022, 233–237.
J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, 1–22.
S. Zhang, M. Ma, Y. Li, K. L. Man, J. Smith, and Y. Yue. The effects of LLM-empowered chatbots and avatar guides on the engagement, experience, and learning in virtual museums. Int. J. Hum. Comput. Interact. 0(0):1–13, May 2025.
H. Jin, M. Yoo, J. Park, Y. Lee, X. Wang, and J. Kim. TeachTune Reviewing pedagogical agents against diverse student profiles with simulated students. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. ACM, Apr. 2025, 1–28.
M. Shanahan, K. McDonell, and L. Reynolds. Role play with large language models. Nature, 623(7987):493–498, 2023.
Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang. Retrieval-augmented generation for large language models A survey. arXiv preprint arXiv:2312.10997, 2(1), 2023.
S. Wu, Y. Xiong, Y. Cui, H. Wu, C. Chen, Y. Yuan, L. Huang, X. Liu, T.-W. Kuo, N. Guan et al. Retrieval-augmented generation for natural language processing a survey. arXiv preprint arXiv:2407.13193, 2024.
Z. Li, G. Chen, R. Shao, Y. Xie, D. Jiang, and L. Nie. Enhancing emotional generation capability of large language models via emotional chain-of-thought. arXiv preprint arXiv:2401.06836, 2024.
OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
A. Trivedi, N. Pant, P. Shah, S. Sonik, and S. Agrawal. Speech to text and text to speech recognition systems A review. IOSR J. Comput. Eng. 20(2):36–43, 2018.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning. PMLR, 2023, 28492–28518.
G. Llorach, A. Evans, J. Blat, G. Grimm, and V. Hohmann. Web-based live speech-driven lip-sync. In 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES). IEEE, 2016, 1–4.
C. Nass and S. Brave. Wired for speech How voice activates and advances the human-computer relationship. MIT Press, 2005.
B. Reeves and C. Nass. The media equation How people treat computers, television, and new media like real people. Cambridge, UK, 10(10):19–36, 1996.
S. Pandita. Affective embodiment The effect of avatar appearance and posture representation on emotions in VR. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2020, 539–540.
J. Kim, J. Lee, Y. Kim, S. Park, and H. Lee. Exploring the uncanny valley of realistic avatars in virtual reality. Computers in Human Behavior, 131:107246, 2023.
K. L. Nowak and F. Biocca. The effect of the agency and anthropomorphism on users sense of telepresence, copresence, and social presence in virtual environments. Presence Teleoperators and Virtual Environments, 12(5):481–494, 2003.
S. Fox, J. Arena, and M. Bailenson. Virtual reality Theories, research, and applications. Routledge, 2022.
C. B. Araujo and M. A. Santos. Social presence and virtual environments A review. Presence Teleoperators and Virtual Environments, 31(3):247–266, 2022.
G. Bente, S. Ruggenberg, N. C. Kramer, and F. Eschenburg. Avatar-mediated networking The impact of avatar realism and eye gaze control on perceived quality of communication in virtual environments. Human Communication Research, 34(3):298–318, 2008.
N. Yee and J. N. Bailenson. The proteus effect The effect of transformed self-representation on behavior. Human Communication Research, 33(3):271–290, 2007.
H. Blascovich and J. Bailenson. Infinite Reality Avatars, Eternal Life, New Worlds, and the Dawn of the Virtual Revolution. HarperCollins, 2011.
R. Schroeder. Social theory after the internet Media, technology, and globalization. UCL Press, 2018.
Y. Sun, Z. Zhang, H. Wang, C. Zhang, and L. Zhao. Towards emotionally expressive virtual agents A survey of emotion generation in human-agent interaction. ACM Computing Surveys, 57(4):1–36, 2024.
R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 18(1):32–80, 2001.
P. Ekman. An argument for basic emotions. Cognition and Emotion, 6(3–4):169–200, 1992.
J. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161–1178, 1980.
K. R. Scherer. Appraisal processes in emotion theory, methods, research. Oxford University Press, 2001.
R. Plutchik. The nature of emotions. American Scientist, 89(4):344–350, 2001.
C. Bartneck, D. Kulic, E. Croft, and S. Zoghbi. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, 1(1):71–81, 2009.
J. Heer. Agency and automation in data visualization. Communications of the ACM, 66(9):76–87, 2023.
R. Held and N. Durlach. Telepresence. Presence Teleoperators and Virtual Environments, 1(1):109–112, 1992.
M. C. Whitton. Making virtual environments compelling. Communications of the ACM, 46(7):40–47, 2003.
A. B. Vila Nova, C. A. Vidal, J. B. Cavalcante-Neto, and L. O. Carneiro. Facial conformation modeling by anthropometric measurements. Proceedings of the 26th Symposium on Virtual and Augmented Reality (SVR), 97–102, 2024.
A. Jaoua, E. Negri, S. Bhar Layeb, and Z. Ayed. Framework for smart online 3D bin packing using augmented reality. IC ASET, IEEE, 01–06, 2023.
T. Porcino, D. Reilly, E. Clua, and D. Trevisan. A guideline proposal for minimizing cybersickness in VR-based serious games and applications. In 2022 IEEE 10th International Conference on Serious Games and Applications for Health (SeGAH), IEEE, 1–8, 2022.
Meta Platforms Inc. Meta Quest 3 mixed/virtual reality headset. Available [link], 2024.
OpenAI. ChatGPT Plus. Product documentation. Available [link], accessed July 2025.
M. Mondellini, S. Arlati, H. Gapeyeva, K. Lees, I. Maritz, S. L. Pizzagalli, T. Otto, M. Sacco, and A. Teder-Braschinsky. User experience during an immersive virtual reality-based cognitive task A comparison between Estonian and Italian older adults with MCI. Sensors, 22(21):8249, 2022.
S. Arlati, D. Spoladore, D. Baldassini, M. Sacco, and L. Greci. VirtualCruiseTour An AR/VR application to promote shore excursions on cruise ships. In Augmented Reality, Virtual Reality, and Computer Graphics 5th International Conference AVR 2018, Otranto, Italy, Proceedings, Part I 5, Springer, 133–147, 2018.
M. Slater and S. Wilbur. A framework for immersive virtual environments (five) Speculations on the role of presence in virtual environments. Presence Teleoperators and Virtual Environments, 6(6):603–616, 1997.
G. Riva, B. K. Wiederhold, and F. Mantovani. Neuroscience of virtual reality From virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. 22(1):82–96, 2019.
J. Cassell. Embodied conversational agents representation and intelligence in user interfaces. AI Magazine, 22(4):67–67, 2001.
E. Adamopoulou and L. Moussiades. An overview of chatbot technology. In Artificial Intelligence Applications and Innovations 16th IFIP WG 12.5 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16. Springer, 2020, 373–383.
W. Zhu, X. Fan, and Y. Zhang. Applications and research trends of digital human models in the manufacturing industry. Virtual Reality & Intelligent Hardware, 1(6):558–579, 2019.
K. Loveys, M. Sagar, M. Billinghurst, N. Saffaryazdi, and E. Broadbent. Exploring empathy with digital humans. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 2022, 233–237.
J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, 1–22.
S. Zhang, M. Ma, Y. Li, K. L. Man, J. Smith, and Y. Yue. The effects of LLM-empowered chatbots and avatar guides on the engagement, experience, and learning in virtual museums. Int. J. Hum. Comput. Interact. 0(0):1–13, May 2025.
H. Jin, M. Yoo, J. Park, Y. Lee, X. Wang, and J. Kim. TeachTune Reviewing pedagogical agents against diverse student profiles with simulated students. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. ACM, Apr. 2025, 1–28.
M. Shanahan, K. McDonell, and L. Reynolds. Role play with large language models. Nature, 623(7987):493–498, 2023.
Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang. Retrieval-augmented generation for large language models A survey. arXiv preprint arXiv:2312.10997, 2(1), 2023.
S. Wu, Y. Xiong, Y. Cui, H. Wu, C. Chen, Y. Yuan, L. Huang, X. Liu, T.-W. Kuo, N. Guan et al. Retrieval-augmented generation for natural language processing a survey. arXiv preprint arXiv:2407.13193, 2024.
Z. Li, G. Chen, R. Shao, Y. Xie, D. Jiang, and L. Nie. Enhancing emotional generation capability of large language models via emotional chain-of-thought. arXiv preprint arXiv:2401.06836, 2024.
OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
A. Trivedi, N. Pant, P. Shah, S. Sonik, and S. Agrawal. Speech to text and text to speech recognition systems A review. IOSR J. Comput. Eng. 20(2):36–43, 2018.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning. PMLR, 2023, 28492–28518.
G. Llorach, A. Evans, J. Blat, G. Grimm, and V. Hohmann. Web-based live speech-driven lip-sync. In 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES). IEEE, 2016, 1–4.
C. Nass and S. Brave. Wired for speech How voice activates and advances the human-computer relationship. MIT Press, 2005.
B. Reeves and C. Nass. The media equation How people treat computers, television, and new media like real people. Cambridge, UK, 10(10):19–36, 1996.
S. Pandita. Affective embodiment The effect of avatar appearance and posture representation on emotions in VR. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2020, 539–540.
J. Kim, J. Lee, Y. Kim, S. Park, and H. Lee. Exploring the uncanny valley of realistic avatars in virtual reality. Computers in Human Behavior, 131:107246, 2023.
K. L. Nowak and F. Biocca. The effect of the agency and anthropomorphism on users sense of telepresence, copresence, and social presence in virtual environments. Presence Teleoperators and Virtual Environments, 12(5):481–494, 2003.
S. Fox, J. Arena, and M. Bailenson. Virtual reality Theories, research, and applications. Routledge, 2022.
C. B. Araujo and M. A. Santos. Social presence and virtual environments A review. Presence Teleoperators and Virtual Environments, 31(3):247–266, 2022.
G. Bente, S. Ruggenberg, N. C. Kramer, and F. Eschenburg. Avatar-mediated networking The impact of avatar realism and eye gaze control on perceived quality of communication in virtual environments. Human Communication Research, 34(3):298–318, 2008.
N. Yee and J. N. Bailenson. The proteus effect The effect of transformed self-representation on behavior. Human Communication Research, 33(3):271–290, 2007.
H. Blascovich and J. Bailenson. Infinite Reality Avatars, Eternal Life, New Worlds, and the Dawn of the Virtual Revolution. HarperCollins, 2011.
R. Schroeder. Social theory after the internet Media, technology, and globalization. UCL Press, 2018.
Y. Sun, Z. Zhang, H. Wang, C. Zhang, and L. Zhao. Towards emotionally expressive virtual agents A survey of emotion generation in human-agent interaction. ACM Computing Surveys, 57(4):1–36, 2024.
R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 18(1):32–80, 2001.
P. Ekman. An argument for basic emotions. Cognition and Emotion, 6(3–4):169–200, 1992.
J. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161–1178, 1980.
K. R. Scherer. Appraisal processes in emotion theory, methods, research. Oxford University Press, 2001.
R. Plutchik. The nature of emotions. American Scientist, 89(4):344–350, 2001.
C. Bartneck, D. Kulic, E. Croft, and S. Zoghbi. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, 1(1):71–81, 2009.
J. Heer. Agency and automation in data visualization. Communications of the ACM, 66(9):76–87, 2023.
R. Held and N. Durlach. Telepresence. Presence Teleoperators and Virtual Environments, 1(1):109–112, 1992.
M. C. Whitton. Making virtual environments compelling. Communications of the ACM, 46(7):40–47, 2003.
A. B. Vila Nova, C. A. Vidal, J. B. Cavalcante-Neto, and L. O. Carneiro. Facial conformation modeling by anthropometric measurements. Proceedings of the 26th Symposium on Virtual and Augmented Reality (SVR), 97–102, 2024.
A. Jaoua, E. Negri, S. Bhar Layeb, and Z. Ayed. Framework for smart online 3D bin packing using augmented reality. IC ASET, IEEE, 01–06, 2023.
T. Porcino, D. Reilly, E. Clua, and D. Trevisan. A guideline proposal for minimizing cybersickness in VR-based serious games and applications. In 2022 IEEE 10th International Conference on Serious Games and Applications for Health (SeGAH), IEEE, 1–8, 2022.
Meta Platforms Inc. Meta Quest 3 mixed/virtual reality headset. Available [link], 2024.
OpenAI. ChatGPT Plus. Product documentation. Available [link], accessed July 2025.
M. Mondellini, S. Arlati, H. Gapeyeva, K. Lees, I. Maritz, S. L. Pizzagalli, T. Otto, M. Sacco, and A. Teder-Braschinsky. User experience during an immersive virtual reality-based cognitive task A comparison between Estonian and Italian older adults with MCI. Sensors, 22(21):8249, 2022.
S. Arlati, D. Spoladore, D. Baldassini, M. Sacco, and L. Greci. VirtualCruiseTour An AR/VR application to promote shore excursions on cruise ships. In Augmented Reality, Virtual Reality, and Computer Graphics 5th International Conference AVR 2018, Otranto, Italy, Proceedings, Part I 5, Springer, 133–147, 2018.
Publicado
30/09/2025
Como Citar
OLIVEIRA, Elisa A. M.; WEBSTER, Gustavo; NEGRÃO, Matheus D.; CINTRA, Luiza M. F.; SOUSA, Rafael Teixeira; GALVÃO FILHO, Arlindo Rodrigues.
Enhancing Social Presence in VR with Emotionally Expressive, Knowledge-Grounded Avatars. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 27. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 246-252.
