Enhancing Social Presence in VR with Emotionally Expressive, Knowledge-Grounded Avatars - Oliveira, Elisa A. M.; Webster, Gustavo; Negrão, Matheus D.; Cintra, Luiza M. F.; Sousa, Rafael Teixeira; Galvão Filho, Arlindo Rodrigues (Anais do Simpósio de Realidade Virtual e Aumentada (SVR); 2025: Anais do XXVII Simpósio de Realidade Virtual e Aumentada, 2025)
Large language models (LLMs) and generative AI are revolutionizing avatars, transforming them from static systems to intelligent, adaptive virtual agents capable of real-time interaction. This paper presents an advanced avatar system that brings Romeo, the iconic character from Shakespeare’s works, to life. The avatar leverages LLMs, Retrieval-Augmented Generation (RAG), and sentiment analysis to create a dynamic and emotionally expressive virtual representation. It generates contextually accurate responses based on a tailored database of Shakespeare’s works, adapting its tone and expressions to enhance user engagement. The system employs GPT-4o mini for dialogue generation, RAG for precise knowledge retrieval, and sentiment analysis to control Romeo’s facial expressions in response to emotional cues. Designed with Blender and animated using Mixamo, the avatar embodies a cartoon-inspired aesthetic that ensures approachability and supports lifelike speech-driven lip-sync and animations. This integration delivers fluid, context-aware interactions, making Romeo an innovative tool for education, entertainment, and cultural preservation. By combining advanced AI capabilities with immersive technologies, this Romeo avatar exemplifies the future of human-machine interaction. It redefines virtual agents by enabling intelligent, emotionally expressive, and highly interactive experiences, offering a transformative approach to engaging with Shakespeare’s timeless stories.