Conversational Historical Avatars in Virtual Reality Powered by Local Large Language Models
Resumo
This paper presents Avataredu, an open-source framework for creating intelligent, historically themed avatars inside virtual-reality (VR) environments. It comprehends an end-to-end architecture that combines on-device AI inference with a multiplayer VR stack (Ubiq). Our prototype instantiates an Avatar powered by a local large language model (LLM), Whisper speech-to-text, and a lightweight text-to-speech (TTS) engine, running entirely on commodity hardware. We provide a fully documented codebase, a dataset of interaction logs, and replication instructions. We conducted a quantitative cross-persona analysis (accuracy + latency) using avatars such as Marie Curie, Malala Yousafzai, and Martin Luther King Jr to validate its applicability. Our analysis revealed low error rates, with a mean of ∼ 9.3%, and that persona swap requires minimal additional computational power while maintaining answer quality. The full source code, including how to handle the Unity project files, Python backend scripts, and integration assets, is openly available at: https://github.com/Gvascons/intelligent-avatars-vr.
Referências
R. Clark and R. Mayer. E-Learning and the Science of Instruction, 4th ed. Wiley, 2016.
S. Hobert and R. Meyer. Say hello to your new automated tutor: A structured literature review on pedagogical conversational agents. 2019.
G. D. Voinea et al. Study of social presence while interacting in metaverse with an augmented avatar during autonomous driving. Applied Sciences, 12(22), 11804, 2022.
C. Kyrlitsias and D. Michael-Grigoriou. Social interaction with agents and avatars in immersive virtual environments: A survey. Frontiers in Virtual Reality, 2, 786665, 2022.
S. Bubeck et al. Sparks of artificial general intelligence. arXiv:2303.12712, 2023.
D. Driess et al. PaLM-E: An embodied multimodal language model. arXiv:2303.03378, 2023.
M. C. Fink, S. A. Robinson, and B. Ertl. AI-based avatars are changing the way we learn and teach: benefits and challenges. 2024.
F. Weidner et al. A systematic review on the visualization of avatars and agents in AR and VR displayed using head-mounted displays. IEEE Transactions on Visualization and Computer Graphics, 29(5), 2596–2606, May 2023.
A. Venkatesh et al. On Evaluating and Comparing Open Domain Dialog Systems. Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), 2018.
Nobel Foundation. Marie Curie — Facts. 1903. [link]
Nobel Foundation. Malala Yousafzai — Biographical. 2014. [link]
Nobel Foundation. Questions and Answers: Martin Luther King Jr. 1964. [link]
Ready Player Me. Create your personal 3D avatar. [link]
Adobe Mixamo. Free 3D character animations. [link]
