Improving VR Accessibility Through Automatic 360 Scene Description Using Multimodal Large Language Models - Oliveira, Elisa Ayumi Masasi de; Silva, Diogo Fernandes Costa; Galvão Filho, Arlindo Rodrigues (Anais do Simpósio de Realidade Virtual e Aumentada (SVR); 2024: Anais do XXVI Simpósio de Realidade Virtual e Aumentada, 2024)
Advancements in Virtual Reality (VR) technology hold immense promise for enriching immersive experiences. Despite the advancements in VR technology, there remains a significant gap in addressing accessibility concerns, particularly in automatically providing descriptive information for VR scenes. This paper combines the potential of leveraging Multimodal Large Language Models (MLLMs) to automatically generate text descriptions for 360 VR scenes according to Speech-to-Text (STT) prompts. As a case study, we conduct experiments on educational settings in VR museums, improving dynamic experiences across various contexts. Despite minor challenges in adapting MLLMs to VR Scenes, the experiments demonstrate that they can generate descriptions with high quality. Our findings provide insights for enhancing VR experiences and ensuring accessibility to individuals with disabilities or diverse needs.