What if HMDs could generate audio descriptions of real scenes for visually impaired people?
Resumo
This paper explores the feasibility and effectiveness of using the Meta Quest 3 head-mounted display to generate real-time audio descriptions of real scenes for visually impaired individuals. The objective is to enhance the spatial awareness and interaction capabilities of visually impaired users by providing continuous, detailed audio descriptions of their surroundings. The methodology involves leveraging the Meta Quest 3 to capture real-time visual data, which is then processed and converted into audio descriptions using AI-driven software. Experiments were conducted in various real-world environments to test the accuracy and usability of the audio description system. Key findings indicate that the Meta Quest 3 can effectively deliver real-time audio descriptions with high accuracy, significantly improving the navigation and situational awareness of visually impaired users. This study demonstrates the potential of HMD technology to create more inclusive and accessible experiences, highlighting the importance of continued innovation in assistive technologies for the visually impaired.
Palavras-chave:
HMDs, Visually Impaired, Spatial Awareness, Accessibility
Referências
Manuel Birlo, PJ Eddie Edwards, Matthew Clarkson, and Danail Stoyanov. 2022. Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review. Medical Image Analysis 77 (2022), 102361.
I. J. Carvalho, J. R.; Pádua. 2015. As tecnologias assistivas para cegos: das potencialidades instrumentais aos limites impostos numa moral concreta fetichizada. In Pessoa com deficiência, educação e trabalho: reflexões críticas(incoll). EDUNIOESTE, Cascavel, PR, BR, 170–197.
Evgenia Gkini, Ioannis Voyiatzis, and Cleo Sgouropoulou. 2022. Head-Mounted Display Systems as Visual Aids for the Visually Impaired: A Survey. In Proceedings of the 25th Pan-Hellenic Conference on Informatics (Volos, Greece) (PCI ’21). Association for Computing Machinery, New York, NY, USA, 323–327.
Lilit Hakobyan, Jo Lumsden, Dympna O’Sullivan, and Hannah Bartlett. 2013. Mobile assistive technologies for the visually impaired. Survey of ophthalmology 58, 6 (2013), 513–528.
Hein Min Htike, Tom H Margrain, Yu-Kun Lai, and Parisa Eslambolchilar. 2020. Ability of head-mounted display technology to improve mobility in people with low vision: A systematic review. Translational Vision Science & Technology 9, 10 (2020), 26–26.
Hiroyuki Kawabe, Yuko Shimomura, Hidetaka Nambo, Shuichi Seto, and Hiroshi Arai. 2016. HMD system for visual field impaired students. Proceedings of the 16th Asia Pacific industrial engineering and management systems (2016).
Jinmo Kim. 2020. VIVR: Presence of immersive interaction for visual impairment virtual reality. IEEE Access 8 (2020), 196151–196159.
Steven M LaValle. 2023. Virtual reality. Cambridge university press.
Anatole Lécuyer, Pascal Mobuchon, Christine Mégard, Jérôme Perret, Claude Andriot, and J-P Colinot. 2003. HOMERE: a multimodal system for visually impaired people to explore virtual environments. In IEEE Virtual Reality, 2003. Proceedings. IEEE, 251–258.
Johnathan R Lex, Robert Koucheki, Jay Toor, and David J Backstein. 2023. Clinical applications of augmented reality in orthopaedic surgery: A comprehensive narrative review. International Orthopaedics 47, 2 (2023), 375–391.
Yifan Li, Kangsoo Kim, Austin Erickson, Nahal Norouzi, Jonathan Jules, Gerd Bruder, and Gregory F. Welch. 2022. A Scoping Review of Assistance and Therapy with Head-Mounted Displays for People Who Are Visually Impaired. ACM Trans. Access. Comput. 15, 3, Article 25 (aug 2022), 28 pages.
R Randall McKnight, Christian A Pean, J Stewart Buck, John S Hwang, Joseph R Hsu, and Sarah N Pierrie. 2020. Virtual reality and augmented reality—translating surgical training into surgical technique. Current reviews in musculoskeletal medicine 13 (2020), 663–674.
Naheem Noah, Sommer Shearer, and Sanchari Das. 2022. Security and privacy evaluation of popular augmented and virtual reality technologies. In Proceedings of the 2022 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence, and Neural Engineering (IEEE MetroXRAINE 2022).
M Saquib Sarfraz, Angela Constantinescu, Melanie Zuzej, and Rainer Stiefelhagen. 2017. A multimodal assistive system for helping visually impaired in social interactions. Informatik-Spektrum 40 (2017), 540–545.
Jens T Verhey, Jack M Haglin, Erik M Verhey, and David E Hartigan. 2020. Virtual, augmented, and mixed reality applications in orthopedic surgery. The International Journal of Medical Robotics and Computer Assisted Surgery 16, 2 (2020), e2067.
Peijie Xu, Gerard A Kennedy, Fei-Yi Zhao, Wen-Jing Zhang, and Ron Van Schyndel. 2023. Wearable obstacle avoidance electronic travel aids for blind and visually impaired individuals: A systematic review. IEEE Access (2023).
G Telli Yamamoto and Deniz Altun. 2021. Virtual reality (vr) technology in the future of military training. Unpublished manuscript (2021).
I. J. Carvalho, J. R.; Pádua. 2015. As tecnologias assistivas para cegos: das potencialidades instrumentais aos limites impostos numa moral concreta fetichizada. In Pessoa com deficiência, educação e trabalho: reflexões críticas(incoll). EDUNIOESTE, Cascavel, PR, BR, 170–197.
Evgenia Gkini, Ioannis Voyiatzis, and Cleo Sgouropoulou. 2022. Head-Mounted Display Systems as Visual Aids for the Visually Impaired: A Survey. In Proceedings of the 25th Pan-Hellenic Conference on Informatics (Volos, Greece) (PCI ’21). Association for Computing Machinery, New York, NY, USA, 323–327.
Lilit Hakobyan, Jo Lumsden, Dympna O’Sullivan, and Hannah Bartlett. 2013. Mobile assistive technologies for the visually impaired. Survey of ophthalmology 58, 6 (2013), 513–528.
Hein Min Htike, Tom H Margrain, Yu-Kun Lai, and Parisa Eslambolchilar. 2020. Ability of head-mounted display technology to improve mobility in people with low vision: A systematic review. Translational Vision Science & Technology 9, 10 (2020), 26–26.
Hiroyuki Kawabe, Yuko Shimomura, Hidetaka Nambo, Shuichi Seto, and Hiroshi Arai. 2016. HMD system for visual field impaired students. Proceedings of the 16th Asia Pacific industrial engineering and management systems (2016).
Jinmo Kim. 2020. VIVR: Presence of immersive interaction for visual impairment virtual reality. IEEE Access 8 (2020), 196151–196159.
Steven M LaValle. 2023. Virtual reality. Cambridge university press.
Anatole Lécuyer, Pascal Mobuchon, Christine Mégard, Jérôme Perret, Claude Andriot, and J-P Colinot. 2003. HOMERE: a multimodal system for visually impaired people to explore virtual environments. In IEEE Virtual Reality, 2003. Proceedings. IEEE, 251–258.
Johnathan R Lex, Robert Koucheki, Jay Toor, and David J Backstein. 2023. Clinical applications of augmented reality in orthopaedic surgery: A comprehensive narrative review. International Orthopaedics 47, 2 (2023), 375–391.
Yifan Li, Kangsoo Kim, Austin Erickson, Nahal Norouzi, Jonathan Jules, Gerd Bruder, and Gregory F. Welch. 2022. A Scoping Review of Assistance and Therapy with Head-Mounted Displays for People Who Are Visually Impaired. ACM Trans. Access. Comput. 15, 3, Article 25 (aug 2022), 28 pages.
R Randall McKnight, Christian A Pean, J Stewart Buck, John S Hwang, Joseph R Hsu, and Sarah N Pierrie. 2020. Virtual reality and augmented reality—translating surgical training into surgical technique. Current reviews in musculoskeletal medicine 13 (2020), 663–674.
Naheem Noah, Sommer Shearer, and Sanchari Das. 2022. Security and privacy evaluation of popular augmented and virtual reality technologies. In Proceedings of the 2022 IEEE International Conference on Metrology for eXtended Reality, Artificial Intelligence, and Neural Engineering (IEEE MetroXRAINE 2022).
M Saquib Sarfraz, Angela Constantinescu, Melanie Zuzej, and Rainer Stiefelhagen. 2017. A multimodal assistive system for helping visually impaired in social interactions. Informatik-Spektrum 40 (2017), 540–545.
Jens T Verhey, Jack M Haglin, Erik M Verhey, and David E Hartigan. 2020. Virtual, augmented, and mixed reality applications in orthopedic surgery. The International Journal of Medical Robotics and Computer Assisted Surgery 16, 2 (2020), e2067.
Peijie Xu, Gerard A Kennedy, Fei-Yi Zhao, Wen-Jing Zhang, and Ron Van Schyndel. 2023. Wearable obstacle avoidance electronic travel aids for blind and visually impaired individuals: A systematic review. IEEE Access (2023).
G Telli Yamamoto and Deniz Altun. 2021. Virtual reality (vr) technology in the future of military training. Unpublished manuscript (2021).
Publicado
30/09/2024
Como Citar
MAURICIO, Claudio; DOMINGUES, Gustavo; PADUA, Ivã; PERES, Fabiana; TEIXEIRA, João Marcelo.
What if HMDs could generate audio descriptions of real scenes for visually impaired people?. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 26. , 2024, Manaus/AM.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 203-212.
