Shared Reconstructed Environments in Extended Reality

• André Vital Tavernaro Insper http://orcid.org/0009-0006-6434-0997
• Matheus Freitas Sant’Ana Insper
• Henrique Mualem Marti Insper https://orcid.org/0009-0000-1645-876X
• João Guilherme Cintra de Freitas Almeida Insper http://orcid.org/0009-0004-0214-3002
• Luciano Pereira Soares Insper https://orcid.org/0000-0003-3187-4168

Resumo

Modern extended reality headsets can identify the geometry of the physical surrounding space, which can be shared and synchronized with remote users, allowing them to view the space through their own extended reality headsets from different points of view. Conversely, remote users can augment this shared space with additional virtual geometry. For instance, imagine wearing an augmented reality (AR) headset that identifies and shares your surrounding space with a remotely located interior designer. This interior designer can then virtually teleport into your environment and suggest furniture placements, which you can see in real-time through your AR glasses. Moreover, you can interact with each other via avatars representing each user. This research project demonstrates how two people in different locations can experience one’s real physical environment in real-time and interact with each other. The strategies used to share geometry and user poses are discussed, along with the limitations of the hardware used for the prototype. The developed prototype involves one user wearing a Magic Leap 1 headset to scan their space, enabling a second user to remotely explore this space using an HTC Vive virtual reality (VR) headset. The VR user can also add virtual objects that both users can see.

Palavras-chave: Shared Virtual Environment, Digital Twin, Augmented Reality, Virtual Reality, Extended Reality, Virtual Collaboration, Metaverse

Referências

Botín-Sanabria, Diego M., Adriana-Simona Mihaita, Rodrigo E. Peimbert-García, Mauricio A. Ramírez-Moreno, Ricardo A. Ramírez-Mendoza, and Jorge de J. Lozoya-Santos. 2022. Digital Twin Technology Challenges and Applications: A Comprehensive Review. Remote Sensing, 14(6): 1335. DOI: 10.3390/rs14061335

Bhaskar Raman. 2022. How digital twins are helping build the foundation of industrial metaverse. Gulf Business. Available at: [link]

Charles Thomson. 2019. The cost of commissioning a 3D point cloud survey: 4 things you need to know when speaking with surveyors. Vercator. Available at: [link]

8i. The Importance of Real-Time Holograms. Available at: [link]

Manolova, A., Tonchev, K., Poulkov, V., et al. 2021. Context-Aware Holographic Communication Based on Semantic Knowledge Extraction. Wireless Pers Commun, 120: 2307–2319. DOI: 10.1007/s11277-021-08560-7

Minnalearn. 2024. Introduction to extended reality: AR, VR and MR. Available at: [link]

Gamify. 2018. The Reality of AR, VR and MR Technology. Available at: [link]

Joan Sol Roo and Martin Hachet. 2017. One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). Association for Computing Machinery, New York, NY, USA, 787–795. DOI: 10.1145/3126594.3126638

Microsoft. Holoportation Research. Available at: [link]

B. Anand, V. Barsaiyan, M. Senapati and P. Rajalakshmi. 2019. Real Time LiDAR Point Cloud Compression and Transmission for Intelligent Transportation System. 2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring), Kuala Lumpur, Malaysia, 2019, pp. 1-5. DOI: 10.1109/VTCSpring.2019.8746417,

Khoshelham, K., Tran, H., and Acharya, D. 2019. INDOOR MAPPING EYEWEAR: GEOMETRIC EVALUATION OF SPATIAL MAPPING CAPABILITY OF HOLOLENS. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-2/W13, 805–810. DOI: 10.5194/isprs-archives-XLII-2-W13-805-2019

Magic Leap 1 devices, equipment, and sensors. Available at: [link]

Adam Sinicki. 2021. What is Unity? Everything you need to know. Android Authority. Available at: [link]

Khronos Groups. OpenXR. [link]

Photon. Photon Engine. [link]

STA. WebSocket Sharp. [link]

Magic Leap. Install the Magic Leap Unity Package. Available at: [link]

Unity PUN package. Available at: [link]

Photon Monobehaviour PUN callbacks. Available at: [link]

Unity Photon Tutorial. Available at: [link]