Augmented Reality Support for Oncological Neurosurgery

• Ana Clara Carneiro de Freitas Insper http://orcid.org/0009-0002-7802-0258
• Beatriz Muniz de Castro e Silva Insper http://orcid.org/0009-0009-2939-1512
• Beatriz Rianho Bernardino Insper https://orcid.org/0009-0004-8288-7094
• Gabriela Moreno Boriero Insper https://orcid.org/0009-0005-0517-0660
• Marcio Tiago de Oliveira Barbosa A. C. Camargo Cancer Center http://orcid.org/0009-0006-1814-3917
• Rubens Chojniak A. C. Camargo Cancer Center https://orcid.org/0000-0002-8096-252X
• Luciano Pereira Soares Insper https://orcid.org/0000-0003-3187-4168

Resumo

This project aims to enhance oncological neurosurgery by displaying brain tumors using Augmented Reality (AR) as an overlay image during surgical procedures. Its purpose is to assist neurosurgeons in tasks such as brain biopsies by providing a clear and precise visualization of the affected area, thus enabling faster and less invasive surgeries, ultimately offering a more comfortable experience for patients. Currently, medical procedures achieve this through neuronavigation systems, which display two-dimensional images of the patient’s brain on a screen, typically positioned beside the patient or in separate imaging support rooms. By utilizing AR, surgeons can directly visualize the tumor on the patient’s body through a virtual 3D model, eliminating the need to frequently refer to external images. This project was developed using Microsoft HoloLens, a specialized AR headset, and Unity, a game engine widely used for 3D environment development. The system uses DICOM files as input, which are converted into 3D objects displayed through the AR headset.

Palavras-chave: Augmented Reality, Brain Tumor, DICOM, Neurosurgery, Motion Tracking, Neuronavigation

Referências

Tori, Romero; Hounsell, Marcelo da Silva (org.). 2018. Introdução a Realidade Virtual e Aumentada. Porto Alegre: Editora SBC.

Macedo Filho, Evaldo Dacheux de. 2021. Microcirurgia de laringe: técnica cirúrgica em realidade aumentada. Rio de Janeiro: Thieme Revinter.

Maniglia, João Jairney; Maniglia, Fábio F.; Maniglia, Ricardo Fabricio. 2021. Blefaroplastia: Em Realidade Aumentada. Rio de Janeiro: Thieme Revinter.

Mocellin, Marcos; Patrocinio, José A. 2021. Rinoplastia: ponta nasal em realidade aumentada. Rio de Janeiro: Thieme Revinter.

Masi, Elen de. 2021. Cirurgia Plástica Facial: Em Realidade Aumentada. Rio de Janeiro: Thieme Revinter.

Tagaytayan, Raniel; Kelemen, Arpad; Sik-Lanyi, Cecilia. 2018. Augmented reality in neurosurgery. Archives of Medical Science, v. 14, n. 3, p. 572–578.

Mildenberger, Peter; Eichelberg, Marco; Martin, Eric. 2001. Introduction to the DICOM standard. European Radiology, v. 12, n. 4, p. 920–927.

Marcus, Hani J. et al. 2015. Technological Innovation in Neurosurgery: A Quantitative Study. Journal of neurosurgery, v. 123, n. 1, p. 174–181.

Cobb, Mary I.-P. H. et al. 2016. Simulation in Neurosurgery—A Brief Review and Commentary. World Neurosurgery, v. 89, p. 583–586.

Kockro RA, Serra L, Tseng-Tsai Y, Chan C, Yih-Yian S, Gim-Guan C, Lee E, Hoe LY, Hern N, Nowinski WL. 2000. Planning and simulation of neurosurgery in a virtual reality environment. Neurosurgery, 46(1):118-35.

Lee, Chester; Wong, George K. C. 2019. Virtual reality and augmented reality in the management of intracranial tumors: A review. Journal of Clinical Neuroscience, v. 62, p. 14–20.

Kersten-Oertel, Marta; Jannin, Pierre; Collins, D. Louis. 2013. The state of the art of visualization in mixed reality image guided surgery. Computerized Medical Imaging and Graphics, v. 37, n. 2, p. 98–112.

Mahvash, Mehran; Besharati Tabrizi, Leila. 2013. A novel augmented reality system of image projection for image-guided neurosurgery. Acta Neurochirurgica, v. 155, n. 5, p. 943–947.

Satoh, Makoto et al. 2021. Evaluation of augmented-reality based navigation for brain tumor surgery. Journal of Clinical Neuroscience, v. 94, p. 305–314.

Hübner, Patrick et al. 2020. Evaluation of HoloLens Tracking and Depth Sensing for Indoor Mapping Applications. Sensors, v. 20, n. 4, p. 1021.

Sun, Qichang et al. 2020. Fast and accurate online calibration of optical see-through head-mounted display for AR-based surgical navigation using Microsoft HoloLens. International Journal of Computer Assisted Radiology and Surgery, v. 15, n. 11, p. 1907–1919.

Haxthausen, Felix Von; Chen, Yenjung; Ernst, Floris. 2021. Superimposing holograms on real world objects using HoloLens 2 and its depth camera. Current Directions in Biomedical Engineering, v. 7, n. 1, p. 111–115.

American Cancer Society. 2022. Key Statistics for Brain and Spinal Cord Tumors. Available at: [link]. Accessed 2024 June 29.

Instituto Nacional de Câncer – INCA. 2022. Estatísticas de câncer. Available at: [link]. Accessed 2024 Jun 29.

Gradisnik, Lidija & Bosnjak, Roman & Bunc, Gorazd & Ravnik, Janez & Maver, Tina & Velnar, Tomaž. 2021. Neurosurgical Approaches to Brain Tissue Harvesting for the Establishment of Cell Cultures in Neural Experimental Cell Models. Materials. 14. 6857. 10.3390/ma14226857.

PTC. 2024. Vuforia. [link].

Vicon. 2022. ACCURACY IN MOTION. [link].

Frantz, Taylor et al. 2018. Augmenting Microsoft’s HoloLens with vuforia tracking for neuronavigation. Healthcare Technology Letters, v. 5, n. 5, p. 221–225.

Unity. [link].

Amorim, Paulo H.J; Moraes, Thiago F. de; Azevedo, Fábio de S; Silva, Jorge V.L. da. InVesalius: Software Livre de Imagens Médicas. Available at: [link]. Accessed 2024 June 29.