Immersive Visualization Interface for Endoscopy Analytics and Debriefing
Resumo
Endoscopic surgery is a complex procedure and is challenging for students to learn without risking the patient. Meanwhile, simulators have been developed to provide a safe environment for students to practice and learn, guaranteeing the patient’s safety. During a surgery simulation, many data are produced, including the patient’s vital signs and the indicators of the user’s performance, i.e., the endoscope operation. Combining both data in a timeline provides a picture of the patient’s condition. This work proposes a new perspective for immersive visualization of an endoscopic surgery simulation by integrating a virtual reality simulator with a series of panels for presenting data collected during the simulation. Additionally, after the surgery training, one may graphically analyze and reason the simulation session in a timeline without leaving the immersive environment. This practice, so-called debriefing, is usual in any medical practice. The proposed interface can potentially enhance the comprehension of a patient’s state during surgical procedures. Medical practitioners and surgeons can also use it to evaluate medical students’ proficiency, leading to possible changes in surgical training and providing new tools and resources for medical education.
Palavras-chave:
virtual reality, immersive visualization, endoscopy simulation
Referências
Lemonia Argyriou, Daphne Economou, and Vassiliki Bouki. 2020. Design methodology for 360 immersive video applications: the case study of a cultural heritage virtual tour. Personal and Ubiquitous Computing 24 (2020), 843–859.
Francesca Bonetti, Gary Warnaby, and Lee Quinn. 2018. Augmented reality and virtual reality in physical and online retailing: A review, synthesis and research agenda. Augmented reality and virtual reality: Empowering human, place and business (2018), 119–132.
Aaron Bray, Jeffrey B. Webb, Andinet Enquobahrie, Jared Vicory, Jerry Heneghan, Robert Hubal, Stephanie TerMaath, Philip Asare, and Rachel B. Clipp. 2019. Pulse Physiology Engine: an Open-Source Software Platform for Computational Modeling of Human Medical Simulation. SN Comprehensive Clinical Medicine 1 (5 2019), 362–377. Issue 5. https://doi.org/10.1007/s42399-019-00053-w
João Cartucho, David Shapira, Hutan Ashrafian, and Stamatia Giannarou. 2020. Multimodal mixed reality visualisation for intraoperative surgical guidance. International Journal of Computer Assisted Radiology and Surgery 15 (5 2020), 819–826. Issue 5. https://doi.org/10.1007/s11548-020-02165-4
Oculus Developers. 2023. Oculus Developers: Interaction SDK Overview. [link]. Accessed: May 24, 2023.
Ciro Donalek, S. G. Djorgovski, Alex Cioc, Anwell Wang, Jerry Zhang, Elizabeth Lawler, Stacy Yeh, Ashish Mahabal, Matthew Graham, Andrew Drake, Scott Davidoff, Jeffrey S. Norris, and Giuseppe Longo. 2014. Immersive and collaborative data visualization using virtual reality platforms. 2014 IEEE International Conference on Big Data (Big Data), 609–614. https://doi.org/10.1109/BigData.2014.7004282
Bu’Hussain Hayee, David S. Rowbotham, Vishal Saxena, and Alistair Mcnair. 2006. Pain Assessment During Colonoscopy: Who Guides Sedation? Gastrointestinal Endoscopy 63 (4 2006), AB197. Issue 5. https://doi.org/10.1016/j.gie.2006.03.464
Yark Hazewinkel and Evelien Dekker. 2011. Colonoscopy: basic principles and novel techniques. Nature Reviews Gastroenterology & Hepatology 8 (10 2011), 554–564. Issue 10. https://doi.org/10.1038/nrgastro.2011.141
Wee Sim Khor, Benjamin Baker, Kavit Amin, Adrian Chan, Ketan Patel, and Jason Wong. 2016. Augmented and virtual reality in surgery—the digital surgical environment: applications, limitations and legal pitfalls. Annals of translational medicine 4, 23 (2016).
Matthias Kraus, Johannes Fuchs, Björn Sommer, Karsten Klein, Ulrich Engelke, Daniel Keim, and Falk Schreiber. 2022. Immersive Analytics with Abstract 3D Visualizations: A Survey. Computer Graphics Forum 41 (2 2022), 201–229. Issue 1. https://doi.org/10.1111/cgf.14430
Lucy Lan, Randi Q. Mao, Reva Y. Qiu, Jeffrey Kay, and Darren de Sa. 2023. Immersive Virtual Reality for Patient-Specific Preoperative Planning: A Systematic Review. Surgical Innovation 30 (2 2023), 109–122. Issue 1. https://doi.org/10.1177/15533506221143235
Mark A Livingston, Lawrence J Rosenblum, Dennis G Brown, Gregory S Schmidt, Simon J Julier, Yohan Baillot, J Edward Swan, Zhuming Ai, and Paul Maassel. 2011. Military applications of augmented reality. Handbook of augmented reality (2011), 671–706.
Abel J. Lungu, Wout Swinkels, Luc Claesen, Puxun Tu, Jan Egger, and Xiaojun Chen. 2021. A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery. Expert Review of Medical Devices 18 (2021), 47–62. Issue 1. https://doi.org/10.1080/17434440.2021.1860750
Xun Luo, Tiffany Kline, Heidi C Fischer, Kathy A Stubblefield, Robert V Kenyon, and Derek G Kamper. 2006. Integration of augmented reality and assistive devices for post-stroke hand opening rehabilitation. In 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference. IEEE, 6855–6858.
Prasit Mahawongkajit, Ajjana Techagumpuch, and Kharikarn Auksornchat. 2022. Effects of basic endoscopic handling and care training on gastrointestinal endoscopy logistics. Endoscopy International Open 10 (1 2022), E56–E61. Issue 01. https://doi.org/10.1055/a-1630-6403
Randi Q Mao, Lucy Lan, Jeffrey Kay, Ryan Lohre, Olufemi R Ayeni, Danny P Goel, et al. 2021. Immersive virtual reality for surgical training: a systematic review. Journal of Surgical Research 268 (2021), 40–58.
Sophie Maria, Solène Lambert, and Ignacio Avellino. 2022. From Déjà vu to Déjà vécu: Reliving Surgery in Post-Operative Debriefing. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 462–465.
Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein, Nathalie Henry Riche, Takayuki Itoh, Wolfgang Stuerzlinger, and Bruce H. Thomas (Eds.). 2018. Immersive Analytics. Lecture Notes in Computer Science, Vol. 11190. Springer. https://doi.org/10.1007/978-3-030-01388-2
Daniel McDuff, Christophe Hurter, and Mar Gonzalez-Franco. 2017. Pulse and vital sign measurement in mixed reality using a HoloLens. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST Part F131944. https://doi.org/10.1145/3139131.3139134
Lucas Zanusso Morais, Victor Kunde Bergmann, Eduarda Abreu Carvalho, Raquel Zimmer, Marcelo Gomes Martins, Luciana Porcher Nedel, Anderson Maciel, and Rafael P. Torchelsen. 2023. An enhanced interactive endoscope model based on position-based dynamics and Cosserat rods for colonoscopy simulation. Computers & Graphics (2023). https://api.semanticscholar.org/CorpusID:261405897
Georgios Ntakakis, Christina Plomariti, Christos Frantzidis, Panagiotis E. Antoniou, Panagiotis D. Bamidis, and Georgios Tsoulfas. 2023. Exploring the use of virtual reality in surgical education. World Journal of Transplantation 13 (2 2023), 36–43. Issue 2. https://doi.org/10.5500/wjt.v13.i2.36
Armando Mónica Oliveira, Luís Manuel Cunha Batalha, Ananda Maria Fernandes, Joana Castro Gonçalves, and Ricardo Gaspar Viegas. 2014. A functional analysis of the Wong-Baker Faces Pain Rating Scale: linearity, discrim-inability and amplitude. Revista de Enfermagem Referencia 4 (12 2014), 121–130. Issue 3. https://doi.org/10.12707/RIV14018
Fernando Pérez-Escamirosa, David Medina-Alvarez, Eduardo Alfredo Ruíz-Vereo, Ricardo Manuel Ordorica-Flores, Arturo Minor-Martínez, and Jesús Tapia-Jurado. 2020. Immersive Virtual Operating Room Simulation for Surgical Resident Education During COVID-19. Surgical Innovation 27 (10 2020), 549–550. Issue 5. https://doi.org/10.1177/1553350620952183
Damaruka Priya Rajasagi. 2023. Immersive Health Data Visualization in Virtual Reality. Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023, 971–972. https://doi.org/10.1109/VRW58643.2023.00328
Francesca Bonetti, Gary Warnaby, and Lee Quinn. 2018. Augmented reality and virtual reality in physical and online retailing: A review, synthesis and research agenda. Augmented reality and virtual reality: Empowering human, place and business (2018), 119–132.
Aaron Bray, Jeffrey B. Webb, Andinet Enquobahrie, Jared Vicory, Jerry Heneghan, Robert Hubal, Stephanie TerMaath, Philip Asare, and Rachel B. Clipp. 2019. Pulse Physiology Engine: an Open-Source Software Platform for Computational Modeling of Human Medical Simulation. SN Comprehensive Clinical Medicine 1 (5 2019), 362–377. Issue 5. https://doi.org/10.1007/s42399-019-00053-w
João Cartucho, David Shapira, Hutan Ashrafian, and Stamatia Giannarou. 2020. Multimodal mixed reality visualisation for intraoperative surgical guidance. International Journal of Computer Assisted Radiology and Surgery 15 (5 2020), 819–826. Issue 5. https://doi.org/10.1007/s11548-020-02165-4
Oculus Developers. 2023. Oculus Developers: Interaction SDK Overview. [link]. Accessed: May 24, 2023.
Ciro Donalek, S. G. Djorgovski, Alex Cioc, Anwell Wang, Jerry Zhang, Elizabeth Lawler, Stacy Yeh, Ashish Mahabal, Matthew Graham, Andrew Drake, Scott Davidoff, Jeffrey S. Norris, and Giuseppe Longo. 2014. Immersive and collaborative data visualization using virtual reality platforms. 2014 IEEE International Conference on Big Data (Big Data), 609–614. https://doi.org/10.1109/BigData.2014.7004282
Bu’Hussain Hayee, David S. Rowbotham, Vishal Saxena, and Alistair Mcnair. 2006. Pain Assessment During Colonoscopy: Who Guides Sedation? Gastrointestinal Endoscopy 63 (4 2006), AB197. Issue 5. https://doi.org/10.1016/j.gie.2006.03.464
Yark Hazewinkel and Evelien Dekker. 2011. Colonoscopy: basic principles and novel techniques. Nature Reviews Gastroenterology & Hepatology 8 (10 2011), 554–564. Issue 10. https://doi.org/10.1038/nrgastro.2011.141
Wee Sim Khor, Benjamin Baker, Kavit Amin, Adrian Chan, Ketan Patel, and Jason Wong. 2016. Augmented and virtual reality in surgery—the digital surgical environment: applications, limitations and legal pitfalls. Annals of translational medicine 4, 23 (2016).
Matthias Kraus, Johannes Fuchs, Björn Sommer, Karsten Klein, Ulrich Engelke, Daniel Keim, and Falk Schreiber. 2022. Immersive Analytics with Abstract 3D Visualizations: A Survey. Computer Graphics Forum 41 (2 2022), 201–229. Issue 1. https://doi.org/10.1111/cgf.14430
Lucy Lan, Randi Q. Mao, Reva Y. Qiu, Jeffrey Kay, and Darren de Sa. 2023. Immersive Virtual Reality for Patient-Specific Preoperative Planning: A Systematic Review. Surgical Innovation 30 (2 2023), 109–122. Issue 1. https://doi.org/10.1177/15533506221143235
Mark A Livingston, Lawrence J Rosenblum, Dennis G Brown, Gregory S Schmidt, Simon J Julier, Yohan Baillot, J Edward Swan, Zhuming Ai, and Paul Maassel. 2011. Military applications of augmented reality. Handbook of augmented reality (2011), 671–706.
Abel J. Lungu, Wout Swinkels, Luc Claesen, Puxun Tu, Jan Egger, and Xiaojun Chen. 2021. A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery. Expert Review of Medical Devices 18 (2021), 47–62. Issue 1. https://doi.org/10.1080/17434440.2021.1860750
Xun Luo, Tiffany Kline, Heidi C Fischer, Kathy A Stubblefield, Robert V Kenyon, and Derek G Kamper. 2006. Integration of augmented reality and assistive devices for post-stroke hand opening rehabilitation. In 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference. IEEE, 6855–6858.
Prasit Mahawongkajit, Ajjana Techagumpuch, and Kharikarn Auksornchat. 2022. Effects of basic endoscopic handling and care training on gastrointestinal endoscopy logistics. Endoscopy International Open 10 (1 2022), E56–E61. Issue 01. https://doi.org/10.1055/a-1630-6403
Randi Q Mao, Lucy Lan, Jeffrey Kay, Ryan Lohre, Olufemi R Ayeni, Danny P Goel, et al. 2021. Immersive virtual reality for surgical training: a systematic review. Journal of Surgical Research 268 (2021), 40–58.
Sophie Maria, Solène Lambert, and Ignacio Avellino. 2022. From Déjà vu to Déjà vécu: Reliving Surgery in Post-Operative Debriefing. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 462–465.
Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein, Nathalie Henry Riche, Takayuki Itoh, Wolfgang Stuerzlinger, and Bruce H. Thomas (Eds.). 2018. Immersive Analytics. Lecture Notes in Computer Science, Vol. 11190. Springer. https://doi.org/10.1007/978-3-030-01388-2
Daniel McDuff, Christophe Hurter, and Mar Gonzalez-Franco. 2017. Pulse and vital sign measurement in mixed reality using a HoloLens. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST Part F131944. https://doi.org/10.1145/3139131.3139134
Lucas Zanusso Morais, Victor Kunde Bergmann, Eduarda Abreu Carvalho, Raquel Zimmer, Marcelo Gomes Martins, Luciana Porcher Nedel, Anderson Maciel, and Rafael P. Torchelsen. 2023. An enhanced interactive endoscope model based on position-based dynamics and Cosserat rods for colonoscopy simulation. Computers & Graphics (2023). https://api.semanticscholar.org/CorpusID:261405897
Georgios Ntakakis, Christina Plomariti, Christos Frantzidis, Panagiotis E. Antoniou, Panagiotis D. Bamidis, and Georgios Tsoulfas. 2023. Exploring the use of virtual reality in surgical education. World Journal of Transplantation 13 (2 2023), 36–43. Issue 2. https://doi.org/10.5500/wjt.v13.i2.36
Armando Mónica Oliveira, Luís Manuel Cunha Batalha, Ananda Maria Fernandes, Joana Castro Gonçalves, and Ricardo Gaspar Viegas. 2014. A functional analysis of the Wong-Baker Faces Pain Rating Scale: linearity, discrim-inability and amplitude. Revista de Enfermagem Referencia 4 (12 2014), 121–130. Issue 3. https://doi.org/10.12707/RIV14018
Fernando Pérez-Escamirosa, David Medina-Alvarez, Eduardo Alfredo Ruíz-Vereo, Ricardo Manuel Ordorica-Flores, Arturo Minor-Martínez, and Jesús Tapia-Jurado. 2020. Immersive Virtual Operating Room Simulation for Surgical Resident Education During COVID-19. Surgical Innovation 27 (10 2020), 549–550. Issue 5. https://doi.org/10.1177/1553350620952183
Damaruka Priya Rajasagi. 2023. Immersive Health Data Visualization in Virtual Reality. Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023, 971–972. https://doi.org/10.1109/VRW58643.2023.00328
Publicado
06/11/2023
Como Citar
MONTICELLI, Jonas; WAGNER, Jorge; MACIEL, Anderson; TORCHELSEN, Rafael P.; FREITAS, Carla; NEDEL, Luciana.
Immersive Visualization Interface for Endoscopy Analytics and Debriefing. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 25. , 2023, Rio Grande/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 166–173.
