David Scherfgen
ABSTRACT
Virtual medical emergency training provides complex while safe interactions with virtual patients. Classical non-virtual medical training includes the use of medical manikins. Haptically integrating such a manikin into virtual training has the potential to improve the interaction with a virtual patient (e.g. regarding social presence) and the resulting training experience. This work analyzes user requirements and several technological approaches. The presented system estimates the pose of a medical manikin in order to haptically augment a 3D human model in a virtual reality (VR) training environment, allowing users to physically touch a virtual patient. The system uses a Convolutional Neural Networks-based (CNN) body keypoint detector to locate relevant keypoints of the manikin in the images of the stereo camera built into a head-mounted display (HMD). The system retrieves manikin position, orientation and joint angles using non-linear optimization. A manual precision analysis reports a mean error distance of 48 mm. Unless further optimized, we recommend applying this method for haptically augmented social interaction with virtual patients but not for practicing precise medical treatments.
- Sameer Agarwal, Keir Mierle, and Others. [n.d.]. Ceres Solver. http://ceres-solver.org.Google Scholar
- Omamah Almousa, Joana Prates, Noor Yeslam, Dougal Mac Gregor, Junsong Zhang, Viet Phan, Marc Nielsen, Richard Smith, and Karim Qayumi. 2019. Virtual Reality Simulation Technology for Cardiopulmonary Resuscitation Training: An Innovative Hybrid System With Haptic Feedback. Simulation and Gaming 50, 1 (2019), 6–22. https://doi.org/10.1177/1046878118820905Google ScholarDigital Library
- G. Bradski. 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools(2000).Google Scholar
- Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. 2019. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. IEEE Trans. on Pattern Analysis and Machine Intelligence (2019). https://doi.org/10.1109/TPAMI.2019.2929257Google ScholarDigital Library
- Marco Chirico, Simone Bazurro, Andrea Calandrino, Manuela Chessa, Federica Imperiale, Serena Ricci, Alessio Riitano, Anna Siri, and Giancarlo Torre. 2020. Sharing is caring: A multidisciplinary first aid program. 9th International Workshop on Innovative Simulation for Health Care, IWISH 2020September(2020), 47–51. https://doi.org/10.46354/i3m.2020.iwish.009Google ScholarCross Ref
- Salam Daher. 2017. Optical See-Through vs. Spatial Augmented Reality Simulators for Medical Applications. In 2017 IEEE Virtual Reality (VR). 417–418. https://doi.org/10.1109/VR.2017.7892354Google ScholarCross Ref
- David W. Eggert, Adele Lorusso, and Robert B. Fisher. 1997. Estimating 3-D rigid body transformations: a comparison of four major algorithms. Mach. Vis. Appl. 9, 5/6 (1997), 272–290. https://doi.org/10.1007/s001380050048Google ScholarDigital Library
- Mark Fiala. 2005. ARTag, a Fiducial Marker System Using Digital Techniques. In CVPR (2). IEEE Computer Society, 590–596. https://doi.org/10.1109/CVPR.2005.74Google ScholarDigital Library
- Elisa Girau, Fabrizio Mura, Simone Bazurro, Maura Casadio, Marco Chirico, Fabio Solari, and Manuela Chessa. 2019. A Mixed Reality system for the simulation of emergency and first-aid scenarios. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS(2019), 5690–5695. https://doi.org/10.1109/EMBC.2019.8856777Google ScholarCross Ref
- Mahdi Rahmani Hanzaki and Pierre Boulanger. 2020. Proxy Haptics for Surgical Training. In Proceedings - 2020 22nd Symposium on Virtual and Augmented Reality, SVR 2020. 134–143. https://doi.org/10.1109/SVR51698.2020.00033Google ScholarCross Ref
- Jason Hochreiter, Salam Daher, Arjun Nagendran, Laura Gonzalez, and Greg Welch. 2015. Touch Sensing on Non-Parametric Rear-Projection Surfaces: A Physical-Virtual Head for Hands-On Healthcare Training. In 2015 IEEE Virtual Reality (VR). 69–74. https://doi.org/10.1109/VR.2015.7223326Google ScholarCross Ref
- Matthias Hoppe, Beat Rossmy, Daniel Peter Neumann, Stephan Streuber, Albrecht Schmidt, and Tonja Katrin MacHulla. 2020. A Human Touch: Social Touch Increases the Perceived Human-likeness of Agents in Virtual Reality. Conference on Human Factors in Computing Systems - Proceedings (2020), 1–11. https://doi.org/10.1145/3313831.3376719Google ScholarDigital Library
- Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. 2020. VIBE: Video Inference for Human Body Pose and Shape Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Aaron Kotranza and Benjamin Lok. 2008. Virtual human + tangible interface = Mixed reality human an initial exploration with a virtual breast exam patient. Proceedings - IEEE Virtual Reality(2008), 99–106. https://doi.org/10.1109/VR.2008.4480757Google ScholarCross Ref
- Dieter Lerner, Stefan Mohr, Jonas Schild, Martin Göring, and Thomas Luiz. 2020. An immersive multi-user virtual reality for emergency simulation training: Usability study. JMIR Serious Games 8, 3 (2020), 1–10. https://doi.org/10.2196/18822Google ScholarCross Ref
- Ahmed A A Osman, Timo Bolkart, and Michael J. Black. 2020. STAR: A Sparse Trained Articulated Human Body Regressor. In European Conference on Computer Vision (ECCV). 598–613. https://star.is.tue.mpg.deGoogle ScholarDigital Library
- Pierre Plantard, Edouard Auvinet, Anne-Sophie Le Pierres, and Franck Multon. 2015. Pose Estimation with a Kinect for Ergonomic Studies: Evaluation of the Accuracy Using a Virtual Mannequin. Sensors 15, 1 (2015), 1785–1803. https://doi.org/10.3390/s150101785Google ScholarCross Ref
- Jonas Schild, Christian Elsenbast, and Guillermo Carbonell. 2021. ViTAWiN – Developing Multiprofessional Medical Emergency Training with Mixed Reality. In IEEE SeGAH 2021.Google ScholarCross Ref
- Jonas Schild, Dieter Lerner, Sebastian Misztal, and Thomas Luiz. 2018. EPICSAVE - Enhancing vocational training for paramedics with multi-user virtual reality. 2018 IEEE 6th International Conference on Serious Games and Applications for Health, SeGAH 2018(2018), 1–8. https://doi.org/10.1109/SeGAH.2018.8401353Google ScholarCross Ref
Index Terms
- Estimating the Pose of a Medical Manikin for Haptic Augmentation of a Virtual Patient in Mixed Reality Training
Recommendations
The virtual haptic back for palpatory training
ICMI '04: Proceedings of the 6th international conference on Multimodal interfacesThis paper discusses the Ohio University Virtual Haptic Back (VHB) project, including objectives, implementation, and initial evaluations. Haptics is the science of human tactile sensation and a haptic interface provides force and touch feedback to the ...
Haptic around: multiple tactile sensations for immersive environment and interaction in virtual reality
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologyIn this paper, we present Haptic Around, a hybrid-haptic feedback system, which utilizes fan, hot air blower, mist creator and heat light to recreate multiple tactile sensations in virtual reality for enhancing the immersive environment and interaction. ...
Mixed reality in virtual world teleconferencing
VR '10: Proceedings of the 2010 IEEE Virtual Reality ConferenceIn this paper we present a Mixed Reality (MR) teleconferencing application based on Second Life (SL) and the OpenSim virtual world. Augmented Reality (AR) techniques are used for displaying virtual avatars of remote meeting participants in real physical ...
Comments