Estimating the Pose of a Medical Manikin for Haptic Augmentation of a Virtual Patient in Mixed Reality Training

  • David Scherfgen Hanover University of Applied Sciences and Arts
  • Jonas Schild Hanover University of Applied Sciences and Arts

Resumo


Virtual medical emergency training provides complex while safe interactions with virtual patients. Classical non-virtual medical training includes the use of medical manikins. Haptically integrating such a manikin into virtual training has the potential to improve the interaction with a virtual patient (e.g. regarding social presence) and the resulting training experience. This work analyzes user requirements and several technological approaches. The presented system estimates the pose of a medical manikin in order to haptically augment a 3D human model in a virtual reality (VR) training environment, allowing users to physically touch a virtual patient. The system uses a Convolutional Neural Networks-based (CNN) body keypoint detector to locate relevant keypoints of the manikin in the images of the stereo camera built into a head-mounted display (HMD). The system retrieves manikin position, orientation and joint angles using non-linear optimization. A manual precision analysis reports a mean error distance of 48 mm. Unless further optimized, we recommend applying this method for haptically augmented social interaction with virtual patients but not for practicing precise medical treatments.

Palavras-chave: body pose estimation, medical manikin, haptics, virtual medical training, mixed reality
Publicado
18/10/2021
SCHERFGEN, David; SCHILD, Jonas. Estimating the Pose of a Medical Manikin for Haptic Augmentation of a Virtual Patient in Mixed Reality Training. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 23. , 2021, Evento Online. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2021 . p. 32-40.