Interaction by Hand-Tracking in Virtual Reality
Abstract
This article describes the ellaboration of a low cost system that allows user interaction with VR environments modeled in Unity and viewed through Google Card Board. The proposed system divides the processing in a client-server architecture in which the VR experience is executed on client, while it sends pictures captured from the mobile device inside the Card Board to the server. This server processes the hand pose information and sends it to the client.
Keywords:
Redes Neurais, Aprendizado Profundo, Realidade Virtual, Sistema de Tempo Real, Análise de Imagens, Detecção de Posição
References
SHERMAN, W. R.; CRAIG, A. B. Understanding virtual reality: Interface, application, and design. Cambridge, MA: Morgan Kaufmann, 2018. 582 p.
GOLDSTONE,W. Unity game development essentials. Birmingham, UK: Packt Publishing Ltd, 2009.
MISHRA, P.; SHRAWANKAR, U. Comparison between famous game engines and eminent games. International Journal of Interactive Multimedia & Artificial Intelligence, v. 4, n. 1, 2016.
Sharrab, Y. O.; Sarhan, N. J. Detailed comparative analysis of vp8 and h.264. In: IEEE. 2012 IEEE International Symposium on Multimedia. Irvine, CA, 2012. p. 133–140.
Chen, Y.; Murherjee, D.; Han, J.; Grange, A.; Xu, Y.; Liu, Z.; Parker, S.; Chen, C.; Su, H.; Joshi, U.; Chiang, C.; Wang, Y.; Wilkins, P.; Bankoski, J.; Trudeau, L.; Egge, N.; Valin, J.; Davies, T.; Midtskogen, S.; Norkin, A.; de Rivaz, P. An overview of core coding tools in the av1 video codec. In: IEEE. 2018 Picture Coding Symposium (PCS). San Francisco, CA: arXiv, 2018. p. 41–45.
REDMON, J.; FARHADI, A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
CAO, Z.; SIMON, T.; WEI, S.-E.; SHEIKH, Y. Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition. San Franscisco, CA: IEEE, 2017. p. 7291–7299.
MUELLER, F.; BERNARD, F.; SOTNYCHENKO, O.; MEHTA, D.; SRIDHAR, S.; CASAS, D.; THEOBALT, C. Ganerated hands for realtime 3d hand tracking from monocular rgb. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, CA: IEEE, 2018. p. 49–59.
CASIEZ, G.; ROUSSEL, N.; VOGEL, D. 1C filter: a simple speed-based low-pass filter for noisy input in interactive systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY: ACM, 2012. p. 2527–2530.
KALMAN, R. E. A new approach to linear filtering and prediction problems. 1960.
PANTELERIS, P.; OIKONOMIDIS, I.; ARGYROS, A. Using a single rgb frame for real time 3d hand pose estimation in the wild. In: IEEE. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, NV/CA, 2018. p. 436–445.
KUCUK, S.; BINGUL, Z. Robot kinematics: Forward and inverse kinematics. London, UK: INTECH Open Access Publisher, 2006.
CALIBRATION, C. 3d reconstruction, opencv 2.4. 13.6 documentation. Opencv dev team, 2018.
MORÉ, J. J. The levenberg-marquardt algorithm: implementation and theory. In: Numerical analysis. Berlin, Heidelberg, DE: Springer, 1978. p. 105–116.
ZHANG, F.; BAZAREVSKY, V.; VAKUNOV, A.; TKACHENKA, A.; SUNG, G.; CHANG, C.-L.; GRUNDMANN, M. Mediapipe hands: Ondevice real-time hand tracking. arXiv preprint arXiv:2006.10214, 2020.
GOLDSTONE,W. Unity game development essentials. Birmingham, UK: Packt Publishing Ltd, 2009.
MISHRA, P.; SHRAWANKAR, U. Comparison between famous game engines and eminent games. International Journal of Interactive Multimedia & Artificial Intelligence, v. 4, n. 1, 2016.
Sharrab, Y. O.; Sarhan, N. J. Detailed comparative analysis of vp8 and h.264. In: IEEE. 2012 IEEE International Symposium on Multimedia. Irvine, CA, 2012. p. 133–140.
Chen, Y.; Murherjee, D.; Han, J.; Grange, A.; Xu, Y.; Liu, Z.; Parker, S.; Chen, C.; Su, H.; Joshi, U.; Chiang, C.; Wang, Y.; Wilkins, P.; Bankoski, J.; Trudeau, L.; Egge, N.; Valin, J.; Davies, T.; Midtskogen, S.; Norkin, A.; de Rivaz, P. An overview of core coding tools in the av1 video codec. In: IEEE. 2018 Picture Coding Symposium (PCS). San Francisco, CA: arXiv, 2018. p. 41–45.
REDMON, J.; FARHADI, A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
CAO, Z.; SIMON, T.; WEI, S.-E.; SHEIKH, Y. Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition. San Franscisco, CA: IEEE, 2017. p. 7291–7299.
MUELLER, F.; BERNARD, F.; SOTNYCHENKO, O.; MEHTA, D.; SRIDHAR, S.; CASAS, D.; THEOBALT, C. Ganerated hands for realtime 3d hand tracking from monocular rgb. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, CA: IEEE, 2018. p. 49–59.
CASIEZ, G.; ROUSSEL, N.; VOGEL, D. 1C filter: a simple speed-based low-pass filter for noisy input in interactive systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY: ACM, 2012. p. 2527–2530.
KALMAN, R. E. A new approach to linear filtering and prediction problems. 1960.
PANTELERIS, P.; OIKONOMIDIS, I.; ARGYROS, A. Using a single rgb frame for real time 3d hand pose estimation in the wild. In: IEEE. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, NV/CA, 2018. p. 436–445.
KUCUK, S.; BINGUL, Z. Robot kinematics: Forward and inverse kinematics. London, UK: INTECH Open Access Publisher, 2006.
CALIBRATION, C. 3d reconstruction, opencv 2.4. 13.6 documentation. Opencv dev team, 2018.
MORÉ, J. J. The levenberg-marquardt algorithm: implementation and theory. In: Numerical analysis. Berlin, Heidelberg, DE: Springer, 1978. p. 105–116.
ZHANG, F.; BAZAREVSKY, V.; VAKUNOV, A.; TKACHENKA, A.; SUNG, G.; CHANG, C.-L.; GRUNDMANN, M. Mediapipe hands: Ondevice real-time hand tracking. arXiv preprint arXiv:2006.10214, 2020.
Published
2020-11-07
How to Cite
CASTRO, Mateus; XAVIER, João; ROSA, Paulo; DE OLIVEIRA, Jauvane.
Interaction by Hand-Tracking in Virtual Reality. In: WORKSHOP OF UNDERGRADUATE WORKS - SYMPOSIUM ON VIRTUAL AND AUGMENTED REALITY (SVR), 22. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 44-48.
DOI: https://doi.org/10.5753/svr_estendido.2020.12954.
