Interação por Rastreamento de Mão em ambiente de Realidade Virtual
Resumo
Este artigo descreve a criação de um sistema de baixo custo que permite a interação de um usuário com ambientes virtuais modelados em Unity e visualizados através do Google Card Board - a interação testada se baseia no uso das mãos para mover objetos em uma tarefa de pick and place. O sistema proposto divide o processamento em uma arquitetura cliente-servidor na qual o ambiente virtual é executado no cliente, ao mesmo tempo que este envia fotos do celular do Card Board para o servidor. Este processa e retorna a pose da mão presente na imagem e envia de volta para o cliente, o qual renderiza e processa interações.
Palavras-chave:
Redes Neurais, Aprendizado Profundo, Realidade Virtual, Sistema de Tempo Real, Análise de Imagens, Detecção de Posição
Referências
SHERMAN, W. R.; CRAIG, A. B. Understanding virtual reality: Interface, application, and design. Cambridge, MA: Morgan Kaufmann, 2018. 582 p.
GOLDSTONE,W. Unity game development essentials. Birmingham, UK: Packt Publishing Ltd, 2009.
MISHRA, P.; SHRAWANKAR, U. Comparison between famous game engines and eminent games. International Journal of Interactive Multimedia & Artificial Intelligence, v. 4, n. 1, 2016.
Sharrab, Y. O.; Sarhan, N. J. Detailed comparative analysis of vp8 and h.264. In: IEEE. 2012 IEEE International Symposium on Multimedia. Irvine, CA, 2012. p. 133–140.
Chen, Y.; Murherjee, D.; Han, J.; Grange, A.; Xu, Y.; Liu, Z.; Parker, S.; Chen, C.; Su, H.; Joshi, U.; Chiang, C.; Wang, Y.; Wilkins, P.; Bankoski, J.; Trudeau, L.; Egge, N.; Valin, J.; Davies, T.; Midtskogen, S.; Norkin, A.; de Rivaz, P. An overview of core coding tools in the av1 video codec. In: IEEE. 2018 Picture Coding Symposium (PCS). San Francisco, CA: arXiv, 2018. p. 41–45.
REDMON, J.; FARHADI, A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
CAO, Z.; SIMON, T.; WEI, S.-E.; SHEIKH, Y. Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition. San Franscisco, CA: IEEE, 2017. p. 7291–7299.
MUELLER, F.; BERNARD, F.; SOTNYCHENKO, O.; MEHTA, D.; SRIDHAR, S.; CASAS, D.; THEOBALT, C. Ganerated hands for realtime 3d hand tracking from monocular rgb. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, CA: IEEE, 2018. p. 49–59.
CASIEZ, G.; ROUSSEL, N.; VOGEL, D. 1C filter: a simple speed-based low-pass filter for noisy input in interactive systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY: ACM, 2012. p. 2527–2530.
KALMAN, R. E. A new approach to linear filtering and prediction problems. 1960.
PANTELERIS, P.; OIKONOMIDIS, I.; ARGYROS, A. Using a single rgb frame for real time 3d hand pose estimation in the wild. In: IEEE. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, NV/CA, 2018. p. 436–445.
KUCUK, S.; BINGUL, Z. Robot kinematics: Forward and inverse kinematics. London, UK: INTECH Open Access Publisher, 2006.
CALIBRATION, C. 3d reconstruction, opencv 2.4. 13.6 documentation. Opencv dev team, 2018.
MORÉ, J. J. The levenberg-marquardt algorithm: implementation and theory. In: Numerical analysis. Berlin, Heidelberg, DE: Springer, 1978. p. 105–116.
ZHANG, F.; BAZAREVSKY, V.; VAKUNOV, A.; TKACHENKA, A.; SUNG, G.; CHANG, C.-L.; GRUNDMANN, M. Mediapipe hands: Ondevice real-time hand tracking. arXiv preprint arXiv:2006.10214, 2020.
GOLDSTONE,W. Unity game development essentials. Birmingham, UK: Packt Publishing Ltd, 2009.
MISHRA, P.; SHRAWANKAR, U. Comparison between famous game engines and eminent games. International Journal of Interactive Multimedia & Artificial Intelligence, v. 4, n. 1, 2016.
Sharrab, Y. O.; Sarhan, N. J. Detailed comparative analysis of vp8 and h.264. In: IEEE. 2012 IEEE International Symposium on Multimedia. Irvine, CA, 2012. p. 133–140.
Chen, Y.; Murherjee, D.; Han, J.; Grange, A.; Xu, Y.; Liu, Z.; Parker, S.; Chen, C.; Su, H.; Joshi, U.; Chiang, C.; Wang, Y.; Wilkins, P.; Bankoski, J.; Trudeau, L.; Egge, N.; Valin, J.; Davies, T.; Midtskogen, S.; Norkin, A.; de Rivaz, P. An overview of core coding tools in the av1 video codec. In: IEEE. 2018 Picture Coding Symposium (PCS). San Francisco, CA: arXiv, 2018. p. 41–45.
REDMON, J.; FARHADI, A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
CAO, Z.; SIMON, T.; WEI, S.-E.; SHEIKH, Y. Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition. San Franscisco, CA: IEEE, 2017. p. 7291–7299.
MUELLER, F.; BERNARD, F.; SOTNYCHENKO, O.; MEHTA, D.; SRIDHAR, S.; CASAS, D.; THEOBALT, C. Ganerated hands for realtime 3d hand tracking from monocular rgb. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, CA: IEEE, 2018. p. 49–59.
CASIEZ, G.; ROUSSEL, N.; VOGEL, D. 1C filter: a simple speed-based low-pass filter for noisy input in interactive systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY: ACM, 2012. p. 2527–2530.
KALMAN, R. E. A new approach to linear filtering and prediction problems. 1960.
PANTELERIS, P.; OIKONOMIDIS, I.; ARGYROS, A. Using a single rgb frame for real time 3d hand pose estimation in the wild. In: IEEE. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, NV/CA, 2018. p. 436–445.
KUCUK, S.; BINGUL, Z. Robot kinematics: Forward and inverse kinematics. London, UK: INTECH Open Access Publisher, 2006.
CALIBRATION, C. 3d reconstruction, opencv 2.4. 13.6 documentation. Opencv dev team, 2018.
MORÉ, J. J. The levenberg-marquardt algorithm: implementation and theory. In: Numerical analysis. Berlin, Heidelberg, DE: Springer, 1978. p. 105–116.
ZHANG, F.; BAZAREVSKY, V.; VAKUNOV, A.; TKACHENKA, A.; SUNG, G.; CHANG, C.-L.; GRUNDMANN, M. Mediapipe hands: Ondevice real-time hand tracking. arXiv preprint arXiv:2006.10214, 2020.
Publicado
07/11/2020
Como Citar
CASTRO, Mateus; XAVIER, João; ROSA, Paulo; DE OLIVEIRA, Jauvane.
Interação por Rastreamento de Mão em ambiente de Realidade Virtual. In: WORKSHOP DE INICIAÇÃO CIENTÍFICA - SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 22. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 44-48.
DOI: https://doi.org/10.5753/svr_estendido.2020.12954.
