Playing Games via Personalized Gestural Interaction
Resumo
Motivation is a key element in the learning process, and using games for this goal is an undeniably attractive option. However, individuals with motor disabilities are still finding themselves excluded from many game-based school approaches. This paper presents a solution developed for people with motor and speech impairments that includes a game-based approach to contribute in this context. Based on a methodology in which gestures and their meanings are created and configured by users and their caregivers, we developed a Computer Vision system named PGCA that employs machine learning techniques to create personalized gestural interaction as an Assistive Technology resource for communication purposes. This personalized gestural interaction enabled disabled people to interact with the PGCA system and play a game using their remaining functional movements. Results from experiments carried out with the target audience indicate that the structure used in the developed system can be expanded to create new games, with different purposes, including to educational context.
Palavras-chave:
Assistive Technology, Gestural Interaction, Machine Learning, Games, Education
Referências
P. Piccolo, and E. Guerra, ”Using a card game for teaching design patterns”, in Proceedings of the 11th Latin-American Conference on Pattern Languages of Programming, 2016, pp. 1-10.
H. P. Pontes, ”Game development in the process of learning algorithms and computer programming”, (original) Desenvolvimento de jogos no processo de aprendizado em algoritmos e programação de computadores, in Proceedings of the XII Simpósio Brasileiro de Games e Entretenimento Digital (SBGames), São Paulo, 2013.
W. Chen, ”Gesture-based applications for elderly people”, in International Conference on Human-Computer Interaction, 2013, pp. 186-195. Springer, Berlin, Heidelberg.
S. Cai, G. Zhu, Y. T. Wu, E. Liu, and X. Hu, ”A case study of gesturebased games in enhancing the fine motor skills and recognition of children with autism”, Interactive Learning Environments, 26(8), 2018, pp. 1039-1052.
R. E. S. Ascari, R. Pereira, and L. Silva, ”Computer Vision-based methodology to improve interaction for people with motor and speech impairment”, ACM Transactions on Accessible Computing (TACCESS), 13(4), 2020, pp. 1-33.
R. E. S. Ascari, L. Silva, and R. Pereira, ”Personalized gestural interaction applied in a gesture interactive game-based approach for people with disabilities”, in Proceedings of the 25th International Conference on Intelligent User Interfaces, 2020, pp. 100-110.
Y. J. Chang, W. Y. Han, and Y. C. Tsai, ”A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy”, Research in developmental disabilities, 34(11), 2013, pp. 3654-3659.
A. A. Foletto, M. C. d’Ornellas, and A. C. Prado, ”Serious games for Parkinson’s disease fine motor skills rehabilitation using natural interfaces”, in MEDINFO 2017: Precision Healthcare Through Informatics: Proceedings of the 16th World Congress on Medical and Health Informatics, Vol. 245, 2018, pp. 74. IOS Press.
S. Aced Lopez, F. Corno, and L. De Russis, ”Gnomon: Enabling dynamic one-switch games for children with severe motor disabilities”, in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, 2015, pp. 995-1000.
H. Jiang, B. S. Duerstock, and J. P. Wachs, ”Variability analysis on gestures for people with quadriplegia”, IEEE transactions on cybernetics, 48(1), 2016, pp. 346-356.
H. Jiang, B. S. Duerstock, and J. P. Wachs, ”An analytic approach to decipher usable gestures for quadriplegic users”, in 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2014, pp. 3912-3917, IEEE.
F. R. Camara Machado, P. P. Antunes, J. D. M. Souza, A. C. D. Santos, D. C Levandowski, and A. A. D. Oliveira, ”Motor improvement using motion sensing game devices for cerebral palsy rehabilitation”. Journal of motor behavior, 49(3), 2017, pp. 273-280.
G. Altanis, M. Boloudakis, S. Retalis, and N. Nikou, ”Children with motor impairments play a kinect learning game: first findings from a pilot case in an authentic classroom environment”, J Interact Design Architect, 19, 2013, pp. 91-104.
D. M. Tsai, W. Y. Chiu, and M. H. Lee, ”Optical flow-motion history image (OF-MHI) for action recognition”, Signal, Image and Video Processing, 9(8), 2015, pp. 1897-1906.
B. D. Lucas, and T. Kanade, ”An iterative image registration technique with an application to stereo vision”, in Proceedings of the 7th International Joint Conference on Artificial Intelligence, 1981, Vancouver, BC, Canada.
X. Fan, and T. Tjahjadi, ”A dynamic framework based on local Zernike moment and motion history image for facial expression recognition”, Pattern Recognition, 64, 2017, pp. 399–406.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, and et al., ”Tensorflow: a system for large-scale machine learning”, in OSDI, vol. 16, 2016, pp. 265–283.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ”Rethinking the inception architecture for computer vision”, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, and et al., ”Imagenet large scale visual recognition challenge”, International Journal of Computer Vision 115, 3, 2015, pp. 211–252.
J. Rodrigo, and D. Corral, ”ARASAAC: Aragonese portal of augmentative and alternative communication”, Software, tools and materials for communication and inclusion, Informatica na Educação: teoria & prática, 16, 2, 2013.
Wikimedia Commons, ”CommonsWikimedia”, Available: https://commons.wikimedia.org/wiki/File:Turtle_clip_art.svg. 2021.
S. B. Day, ”OpenGameArt.org”, The Chayed-KIIRA, Available: https://opengameart.org. 2015.
Google, ”Google Image”, Available: http://images.google.com. 2019.
R. E. O. S. Ascari, L. Silva, and R. Pereira, ”Methodology based on Computer Vision and Machine Learning to guide the design of Augmentative and Alternative Communication systems using personalized gestural interaction”, in Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems, Virtual Event, Brazil, 2021.
R. Ossmann, D. Archambault, and K. Miesenberger, ”Accessibility issues in game-like interfaces”, in International Conference on Computers for Handicapped Persons, Springer, 2008, pp. 601–604.
H. P. Pontes, ”Game development in the process of learning algorithms and computer programming”, (original) Desenvolvimento de jogos no processo de aprendizado em algoritmos e programação de computadores, in Proceedings of the XII Simpósio Brasileiro de Games e Entretenimento Digital (SBGames), São Paulo, 2013.
W. Chen, ”Gesture-based applications for elderly people”, in International Conference on Human-Computer Interaction, 2013, pp. 186-195. Springer, Berlin, Heidelberg.
S. Cai, G. Zhu, Y. T. Wu, E. Liu, and X. Hu, ”A case study of gesturebased games in enhancing the fine motor skills and recognition of children with autism”, Interactive Learning Environments, 26(8), 2018, pp. 1039-1052.
R. E. S. Ascari, R. Pereira, and L. Silva, ”Computer Vision-based methodology to improve interaction for people with motor and speech impairment”, ACM Transactions on Accessible Computing (TACCESS), 13(4), 2020, pp. 1-33.
R. E. S. Ascari, L. Silva, and R. Pereira, ”Personalized gestural interaction applied in a gesture interactive game-based approach for people with disabilities”, in Proceedings of the 25th International Conference on Intelligent User Interfaces, 2020, pp. 100-110.
Y. J. Chang, W. Y. Han, and Y. C. Tsai, ”A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy”, Research in developmental disabilities, 34(11), 2013, pp. 3654-3659.
A. A. Foletto, M. C. d’Ornellas, and A. C. Prado, ”Serious games for Parkinson’s disease fine motor skills rehabilitation using natural interfaces”, in MEDINFO 2017: Precision Healthcare Through Informatics: Proceedings of the 16th World Congress on Medical and Health Informatics, Vol. 245, 2018, pp. 74. IOS Press.
S. Aced Lopez, F. Corno, and L. De Russis, ”Gnomon: Enabling dynamic one-switch games for children with severe motor disabilities”, in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, 2015, pp. 995-1000.
H. Jiang, B. S. Duerstock, and J. P. Wachs, ”Variability analysis on gestures for people with quadriplegia”, IEEE transactions on cybernetics, 48(1), 2016, pp. 346-356.
H. Jiang, B. S. Duerstock, and J. P. Wachs, ”An analytic approach to decipher usable gestures for quadriplegic users”, in 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2014, pp. 3912-3917, IEEE.
F. R. Camara Machado, P. P. Antunes, J. D. M. Souza, A. C. D. Santos, D. C Levandowski, and A. A. D. Oliveira, ”Motor improvement using motion sensing game devices for cerebral palsy rehabilitation”. Journal of motor behavior, 49(3), 2017, pp. 273-280.
G. Altanis, M. Boloudakis, S. Retalis, and N. Nikou, ”Children with motor impairments play a kinect learning game: first findings from a pilot case in an authentic classroom environment”, J Interact Design Architect, 19, 2013, pp. 91-104.
D. M. Tsai, W. Y. Chiu, and M. H. Lee, ”Optical flow-motion history image (OF-MHI) for action recognition”, Signal, Image and Video Processing, 9(8), 2015, pp. 1897-1906.
B. D. Lucas, and T. Kanade, ”An iterative image registration technique with an application to stereo vision”, in Proceedings of the 7th International Joint Conference on Artificial Intelligence, 1981, Vancouver, BC, Canada.
X. Fan, and T. Tjahjadi, ”A dynamic framework based on local Zernike moment and motion history image for facial expression recognition”, Pattern Recognition, 64, 2017, pp. 399–406.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, and et al., ”Tensorflow: a system for large-scale machine learning”, in OSDI, vol. 16, 2016, pp. 265–283.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ”Rethinking the inception architecture for computer vision”, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, and et al., ”Imagenet large scale visual recognition challenge”, International Journal of Computer Vision 115, 3, 2015, pp. 211–252.
J. Rodrigo, and D. Corral, ”ARASAAC: Aragonese portal of augmentative and alternative communication”, Software, tools and materials for communication and inclusion, Informatica na Educação: teoria & prática, 16, 2, 2013.
Wikimedia Commons, ”CommonsWikimedia”, Available: https://commons.wikimedia.org/wiki/File:Turtle_clip_art.svg. 2021.
S. B. Day, ”OpenGameArt.org”, The Chayed-KIIRA, Available: https://opengameart.org. 2015.
Google, ”Google Image”, Available: http://images.google.com. 2019.
R. E. O. S. Ascari, L. Silva, and R. Pereira, ”Methodology based on Computer Vision and Machine Learning to guide the design of Augmentative and Alternative Communication systems using personalized gestural interaction”, in Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems, Virtual Event, Brazil, 2021.
R. Ossmann, D. Archambault, and K. Miesenberger, ”Accessibility issues in game-like interfaces”, in International Conference on Computers for Handicapped Persons, Springer, 2008, pp. 601–604.
Publicado
18/10/2021
Como Citar
ASCARI, Rúbia E. O. Schultz; SILVA, Luciano; PEREIRA, Roberto.
Playing Games via Personalized Gestural Interaction. In: TRILHA DE EDUCAÇÃO – ARTIGOS CURTOS - SIMPÓSIO BRASILEIRO DE JOGOS E ENTRETENIMENTO DIGITAL (SBGAMES), 20. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 717-720.
DOI: https://doi.org/10.5753/sbgames_estendido.2021.19716.
