Gaze Estimation em Atividades de Ensino Digitais Utilizando Convolutional Neural Networks e Webcam
Resumo
A área da educação possui desafios devido à variedade de aptidões e restrições dos estudantes. Para inovar o ensino, é necessário desenvolver alternativas que incrementem as propostas atuais. Uma das técnicas existentes na literatura é o gaze estimation, que permite visualizar o comportamento ocular ao realizar atividades no computador. Usualmente esta técnica é realizada com o uso de produtos comerciais ou métodos pouco acessíveis. Este trabalho propõe uma metodologia de baixo custo para realizar gaze estimation e identificar regiões de interesse do olhar do estudante em atividades educacionais, com uso de redes neurais convolucionais e webcam. Observa-se potencial do método no desenvolvimento de ferramentas aplicáveis a educação.
Palavras-chave:
Gaze Estimation, Educação, Redes Neurais Convolucionais
Referências
Ahmad, F. K. (2015). Use of assistive technology in inclusive education: making room for diverse learning needs. Transcience, 6(2):62–77.
Bishop, C. M. (1994). Neural networks and their applications. Review of scientific instruments, 65(6):1803–1832.
Brasil (2018). Base Nacional Comum Curricular. Brasília.
Brazil (1997). Lei de diretrizes e bases da educação nacional. Conselho de Reitores das Universidades Brasileiras
Chen, X. and Chen, Z. (2017). Exploring visual attention using random walks based eye tracking protocols. Journal of Visual Communication and Image Representation, 45:147–155.
Cheng, Y., Lu, F., and Zhang, X. (2018). Appearance-based gaze estimation via evaluation-guided asymmetric regression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 100–115.
Chong, E., Chanda, K., Ye, Z., Southerland, A., Ruiz, N., Jones, R. M., Rozga, A., and Rehg, J. M. (2017). Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1–20.
Costa, A. B. d., Picharillo, A. D. M., and Elias, N. C. (2017). Avaliação de habilidades matemáticas em crianças com síndrome de down e com desenvolvimento típico. Ciência & Educação (Bauru), 23:255–272.
de Araujo Cavalcante, T., Frazão, J., Paiva, A., Maia, I. M., Benitez, P., and Soares, A. (2019). Eye tracking como estratégia de ensino e avalização na educação inclusiva: Aplicação com alunos com autismo. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 1221.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.
Duchowski, A. T. and Duchowski, A. T. (2017). Eye tracking methodology: Theory and practice. Springer
Eckstein, M. K., Guerra-Carrillo, B., Miller Singley, A. T., and Bunge, S. A. (2017). Beyond eye gaze: What else can eye tracking reveal about cognition and cognitive development? Developmental Cognitive Neuroscience, 25:69–91. Sensitive periods across development.
Fischer, T., Chang, H. J., and Demiris, Y. (2018). Rt-gene: Real-time eye gaze estimation in natural environments. In Proceedings of the European Conference on Computer Vision (ECCV), pages 334–352.
George, A. and Routray, A. (2016). Real-time eye gaze direction classification using convolutional neural network. In 2016 International Conference on Signal Processing and Communications (SPCOM), pages 1–5. IEEE.
Halszka, J., Holmqvist, K., and Gruber, H. (2017). Eye tracking in educational science: Theoretical frameworks and research agendas. Journal of eye movement research, 10(1).
Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Jarodzka, H., and Van de ¨ Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. OUP Oxford.
Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., and Torralba, A. (2016). Eye tracking for everyone. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2176–2184.
Lukasova, K., Nucci, M. P., Neto, R. M. d. A., Vieira, G., Sato, J. R., and Amaro Jr, E. (2018). Predictive saccades in children and adults: A combined fmri and eye tracking study. PloS one, 13(5):e0196000.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. (2018). Foundations of machine learning. MIT press.
Nightingale, K. P., Anderson, V., Onens, S., Fazil, Q., and Davies, H. (2019). Developing the inclusive curriculum: Is supplementary lecture recording an effective approach in supporting students with specific learning difficulties (splds)? Computers & Education, 130:13–25.
Núñez Fernández, D., Barrientos Porras, F., Gilman, R. H., Vittet Mondonedo, M., Sheen, P., and Zimic, M. (2020). A convolutional neural network for gaze preference detection: A potential tool for diagnostics of autism spectrum disorder in children. arXiv e-prints, pages arXiv–2007.
Orsati, F. T., Schwartzman, J. S., Brunoni, D., Mecca, T., and de Macedo, E. C. (2008). Novas possibilidades na avaliação neuropsicológica dos transtornos invasivos do desenvolvimento: Análise dos movimentos oculares. Avaliação Psicológica: Interamerican Journal of Psychological Assessment, 7(3):281–290.
Ponti, M. A. and da Costa, G. B. P. (2018). Como funciona o deep learning. arXiv preprint arXiv:1806.07908.
Poole, A. and Ball, L. J. (2006). Eye tracking in hci and usability research. In Encyclopedia of human computer interaction, pages 211–219. IGI Global.
Rodrigues, P. and Rosa, P. J. (2019). Eye-tracking as a research methodology in educational context: A spanning framework.
Rogers, C. R. and Carmichael, L. (1942). Counseling and psychotherapy: Newer concepts in practice.
Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition.
Strandberg, A. (2019). Eye movements during reading and reading assessment in swedish school children: a new window on reading difficulties. In Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, pages 1–3.
Sugano, Y., Matsushita, Y., and Sato, Y. (2012). Appearance-based gaze estimation using visual saliency. IEEE transactions on pattern analysis and machine intelligence, 35(2):329–341.
Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A. (2017). Inception-v4, inceptionresnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence.
Tobii (2020). What is eye tracking?
Vargas-Cuentas, N. I., Roman-Gonzalez, A., Gilman, R. H., Barrientos, F., Ting, J., Hidalgo, D., Jensen, K., and Zimic, M. (2017). Developing an eye-tracking algorithm as a potential tool for early diagnosis of autism spectrum disorder in children. PloS one, 12(11):e0188826
Zhang, J., Zhuo, L., Li, Z., and Zhao, Y. (2012). An approach of region of interest detection based on visual attention and gaze tracking. In 2012 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2012), pages 228–233. IEEE.
Zhang, X., Sugano, Y., Fritz, M., and Bulling, A. (2015). Appearance-based gaze estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4511–4520.
Bishop, C. M. (1994). Neural networks and their applications. Review of scientific instruments, 65(6):1803–1832.
Brasil (2018). Base Nacional Comum Curricular. Brasília.
Brazil (1997). Lei de diretrizes e bases da educação nacional. Conselho de Reitores das Universidades Brasileiras
Chen, X. and Chen, Z. (2017). Exploring visual attention using random walks based eye tracking protocols. Journal of Visual Communication and Image Representation, 45:147–155.
Cheng, Y., Lu, F., and Zhang, X. (2018). Appearance-based gaze estimation via evaluation-guided asymmetric regression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 100–115.
Chong, E., Chanda, K., Ye, Z., Southerland, A., Ruiz, N., Jones, R. M., Rozga, A., and Rehg, J. M. (2017). Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1–20.
Costa, A. B. d., Picharillo, A. D. M., and Elias, N. C. (2017). Avaliação de habilidades matemáticas em crianças com síndrome de down e com desenvolvimento típico. Ciência & Educação (Bauru), 23:255–272.
de Araujo Cavalcante, T., Frazão, J., Paiva, A., Maia, I. M., Benitez, P., and Soares, A. (2019). Eye tracking como estratégia de ensino e avalização na educação inclusiva: Aplicação com alunos com autismo. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 1221.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.
Duchowski, A. T. and Duchowski, A. T. (2017). Eye tracking methodology: Theory and practice. Springer
Eckstein, M. K., Guerra-Carrillo, B., Miller Singley, A. T., and Bunge, S. A. (2017). Beyond eye gaze: What else can eye tracking reveal about cognition and cognitive development? Developmental Cognitive Neuroscience, 25:69–91. Sensitive periods across development.
Fischer, T., Chang, H. J., and Demiris, Y. (2018). Rt-gene: Real-time eye gaze estimation in natural environments. In Proceedings of the European Conference on Computer Vision (ECCV), pages 334–352.
George, A. and Routray, A. (2016). Real-time eye gaze direction classification using convolutional neural network. In 2016 International Conference on Signal Processing and Communications (SPCOM), pages 1–5. IEEE.
Halszka, J., Holmqvist, K., and Gruber, H. (2017). Eye tracking in educational science: Theoretical frameworks and research agendas. Journal of eye movement research, 10(1).
Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Jarodzka, H., and Van de ¨ Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. OUP Oxford.
Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., and Torralba, A. (2016). Eye tracking for everyone. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2176–2184.
Lukasova, K., Nucci, M. P., Neto, R. M. d. A., Vieira, G., Sato, J. R., and Amaro Jr, E. (2018). Predictive saccades in children and adults: A combined fmri and eye tracking study. PloS one, 13(5):e0196000.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. (2018). Foundations of machine learning. MIT press.
Nightingale, K. P., Anderson, V., Onens, S., Fazil, Q., and Davies, H. (2019). Developing the inclusive curriculum: Is supplementary lecture recording an effective approach in supporting students with specific learning difficulties (splds)? Computers & Education, 130:13–25.
Núñez Fernández, D., Barrientos Porras, F., Gilman, R. H., Vittet Mondonedo, M., Sheen, P., and Zimic, M. (2020). A convolutional neural network for gaze preference detection: A potential tool for diagnostics of autism spectrum disorder in children. arXiv e-prints, pages arXiv–2007.
Orsati, F. T., Schwartzman, J. S., Brunoni, D., Mecca, T., and de Macedo, E. C. (2008). Novas possibilidades na avaliação neuropsicológica dos transtornos invasivos do desenvolvimento: Análise dos movimentos oculares. Avaliação Psicológica: Interamerican Journal of Psychological Assessment, 7(3):281–290.
Ponti, M. A. and da Costa, G. B. P. (2018). Como funciona o deep learning. arXiv preprint arXiv:1806.07908.
Poole, A. and Ball, L. J. (2006). Eye tracking in hci and usability research. In Encyclopedia of human computer interaction, pages 211–219. IGI Global.
Rodrigues, P. and Rosa, P. J. (2019). Eye-tracking as a research methodology in educational context: A spanning framework.
Rogers, C. R. and Carmichael, L. (1942). Counseling and psychotherapy: Newer concepts in practice.
Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition.
Strandberg, A. (2019). Eye movements during reading and reading assessment in swedish school children: a new window on reading difficulties. In Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, pages 1–3.
Sugano, Y., Matsushita, Y., and Sato, Y. (2012). Appearance-based gaze estimation using visual saliency. IEEE transactions on pattern analysis and machine intelligence, 35(2):329–341.
Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A. (2017). Inception-v4, inceptionresnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence.
Tobii (2020). What is eye tracking?
Vargas-Cuentas, N. I., Roman-Gonzalez, A., Gilman, R. H., Barrientos, F., Ting, J., Hidalgo, D., Jensen, K., and Zimic, M. (2017). Developing an eye-tracking algorithm as a potential tool for early diagnosis of autism spectrum disorder in children. PloS one, 12(11):e0188826
Zhang, J., Zhuo, L., Li, Z., and Zhao, Y. (2012). An approach of region of interest detection based on visual attention and gaze tracking. In 2012 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2012), pages 228–233. IEEE.
Zhang, X., Sugano, Y., Fritz, M., and Bulling, A. (2015). Appearance-based gaze estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4511–4520.
Publicado
16/11/2022
Como Citar
FRAZÃO, Jordão; CAVALCANTE, Tardelly A.; BENITEZ, Priscila; AIRES, Kelson; SOARES, André.
Gaze Estimation em Atividades de Ensino Digitais Utilizando Convolutional Neural Networks e Webcam. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 33. , 2022, Manaus.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 786-797.
DOI: https://doi.org/10.5753/sbie.2022.225792.
