Towards a tool for identification of emotions and integration with serious game learning
Resumo
The use of interactive systems in the school context, such as educational games and virtual learning environments, has become increasingly common, especially due to their playful nature, which leads to a more attractive and interesting learning process for the student. However, the development of these systems needs adequate metrics for their evaluation so that their problems are explicit, clear, and objective. Therefore, observing the user’s emotional states during an interactive tool is highlighted as an important methodology for classifying facial expressions. In view of the above, the present work aims to develop a convolutional neural network model capable of classifying emotions in order to, in future works, create an application capable of recognizing the emotions of a user through a webcam. Thus, a model was developed which, through tests, presented the best accuracy of 94.69%, a value that, when compared with the best value obtained in related works (97%), indicates the quality of the model’s architecture the relevance of its use in possible future work.
Palavras-chave:
convolutional neural network, emotion recognition, serious games
Referências
Renan Vinicius Aranha. 2022. Adaptação automática de jogos sérios a partir de emoções e traços de personalidade.Ph. D. Dissertation. Universidade de São Paulo.
Renan Vinicius Aranha and Fátima LS Nunes. 2021. De expressões faciais à experiência do jogador: desafios da adaptação automática de jogos sérios com Computação Afetiva. In Anais Estendidos do XXIII Simpósio de Realidade Virtual e Aumentada. SBC, 5–6.
Gustavo Enrique de Almeida Prado Batista 2003. Pré-processamento de dados em aprendizado de máquina supervisionado. Ph. D. Dissertation. Universidade de São Paulo.
James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of machine learning research 13, 2 (2012).
Boyan Bontchev and Dessislava Vassileva. 2017. Affect-based adaptation of an applied video game for educational purposes. Interactive Technology and Smart Education 14, 1 (2017), 31–49.
Clara Pereira Coutinho and Eliana Santana Lisbôa. 2011. Sociedade da informação, do conhecimento e da aprendizagem: desafios para educação no século XXI. (2011).
Zixiang Fei, Erfu Yang, David Day-Uei Li, Stephen Butler, Winifred Ijomah, Xia Li, and Huiyu Zhou. 2020. Deep convolution network based emotion analysis towards mental health care. Neurocomputing 388 (2020), 212–227.
Ernani Gottardo and Andrey Ricardo Pimentel. 2016. Sistemas Tutores Afetivos: Estado da Arte e Desafios para o Reconhecimento de Emoções Relacionadas à Aprendizagem via Expressões Faciais. In Anais do V Workshop de Desafios da Computação aplicada à Educação. SBC, 557–566.
Xing Hao, Guigang Zhang, and Shang Ma. 2016. Deep learning. International Journal of Semantic Computing 10, 03 (2016), 417–439.
Seyyed Hossein Hasanpour, Mohammad Rouhani, Mohsen Fayyaz, and Mohammad Sabokrou. 2016. Lets keep it simple, using simple architectures to outperform deeper and more complex architectures. arXiv preprint arXiv:1608.06037 (2016).
Deepak Kumar Jain, Pourya Shamsolmoali, and Paramjit Sehdev. 2019. Extended deep neural network for facial emotion recognition. Pattern Recognition Letters 120 (2019), 69–74.
Takeo Kanade, Jeffrey F Cohn, and Yingli Tian. 2000. Comprehensive database for facial expression analysis. In Proceedings fourth IEEE international conference on automatic face and gesture recognition (cat. No. PR00580). IEEE, 46–53.
Siddharth Karanchery and Suja Palaniswamy. 2021. Emotion recognition using one-shot learning for human-computer interactions. In 2021 International conference on communication, control and information sciences (ICCISc), Vol. 1. IEEE, 1–8.
Naveen Kumari and Rekha Bhatia. 2022. Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. Soft Computing 26, 16 (2022), 7817–7830.
Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. 2010. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 ieee computer society conference on computer vision and pattern recognition-workshops. IEEE, 94–101.
Daniel McDuff, Abdelrahman Mahmoud, Mohammad Mavadati, May Amr, Jay Turcot, and Rana el Kaliouby. 2016. AFFDEX SDK: a cross-platform real-time multi-face expression recognition toolkit. In Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. 3723–3726.
Ninad Mehendale. 2020. Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences 2, 3 (2020), 1–8.
Vincent Muhler. 2018. face-api. js-JavaScript API for Face Recognition in the Browser with tensorflow. js. [link]. Medium (2018).
Jorge Luis Melo Ribeiro. 2018. Meta-aprendizado aplicado ao Problema de Reconhecimento de Expressões Faciais. (2018).
Cristina Paludo Santos and VR Bourscheidt. 2013. Concepção de jogos educacionais sensíveis ao conteúdo. Nuevas Ideas en Informática Educativa TISE (2013), 826–829.
Alexandre Sant’Anna and Paulo Roberto Nascimento. 2011. A história do lúdico na educação. REVEMAT: Revista Eletrônica de matemática 6, 2 (2011), 19–36.
Klaus Schlünzen, Cicera A Lima Malheiro, Elisa Tomoe Moryia Schlünzen, and Carina Morais Magri. 2016. Games acessíveis para a formação de educadores. Journal of Research in Special Educational Needs 16 (2016), 555–558.
Richard Henrique Herrera Silva. 2022. Análise facial para detecção de estresse e tédio em jogos digitais. (2022).
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
Jiten Sinha. 2019. Realtime Facial Expression Recognition using geometric analysis. [link].
Ivona Tautkute, Tomasz Trzcinski, and Adam Bielski. 2018. I know how you feel: Emotion recognition with facial landmarks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 1878–1880.
Tim Tijs, Dirk Brokken, and Wijnand IJsselsteijn. 2009. Creating an emotionally adaptive game. In Entertainment Computing-ICEC 2008: 7th International Conference, Pittsburgh, PA, USA, September 25-27, 2008. Proceedings 7. Springer, 122–133.
Michael Zyda. 2005. From visual simulation to virtual reality to games. Computer 38, 9 (2005), 25–32.
Renan Vinicius Aranha and Fátima LS Nunes. 2021. De expressões faciais à experiência do jogador: desafios da adaptação automática de jogos sérios com Computação Afetiva. In Anais Estendidos do XXIII Simpósio de Realidade Virtual e Aumentada. SBC, 5–6.
Gustavo Enrique de Almeida Prado Batista 2003. Pré-processamento de dados em aprendizado de máquina supervisionado. Ph. D. Dissertation. Universidade de São Paulo.
James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of machine learning research 13, 2 (2012).
Boyan Bontchev and Dessislava Vassileva. 2017. Affect-based adaptation of an applied video game for educational purposes. Interactive Technology and Smart Education 14, 1 (2017), 31–49.
Clara Pereira Coutinho and Eliana Santana Lisbôa. 2011. Sociedade da informação, do conhecimento e da aprendizagem: desafios para educação no século XXI. (2011).
Zixiang Fei, Erfu Yang, David Day-Uei Li, Stephen Butler, Winifred Ijomah, Xia Li, and Huiyu Zhou. 2020. Deep convolution network based emotion analysis towards mental health care. Neurocomputing 388 (2020), 212–227.
Ernani Gottardo and Andrey Ricardo Pimentel. 2016. Sistemas Tutores Afetivos: Estado da Arte e Desafios para o Reconhecimento de Emoções Relacionadas à Aprendizagem via Expressões Faciais. In Anais do V Workshop de Desafios da Computação aplicada à Educação. SBC, 557–566.
Xing Hao, Guigang Zhang, and Shang Ma. 2016. Deep learning. International Journal of Semantic Computing 10, 03 (2016), 417–439.
Seyyed Hossein Hasanpour, Mohammad Rouhani, Mohsen Fayyaz, and Mohammad Sabokrou. 2016. Lets keep it simple, using simple architectures to outperform deeper and more complex architectures. arXiv preprint arXiv:1608.06037 (2016).
Deepak Kumar Jain, Pourya Shamsolmoali, and Paramjit Sehdev. 2019. Extended deep neural network for facial emotion recognition. Pattern Recognition Letters 120 (2019), 69–74.
Takeo Kanade, Jeffrey F Cohn, and Yingli Tian. 2000. Comprehensive database for facial expression analysis. In Proceedings fourth IEEE international conference on automatic face and gesture recognition (cat. No. PR00580). IEEE, 46–53.
Siddharth Karanchery and Suja Palaniswamy. 2021. Emotion recognition using one-shot learning for human-computer interactions. In 2021 International conference on communication, control and information sciences (ICCISc), Vol. 1. IEEE, 1–8.
Naveen Kumari and Rekha Bhatia. 2022. Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. Soft Computing 26, 16 (2022), 7817–7830.
Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. 2010. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 ieee computer society conference on computer vision and pattern recognition-workshops. IEEE, 94–101.
Daniel McDuff, Abdelrahman Mahmoud, Mohammad Mavadati, May Amr, Jay Turcot, and Rana el Kaliouby. 2016. AFFDEX SDK: a cross-platform real-time multi-face expression recognition toolkit. In Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. 3723–3726.
Ninad Mehendale. 2020. Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences 2, 3 (2020), 1–8.
Vincent Muhler. 2018. face-api. js-JavaScript API for Face Recognition in the Browser with tensorflow. js. [link]. Medium (2018).
Jorge Luis Melo Ribeiro. 2018. Meta-aprendizado aplicado ao Problema de Reconhecimento de Expressões Faciais. (2018).
Cristina Paludo Santos and VR Bourscheidt. 2013. Concepção de jogos educacionais sensíveis ao conteúdo. Nuevas Ideas en Informática Educativa TISE (2013), 826–829.
Alexandre Sant’Anna and Paulo Roberto Nascimento. 2011. A história do lúdico na educação. REVEMAT: Revista Eletrônica de matemática 6, 2 (2011), 19–36.
Klaus Schlünzen, Cicera A Lima Malheiro, Elisa Tomoe Moryia Schlünzen, and Carina Morais Magri. 2016. Games acessíveis para a formação de educadores. Journal of Research in Special Educational Needs 16 (2016), 555–558.
Richard Henrique Herrera Silva. 2022. Análise facial para detecção de estresse e tédio em jogos digitais. (2022).
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
Jiten Sinha. 2019. Realtime Facial Expression Recognition using geometric analysis. [link].
Ivona Tautkute, Tomasz Trzcinski, and Adam Bielski. 2018. I know how you feel: Emotion recognition with facial landmarks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 1878–1880.
Tim Tijs, Dirk Brokken, and Wijnand IJsselsteijn. 2009. Creating an emotionally adaptive game. In Entertainment Computing-ICEC 2008: 7th International Conference, Pittsburgh, PA, USA, September 25-27, 2008. Proceedings 7. Springer, 122–133.
Michael Zyda. 2005. From visual simulation to virtual reality to games. Computer 38, 9 (2005), 25–32.
Publicado
16/10/2023
Como Citar
MARTINS DA COSTA COURA MARINHO, Laura; GOMES CAMPOS BIANCHI, Andrea; DELABRIDA, Saul.
Towards a tool for identification of emotions and integration with serious game learning. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
