Facial Expression Recognition to Aid Visually Impaired People
Resumo
Facial expression recognition systems can help a visually impaired person to identify the emotions of the person with whom she interacts, assisting in her non-verbal communication. Among the various researches carried out in recent years on recognition of facial expressions, the best results obtained come from methods that use deep learning, mainly with the use of convolutional neural networks. This work presents a literature review on the problem of recognition of facial expressions, through the use of convolutional neural networks and proposes two approaches in which the first one uses pre-trained CNN models together with the Linear SVM classifier that, applied to the bases CK+ and JAFFE data, obtained maximum accuracy of 89.6% and 95.7%, respectively. And in the second approach, a CNN model built from scratch is used with the CK+ and FER2013 databases, which obtained accuracy rates of 85% and 65.8%, respectively.
Palavras-chave:
facial expression recognition, convolutional neural network, deep learning, visually impaired people
Referências
A. Mehrabian and J. A. Russell, “An approach to environmental psychology,” MIT, 1974.
A. Ashok and J. John, “Facial expression recognition system for visually impaired,” In: International Conference on Intelligent Data Communication Technologies and Internet of Things (ICICI 2018), pp. 244–250, 2018.
H. Hakim and A. Fadhil, “Survey: Convolution neural networks in object detection,” Journal of Physics: Conference Series, vol. 1804, pp. 22–23, 10 2020.
M. A. Ponti, L. S. F. Ribeiro, T. S. Nazare, T. Bui, and J. Collomosse, “Everything you wanted to know about deep learning for computer vision but were afraid to ask,” 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T), 2017.
L. Britto Neto, V. R. M. L. Maike, F. L. Koch, M. C. C. Baranauskas, A. Rocha, and S. Goldenstein, “A wearable face recognition system built into a smartwatch and the visually impaired user,” in ICEIS, INSTICC. SciTePress, 2015, pp. 5–12.
L. Britto Neto, F. Grijalva, V. R. M. L. Maike, L. C. Martini, D. Florencio, M. C. C. Baranauskas, A. Rocha, and S. Goldenstein, “A kinectbased wearable face recognition system to aid visually impaired users,” IEEE THMS, vol. 47, no. 1, pp. 52–64, 2017.
S. Li and W. Deng, “Deep facial expression recognition: A survey,” Computer Vision and Pattern Recognition, 2018.
A. Sajjanhar, Z. Wu, and Q. Wen, “Deep learning models for facial expression recognition,” 2018 Digital Image Computing: Techniques and Applications (DICTA), 2018.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 2818–2826, 2016.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” CoRR, vol. abs/1409.1556, 2015. [Online]. Available: http://arxiv.org/abs/1409.1556
O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” British machine vision conference, 2015.
D. Silver and K. Bennett, “Guest editor’s introduction: Special issue on inductive transfer learning,” Machine Learning, vol. 73, pp. 215–220, 12 2008.
P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete expression dataset for action unit and emotion-specified expression,” Proceedings of the Third International Workshop on CVPR for Human Communicative Behavior Analysis (CVPR4HB 2010), pp. 94–101, 2010.
M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, “Coding facial expressions with gabor wavelets,” 3rd IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205, 1998.
N. C. Ebner, M. Riediger, and U. Lindenberger, “Faces—a database of facial expressions in young, middle-aged, and older women and men: Development and validation,” Behavior research methods, pp. 351–362, 2010.
W. Wu, Y. Yin, Y. Wang, X. Wang, and D. Xu, “Facial expression recognition for different pose faces based on special landmark detection,” 2018 24th International Conference on Pattern Recognition (ICPR), 2018.
M. Valstar and M. Pantic, “Induced disgust, happiness and surprise: an addition to the mmi facial expression database,” in Proc. IEEE Int. Workshop on EMOTION: Corpora for Research on Emotion and Affect, 2010.
G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. Pietikäinen, “Facial expression recognition from near-infrared videos,” Image and vision computing, p. 607–619, 2011.
M. Georgescu, R. T. Ionescu, and M. Popescu, “Local learning with deep and handcrafted features for facial expression recognition,” IEEE Access, vol. 7, pp. 64 827–64 836, 2019.
Sivic and Zisserman, “Video google: a text retrieval approach to object matching in videos,” in Proceedings Ninth IEEE International Conference on Computer Vision, 2003, pp. 1470–1477 vol.2.
K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” BMVC 2014 - Proceedings of the British Machine Vision Conference 2014, 05 2014.
E. Barsoum, C. Zhang, C. Ferrer, and Z. Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” pp. 279–283, 08 2016.
S. Han, J. Pool, S. Narang, H. Mao, S. Tang, E. Elsen, B. Catanzaro, J. Tran, and W. J. Dally, “DSD: regularizing deep neural networks with dense-sparse-dense training flow,” CoRR, vol. abs/1607.04381, 2016. [Online]. Available: http://arxiv.org/abs/1607.04381
J. Goldberger, G. E. Hinton, S. Roweis, and R. R. Salakhutdinov, “Neighbourhood components analysis,” in Advances in Neural Information Processing Systems, L. Saul, Y. Weiss, and L. Bottou, Eds., vol. 17. MIT Press, 2005, pp. 513–520. [Online]. Available: [link].
C. Cortes and V. Vapnik, “Support vector networks,” Machine Learning, vol. 20, pp. 273–297, 1995.
R. E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, “Liblinear: A library for large linear classification,” in JMLR, 2008.
I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, and Y. Bengio, “Challenges in representation learning: A report on three machine learning contests,” in International Conference on Neural Information Processing, p. 117–124, 2013.
A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, Jan 2017.
J.-H. Kim, B.-G. Kim, P. P. Roy, and D.-M. Jeong, “Efficient facial expression recognition algorithm based on hierarchical deep neural network structure,” IEEE Access, 2019.
S. Xie and H. Hu, “Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks,” IEEE Transactions on Multimedia, 2019.
A. T. Lopes, E. de Aguiar, A. F. D. Souza, and T. Oliveira-Santos, “Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order,” Pattern Recognit, p. 610–628, 2017.
M. I. Ul Haque and D. Valles, “Facial expression recognition using dcnn and development of an ios app for children with asd to enhance communication abilities,” in 2019 IEEE 10th Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), 2019, pp. 0476–0482.
J. Zou, X. Cao, S. Zhang, and B. Ge, “A facial expression recognition based on improved convolutional neural network,” in 2019 IEEE International Conference of Intelligent Applied Systems on Engineering (ICIASE), 2019, pp. 301–304.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016. [Online]. Available: http://www.deeplearningbook.org
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” CoRR, vol. abs/1502.03167, 2015. [Online]. Available: http://arxiv.org/abs/1502.03167
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014. [Online]. Available: http://jmlr.org/papers/v15/srivastava14a.html
D. M. Hawkins, “The problem of overfitting,” Journal of Chemical Information and Computer Sciences, vol. 44, no. 1, pp. 1–12, 2004, pMID: 14741005. [Online]. Available: https://doi.org/10.1021/ci0342472
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, p. 84–90, May 2017. [Online]. Available: https://doi.org/10.1145/3065386
Q. Hu, C. Wu, J. Chi, X. Yu, and H. Wang, “Multi-level feature fusion facial expression recognition network,” in 2020 Chinese Control And Decision Conference (CCDC), 2020, pp. 5267–5272.
G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR, vol. abs/1608.06993, 2016. [Online]. Available: http://arxiv.org/abs/1608.06993
L. Shao, F. Zhu, and X. Li, “Transfer learning for visual categorization: A survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no. 5, pp. 1019–1034, 2015.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li, “Imagenet large scale visual recognition challenge,” CoRR, vol. abs/1409.0575, 2014. [Online]. Available: http://arxiv.org/abs/1409.0575
C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, pp. 1–48, 2019.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 4510–4520.
P. Viola and M. Jones, “Robust real-time face detection,” International Journal of Computer Vision, vol. 57, pp. 137–154, 05 2004.
P. Refaeilzadeh, L. Tang, and H. Liu, Cross-Validation. Boston, MA: Springer US, 2009, pp. 532–538.
R. A. Virrey and L. C. De Silva, “Convolutional neural networks for facial emotion recognition towards the development of automatic pain quantifier,” in 7th Brunei International Conference on Engineering and Technology 2018 (BICET 2018), 2018, pp. 1–4.
A. Ashok and J. John, “Facial expression recognition system for visually impaired,” In: International Conference on Intelligent Data Communication Technologies and Internet of Things (ICICI 2018), pp. 244–250, 2018.
H. Hakim and A. Fadhil, “Survey: Convolution neural networks in object detection,” Journal of Physics: Conference Series, vol. 1804, pp. 22–23, 10 2020.
M. A. Ponti, L. S. F. Ribeiro, T. S. Nazare, T. Bui, and J. Collomosse, “Everything you wanted to know about deep learning for computer vision but were afraid to ask,” 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T), 2017.
L. Britto Neto, V. R. M. L. Maike, F. L. Koch, M. C. C. Baranauskas, A. Rocha, and S. Goldenstein, “A wearable face recognition system built into a smartwatch and the visually impaired user,” in ICEIS, INSTICC. SciTePress, 2015, pp. 5–12.
L. Britto Neto, F. Grijalva, V. R. M. L. Maike, L. C. Martini, D. Florencio, M. C. C. Baranauskas, A. Rocha, and S. Goldenstein, “A kinectbased wearable face recognition system to aid visually impaired users,” IEEE THMS, vol. 47, no. 1, pp. 52–64, 2017.
S. Li and W. Deng, “Deep facial expression recognition: A survey,” Computer Vision and Pattern Recognition, 2018.
A. Sajjanhar, Z. Wu, and Q. Wen, “Deep learning models for facial expression recognition,” 2018 Digital Image Computing: Techniques and Applications (DICTA), 2018.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 2818–2826, 2016.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” CoRR, vol. abs/1409.1556, 2015. [Online]. Available: http://arxiv.org/abs/1409.1556
O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” British machine vision conference, 2015.
D. Silver and K. Bennett, “Guest editor’s introduction: Special issue on inductive transfer learning,” Machine Learning, vol. 73, pp. 215–220, 12 2008.
P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (ck+): A complete expression dataset for action unit and emotion-specified expression,” Proceedings of the Third International Workshop on CVPR for Human Communicative Behavior Analysis (CVPR4HB 2010), pp. 94–101, 2010.
M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, “Coding facial expressions with gabor wavelets,” 3rd IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205, 1998.
N. C. Ebner, M. Riediger, and U. Lindenberger, “Faces—a database of facial expressions in young, middle-aged, and older women and men: Development and validation,” Behavior research methods, pp. 351–362, 2010.
W. Wu, Y. Yin, Y. Wang, X. Wang, and D. Xu, “Facial expression recognition for different pose faces based on special landmark detection,” 2018 24th International Conference on Pattern Recognition (ICPR), 2018.
M. Valstar and M. Pantic, “Induced disgust, happiness and surprise: an addition to the mmi facial expression database,” in Proc. IEEE Int. Workshop on EMOTION: Corpora for Research on Emotion and Affect, 2010.
G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. Pietikäinen, “Facial expression recognition from near-infrared videos,” Image and vision computing, p. 607–619, 2011.
M. Georgescu, R. T. Ionescu, and M. Popescu, “Local learning with deep and handcrafted features for facial expression recognition,” IEEE Access, vol. 7, pp. 64 827–64 836, 2019.
Sivic and Zisserman, “Video google: a text retrieval approach to object matching in videos,” in Proceedings Ninth IEEE International Conference on Computer Vision, 2003, pp. 1470–1477 vol.2.
K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” BMVC 2014 - Proceedings of the British Machine Vision Conference 2014, 05 2014.
E. Barsoum, C. Zhang, C. Ferrer, and Z. Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” pp. 279–283, 08 2016.
S. Han, J. Pool, S. Narang, H. Mao, S. Tang, E. Elsen, B. Catanzaro, J. Tran, and W. J. Dally, “DSD: regularizing deep neural networks with dense-sparse-dense training flow,” CoRR, vol. abs/1607.04381, 2016. [Online]. Available: http://arxiv.org/abs/1607.04381
J. Goldberger, G. E. Hinton, S. Roweis, and R. R. Salakhutdinov, “Neighbourhood components analysis,” in Advances in Neural Information Processing Systems, L. Saul, Y. Weiss, and L. Bottou, Eds., vol. 17. MIT Press, 2005, pp. 513–520. [Online]. Available: [link].
C. Cortes and V. Vapnik, “Support vector networks,” Machine Learning, vol. 20, pp. 273–297, 1995.
R. E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, “Liblinear: A library for large linear classification,” in JMLR, 2008.
I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, and Y. Bengio, “Challenges in representation learning: A report on three machine learning contests,” in International Conference on Neural Information Processing, p. 117–124, 2013.
A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, Jan 2017.
J.-H. Kim, B.-G. Kim, P. P. Roy, and D.-M. Jeong, “Efficient facial expression recognition algorithm based on hierarchical deep neural network structure,” IEEE Access, 2019.
S. Xie and H. Hu, “Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks,” IEEE Transactions on Multimedia, 2019.
A. T. Lopes, E. de Aguiar, A. F. D. Souza, and T. Oliveira-Santos, “Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order,” Pattern Recognit, p. 610–628, 2017.
M. I. Ul Haque and D. Valles, “Facial expression recognition using dcnn and development of an ios app for children with asd to enhance communication abilities,” in 2019 IEEE 10th Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), 2019, pp. 0476–0482.
J. Zou, X. Cao, S. Zhang, and B. Ge, “A facial expression recognition based on improved convolutional neural network,” in 2019 IEEE International Conference of Intelligent Applied Systems on Engineering (ICIASE), 2019, pp. 301–304.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016. [Online]. Available: http://www.deeplearningbook.org
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” CoRR, vol. abs/1502.03167, 2015. [Online]. Available: http://arxiv.org/abs/1502.03167
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014. [Online]. Available: http://jmlr.org/papers/v15/srivastava14a.html
D. M. Hawkins, “The problem of overfitting,” Journal of Chemical Information and Computer Sciences, vol. 44, no. 1, pp. 1–12, 2004, pMID: 14741005. [Online]. Available: https://doi.org/10.1021/ci0342472
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, p. 84–90, May 2017. [Online]. Available: https://doi.org/10.1145/3065386
Q. Hu, C. Wu, J. Chi, X. Yu, and H. Wang, “Multi-level feature fusion facial expression recognition network,” in 2020 Chinese Control And Decision Conference (CCDC), 2020, pp. 5267–5272.
G. Huang, Z. Liu, and K. Q. Weinberger, “Densely connected convolutional networks,” CoRR, vol. abs/1608.06993, 2016. [Online]. Available: http://arxiv.org/abs/1608.06993
L. Shao, F. Zhu, and X. Li, “Transfer learning for visual categorization: A survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no. 5, pp. 1019–1034, 2015.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li, “Imagenet large scale visual recognition challenge,” CoRR, vol. abs/1409.0575, 2014. [Online]. Available: http://arxiv.org/abs/1409.0575
C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, pp. 1–48, 2019.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 4510–4520.
P. Viola and M. Jones, “Robust real-time face detection,” International Journal of Computer Vision, vol. 57, pp. 137–154, 05 2004.
P. Refaeilzadeh, L. Tang, and H. Liu, Cross-Validation. Boston, MA: Springer US, 2009, pp. 532–538.
R. A. Virrey and L. C. De Silva, “Convolutional neural networks for facial emotion recognition towards the development of automatic pain quantifier,” in 7th Brunei International Conference on Engineering and Technology 2018 (BICET 2018), 2018, pp. 1–4.
Publicado
22/11/2021
Como Citar
SILVA, João Marcos; SILVA, Romuere; VERAS, Rodrigo; AIRES, Kelson; BRITTO NETO, Laurindo.
Facial Expression Recognition to Aid Visually Impaired People. In: WORKSHOP DE VISÃO COMPUTACIONAL (WVC), 17. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 48-53.
DOI: https://doi.org/10.5753/wvc.2021.18888.
