Understanding fully-connected and convolution allayers in unsupervised learning using face images
Resumo
The goal of this paper is to implement and compare two unsupervised models of deep learning: Autoencoder and Convolutional Autoencoder. These neural network models have been trained to learn regularities in well-framed face images with different facial expressions. The Autoencoder's basic topology is addressed here, composed of encoding and decoding multilayers. This paper approaches these automatic codings using multivariate statistics to visually understand the bottleneck differences between the fully-connected and convolutional layers and the corresponding importance of the dropout strategy when applied in a model.
Palavras-chave:
deep neural network, autoencoder, convolutional
autoencoder, multivariate statistics
Referências
ALBAWI, Saad; MOHAMMED, Tareq Abed ; AL-ZAWI, Saad. Understanding of a convolutional neural network. In: IEEE. ENGINEERING and Technology (ICET), 2017 International Conference on. [S.l.: s.n.], p. 1 - 6.
BALDI, Pierre; HORNIK, Kurt. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, Elsevier, v. 2, n. 1, p. 53 - 58, DOI: 10.1016/0893-6080(89)90014-2
C. E. Thomaz and G. A. Giraldi. A new ranking method for Principal Components Analysis and its application to face image analysis, Image and Vision Computing, vol. 28, no. 6, pp. 902 - 913, June DOI: 10.1016/j.imavis.2009.11.005
CHEN, Min et al. Deep features learning for medical image analysis with convolutional autoencoder neural network. IEEE Transactions on Big Data, DOI: 10.1109/tbdata.2017.2717439
GOODFELLOW, Ian; BENGIO, Yoshua; COURVILLE, Aaron. Deep Learning. [S.l.]: MIT Press, http://www.deeplearningbook.org. DOI: 10.1007/s10710-017-9314-z
GOODFELLOW, Ian et al. Generative adversarial nets. In: ADVANCES in neural information processing systems. [S.l.: s.n.], p. 2672 - 2680.
HINTON, Geoffrey E. et al. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580,
HINTON, Geoffrey E ; SALAKHUTDINOV, Ruslan R. Reducing the dimensionality of data with neural networks. science, American Association for the Advancement of Science, v. 313, n. 5786, p. 504 - 507, DOI: 10.1126/science.1127647
HINTON, Geoffrey E ; ZEMEL, Richard S. Autoencoders, minimum description length and Helmholtz free energy. In: ADVANCES in neural information processing systems. [S.l.: s.n.], p. 3 - 10.
KINGMA, Diederik P ; WELLING, Max. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,
LECUN, Yann; BENGIO, Yoshua; HINTON, Geoffrey. Deep learning. nature, v. 521, n. 7553, p. 436, DOI: 10.1038/nature14539
LE CUN, Yann et al. Handwritten digit recognition: Applications of neural network chips and automatic learning. IEEE Communications Magazine, IEEE, v. 27, n. 11, p. 41 - 46, DOI: 10.1109/35.41400
LE CUN, Yann et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, IEEE, v. 86, n. 11, p. 2278 - 2324, DOI: 10.1109/5.726791
MASCI, Jonathan et al. Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks. Springer, Berlin, Heidelberg, p. 52 - 59. DOI: 10.1007/978-3-642-21735-7_7
NAGY, George. State of the art in pattern recognition. Proceedings of the IEEE, v. 56, n. 5, p. 836 - 863, DOI: 10.1109/proc.1968.6414
Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, p. 234 - 241, DOI: 10.1007/978-3-319-24574-4_28
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning internal representations by error propagation. In Rumelhart, D. E. and McClelland, J. L., editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1 : Foundations, MIT Press, Cambridge, MA. pp 318 - 1986. DOI: 10.21236/ada164453
SRIVASTAVA, Nitish et al. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, JMLR. org, v. 15, n. 1, p. 1929 - 1958,
SZEGEDY, Christian et al. Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence.
ZEILER, Matthew D.; FERGUS,Rob. Visualizing and understanding convolutional networks. In: European conference on computer vision. springer, Cham, p. 818 - 833, DOI: 10.1007/978-3-319-10590-1_53
BALDI, Pierre; HORNIK, Kurt. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, Elsevier, v. 2, n. 1, p. 53 - 58, DOI: 10.1016/0893-6080(89)90014-2
C. E. Thomaz and G. A. Giraldi. A new ranking method for Principal Components Analysis and its application to face image analysis, Image and Vision Computing, vol. 28, no. 6, pp. 902 - 913, June DOI: 10.1016/j.imavis.2009.11.005
CHEN, Min et al. Deep features learning for medical image analysis with convolutional autoencoder neural network. IEEE Transactions on Big Data, DOI: 10.1109/tbdata.2017.2717439
GOODFELLOW, Ian; BENGIO, Yoshua; COURVILLE, Aaron. Deep Learning. [S.l.]: MIT Press, http://www.deeplearningbook.org. DOI: 10.1007/s10710-017-9314-z
GOODFELLOW, Ian et al. Generative adversarial nets. In: ADVANCES in neural information processing systems. [S.l.: s.n.], p. 2672 - 2680.
HINTON, Geoffrey E. et al. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580,
HINTON, Geoffrey E ; SALAKHUTDINOV, Ruslan R. Reducing the dimensionality of data with neural networks. science, American Association for the Advancement of Science, v. 313, n. 5786, p. 504 - 507, DOI: 10.1126/science.1127647
HINTON, Geoffrey E ; ZEMEL, Richard S. Autoencoders, minimum description length and Helmholtz free energy. In: ADVANCES in neural information processing systems. [S.l.: s.n.], p. 3 - 10.
KINGMA, Diederik P ; WELLING, Max. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,
LECUN, Yann; BENGIO, Yoshua; HINTON, Geoffrey. Deep learning. nature, v. 521, n. 7553, p. 436, DOI: 10.1038/nature14539
LE CUN, Yann et al. Handwritten digit recognition: Applications of neural network chips and automatic learning. IEEE Communications Magazine, IEEE, v. 27, n. 11, p. 41 - 46, DOI: 10.1109/35.41400
LE CUN, Yann et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, IEEE, v. 86, n. 11, p. 2278 - 2324, DOI: 10.1109/5.726791
MASCI, Jonathan et al. Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks. Springer, Berlin, Heidelberg, p. 52 - 59. DOI: 10.1007/978-3-642-21735-7_7
NAGY, George. State of the art in pattern recognition. Proceedings of the IEEE, v. 56, n. 5, p. 836 - 863, DOI: 10.1109/proc.1968.6414
Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, p. 234 - 241, DOI: 10.1007/978-3-319-24574-4_28
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning internal representations by error propagation. In Rumelhart, D. E. and McClelland, J. L., editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1 : Foundations, MIT Press, Cambridge, MA. pp 318 - 1986. DOI: 10.21236/ada164453
SRIVASTAVA, Nitish et al. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, JMLR. org, v. 15, n. 1, p. 1929 - 1958,
SZEGEDY, Christian et al. Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence.
ZEILER, Matthew D.; FERGUS,Rob. Visualizing and understanding convolutional networks. In: European conference on computer vision. springer, Cham, p. 818 - 833, DOI: 10.1007/978-3-319-10590-1_53
Publicado
09/09/2019
Como Citar
BUZUTI, Lucas Fontes; THOMAZ, Carlos Eduardo.
Understanding fully-connected and convolution allayers in unsupervised learning using face images. In: WORKSHOP DE VISÃO COMPUTACIONAL (WVC), 15. , 2019, São Bernardo do Campo.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 13-18.
DOI: https://doi.org/10.5753/wvc.2019.7621.
