On the Training Algorithms for Restricted Boltzmann Machines
Resumo
Deep learning techniques have been studied extensively in the last years due to their good results related to essential tasks on a large range of applications, such as speech and face recognition, as well as object classification. Restrict Boltzmann Machines (RBMs) are among the most employed techniques, which are energy-based stochastic neural networks composed of two layers of neurons whose objective is to estimate the connection weights between them. Recently, the scientific community spent much effort on sampling methods since the effectiveness of RBMs is directly related to the success of such a process. Thereby, this work contributes to studies concerning different training algorithms for RBMs, as well as its variants Deep Belief Networks and Deep Boltzmann Machines. Further, the work covers the application of meta-heuristic methods concerning a proper fine-tune of these techniques. Moreover, the validation of the model is presented in the context of image reconstruction and unsupervised feature learning. In general, we present different approaches to training these techniques, as well as the evaluation of meta-heuristic methods for fine-tuning parameters, and its main contributions are: (i) temperature parameter introduction in DBM formulation, (ii) DBM using adaptive temperature, (iii) DBM meta-parameter optimization through meta-heuristic techniques, and (iv) infinity Restricted Boltzmann Machine (iRBM) meta-parameters optimization through meta-heuristic techniques.
Palavras-chave:
Machine Learning, Restricted Boltzmann Machine, Optimization
Referências
P. Smolensky, “Parallel distributed processing: Explorations in the microstructure of cognition,” D. E. Rumelhart, J. L. McClelland, and C. PDP Research Group, Eds. Cambridge, MA, USA: MIT Press, 1986, vol. 1, ch. Information Processing in Dynamical Systems: Foundations of Harmony Theory, pp. 194–281.
G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527–1554, 2006. https://doi.org/10.1162/neco.2006.18.7.1527
R. Salakhutdinov and G. E. Hinton, “An efficient learning procedure for deep boltzmann machines,” Neural Computation, vol. 24, no. 8, pp. 1967–2006, 2012. https://doi.org/10.1162/NECO_a_00311
G. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Computation, vol. 14, no. 8, pp. 1771–1800, 2002. https://doi.org/10.1162/089976602760128018
T. Tieleman, “Training restricted Boltzmann machines using approximations to the likelihood gradient,” in Proceedings of the 25th International Conference on Machine Learning, ser. ICML ’08. New York, NY, USA: ACM, 2008, pp. 1064–1071. https://doi.org/10.1145/1390156.1390290
T. Tieleman and G. E. Hinton, “Using fast weights to improve persistent contrastive divergence,” in Proceedings of the 26th Annual International Conference on Machine Learning, ser. ICML ’09. New York, NY, USA: ACM, 2009, pp. 1033–1040. https://doi.org/10.1145/1553374.1553506
P. Brakel, S. Dieleman, and B. Schrauwen, “Training restricted boltzmann machines with multi-tempering: Harnessing parallelization,” in Artificial Neural Networks and Machine Learning, ser. Lecture Notes in Computer Science, A. E. P. Villa, W. Duch, P. Érdi, F. Masulli, and G. Palm, Eds. Springer Berlin Heidelberg, 2012, vol. 7553, pp. 92–99. https://doi.org/10.1007/978-3-642-33266-1_12
J. Xu, H. Li, and S. Zhou, “Improving mixing rate with tempered transition for learning restricted boltzmann machines,” Neurocomputing, vol. 139, pp. 328–335, 2014. https://doi.org/10.1016/j.neucom.2014.02.024
P. Smolensky, “Information processing in dynamical systems: Foundations of harmony theory,” DTIC Document, Tech. Rep., 1986.
G. E. Hinton, “Neural networks: Tricks of the trade: Second edition,” G. Montavon, G. B. Orr, and K.-R. Müller, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, ch. A Practical Guide to Training Restricted Boltzmann Machines, pp. 599–619. https://doi.org/10.1007/978-3-642-35289-8_32
M. A. Carreira-Perpiñán and G. E. Hinton, “On Contrastive Divergence Learning,” in Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, R. G. Cowell and Z. Ghahramani, Eds. Society for Artificial Intelligence and Statistics, 2005, pp. 33–40.
L. A. Passos and J. P. Papa, “Temperature-based deep boltzmann machines,” Neural Processing Letters, pp. 1–13, 2017. https://doi.org/10.1007/s11063-017-9707-2
G. Li, L. Deng, Y. Xu, C. Wen, W. Wang, J. Pei, and L. Shi, “Temperature based restricted boltzmann machines,” Scientific reports, vol. 6, 2016. https://doi.org/10.1038/srep19133
L. A. Passos, K. A. Costa, and J. P. Papa, “Deep boltzmann machines using adaptive temperatures,” in International Conference on Computer Analysis of Images and Patterns. Springer, 2017, pp. 172–183. https://doi.org/10.1007/978-3-319-64689-3_14
M.-A. Côté and H. Larochelle, “An infinite restricted boltzmann machine,” Neural computation, 2016. https://doi.org/10.1162/NECO_a_00848
L. A. Passos and J. P. Papa, “Fine-tuning infinity restricted boltzmann machines,” in Electronic Proceedings of the 30th Conference on Graphics, Patterns and Images (SIBGRAPI’17), M. Lage, L. A. F. Fernandes, R. Marroquim, and H. Lopes, Eds., Niteri, RJ, Brazil, october 2017. https://doi.org/10.1109/SIBGRAPI.2017.15
G. H. Rosa, J. P. Papa, K. A. P. Costa, L. A. Passos, C. R. Pereira, and X.-S. Yang, Learning Parameters in Deep Belief Networks Through Firefly Algorithm. Cham: Springer International Publishing, 2016, pp. 138–149. https://doi.org/10.1007/978-3-319-46182-3_12
C. R. Pereira, L. A. Passos, R. R. Lopes, S. A. Weber, C. Hook, and J. P. Papa, “Parkinsons disease identification using restricted boltzmann machines,” in International Conference on Computer Analysis of Images and Patterns. Springer, 2017, pp. 70–80. https://doi.org/10.1007/978-3-319-64698-5_7
L. A. Passos and J. P. Papa, “A metaheuristic-driven approach to finetune deep boltzmann machines,” Applied Soft Computing, p. 105717, 2019. https://doi.org/10.1016/j.asoc.2019.105717
L. A. Passos, C. R. Pereira, E. R. Rezende, T. J. Carvalho, S. A. Weber, C. Hook, and J. P. Papa, “Parkinson disease identification using residual networks and optimum-path forest,” in 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI). IEEE, 2018, pp. 000 325–000 330. https://doi.org/10.1109/SACI.2018.8441012
L. C. Afonso, L. A. Passos, and J. a. P. Papa, “Enhancing brain storm optimization through optimum-path forest,” in 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI). IEEE, 2018, pp. 000 183–000 188. https://doi.org/10.1109/SACI.2018.8440918
L. A. Passos, D. R. Rodrigues, and J. P. Papa, “Fine tuning deep boltzmann machines through meta-heuristic approaches,” in 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI). IEEE, 2018, pp. 000 419–000 424. https://doi.org/10.1109/SACI.2018.8440959
R. R. Guimaraes, L. A. Passos, R. Holanda Filho, V. H. C. de Albuquerque, J. J. Rodrigues, M. M. Komarov, and J. P. Papa, “Intelligent network security monitoring based on optimum-path forest clustering,” IEEE Network, 2018. https://doi.org/10.1109/MNET.2018.1800151
L. A. Passos, L. A. de Souza Jr, R. Mendel, A. Ebigbo, A. Probst, H. Messmann, C. Palm, and J. P. Papa, “Barretts esophagus analysis using infinity restricted boltzmann machines,” Journal of Visual Communication and Image Representation, vol. 59, pp. 475–485, 2019. https://doi.org/10.1016/j.jvcir.2019.01.043
P. Khojasteh, L. A. P. Júnior, T. Carvalho, E. Rezende, B. Aliahmad, J. P. Papa, and D. K. Kumar, “Exudate detection in fundus images using deeply-learnable features,” Computers in biology and medicine, vol. 104, pp. 62–69, 2019. https://doi.org/10.1016/j.compbiomed.2018.10.031
L. A. Passos, D. Rodrigues, and J. P. Papa, “Quaternion-based backtracking search optimization algorithm,” in 2019 IEEE Congress on Evolutionary Computation. IEEE, 2019. https://doi.org/10.1109/CEC.2019.8790209
L. A. Passos, M. C. Santana, T. Moreira, and J. P. Papa, “κ-entropy based restricted boltzmann machines,” in The 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.
G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527–1554, 2006. https://doi.org/10.1162/neco.2006.18.7.1527
R. Salakhutdinov and G. E. Hinton, “An efficient learning procedure for deep boltzmann machines,” Neural Computation, vol. 24, no. 8, pp. 1967–2006, 2012. https://doi.org/10.1162/NECO_a_00311
G. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Computation, vol. 14, no. 8, pp. 1771–1800, 2002. https://doi.org/10.1162/089976602760128018
T. Tieleman, “Training restricted Boltzmann machines using approximations to the likelihood gradient,” in Proceedings of the 25th International Conference on Machine Learning, ser. ICML ’08. New York, NY, USA: ACM, 2008, pp. 1064–1071. https://doi.org/10.1145/1390156.1390290
T. Tieleman and G. E. Hinton, “Using fast weights to improve persistent contrastive divergence,” in Proceedings of the 26th Annual International Conference on Machine Learning, ser. ICML ’09. New York, NY, USA: ACM, 2009, pp. 1033–1040. https://doi.org/10.1145/1553374.1553506
P. Brakel, S. Dieleman, and B. Schrauwen, “Training restricted boltzmann machines with multi-tempering: Harnessing parallelization,” in Artificial Neural Networks and Machine Learning, ser. Lecture Notes in Computer Science, A. E. P. Villa, W. Duch, P. Érdi, F. Masulli, and G. Palm, Eds. Springer Berlin Heidelberg, 2012, vol. 7553, pp. 92–99. https://doi.org/10.1007/978-3-642-33266-1_12
J. Xu, H. Li, and S. Zhou, “Improving mixing rate with tempered transition for learning restricted boltzmann machines,” Neurocomputing, vol. 139, pp. 328–335, 2014. https://doi.org/10.1016/j.neucom.2014.02.024
P. Smolensky, “Information processing in dynamical systems: Foundations of harmony theory,” DTIC Document, Tech. Rep., 1986.
G. E. Hinton, “Neural networks: Tricks of the trade: Second edition,” G. Montavon, G. B. Orr, and K.-R. Müller, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, ch. A Practical Guide to Training Restricted Boltzmann Machines, pp. 599–619. https://doi.org/10.1007/978-3-642-35289-8_32
M. A. Carreira-Perpiñán and G. E. Hinton, “On Contrastive Divergence Learning,” in Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, R. G. Cowell and Z. Ghahramani, Eds. Society for Artificial Intelligence and Statistics, 2005, pp. 33–40.
L. A. Passos and J. P. Papa, “Temperature-based deep boltzmann machines,” Neural Processing Letters, pp. 1–13, 2017. https://doi.org/10.1007/s11063-017-9707-2
G. Li, L. Deng, Y. Xu, C. Wen, W. Wang, J. Pei, and L. Shi, “Temperature based restricted boltzmann machines,” Scientific reports, vol. 6, 2016. https://doi.org/10.1038/srep19133
L. A. Passos, K. A. Costa, and J. P. Papa, “Deep boltzmann machines using adaptive temperatures,” in International Conference on Computer Analysis of Images and Patterns. Springer, 2017, pp. 172–183. https://doi.org/10.1007/978-3-319-64689-3_14
M.-A. Côté and H. Larochelle, “An infinite restricted boltzmann machine,” Neural computation, 2016. https://doi.org/10.1162/NECO_a_00848
L. A. Passos and J. P. Papa, “Fine-tuning infinity restricted boltzmann machines,” in Electronic Proceedings of the 30th Conference on Graphics, Patterns and Images (SIBGRAPI’17), M. Lage, L. A. F. Fernandes, R. Marroquim, and H. Lopes, Eds., Niteri, RJ, Brazil, october 2017. https://doi.org/10.1109/SIBGRAPI.2017.15
G. H. Rosa, J. P. Papa, K. A. P. Costa, L. A. Passos, C. R. Pereira, and X.-S. Yang, Learning Parameters in Deep Belief Networks Through Firefly Algorithm. Cham: Springer International Publishing, 2016, pp. 138–149. https://doi.org/10.1007/978-3-319-46182-3_12
C. R. Pereira, L. A. Passos, R. R. Lopes, S. A. Weber, C. Hook, and J. P. Papa, “Parkinsons disease identification using restricted boltzmann machines,” in International Conference on Computer Analysis of Images and Patterns. Springer, 2017, pp. 70–80. https://doi.org/10.1007/978-3-319-64698-5_7
L. A. Passos and J. P. Papa, “A metaheuristic-driven approach to finetune deep boltzmann machines,” Applied Soft Computing, p. 105717, 2019. https://doi.org/10.1016/j.asoc.2019.105717
L. A. Passos, C. R. Pereira, E. R. Rezende, T. J. Carvalho, S. A. Weber, C. Hook, and J. P. Papa, “Parkinson disease identification using residual networks and optimum-path forest,” in 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI). IEEE, 2018, pp. 000 325–000 330. https://doi.org/10.1109/SACI.2018.8441012
L. C. Afonso, L. A. Passos, and J. a. P. Papa, “Enhancing brain storm optimization through optimum-path forest,” in 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI). IEEE, 2018, pp. 000 183–000 188. https://doi.org/10.1109/SACI.2018.8440918
L. A. Passos, D. R. Rodrigues, and J. P. Papa, “Fine tuning deep boltzmann machines through meta-heuristic approaches,” in 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI). IEEE, 2018, pp. 000 419–000 424. https://doi.org/10.1109/SACI.2018.8440959
R. R. Guimaraes, L. A. Passos, R. Holanda Filho, V. H. C. de Albuquerque, J. J. Rodrigues, M. M. Komarov, and J. P. Papa, “Intelligent network security monitoring based on optimum-path forest clustering,” IEEE Network, 2018. https://doi.org/10.1109/MNET.2018.1800151
L. A. Passos, L. A. de Souza Jr, R. Mendel, A. Ebigbo, A. Probst, H. Messmann, C. Palm, and J. P. Papa, “Barretts esophagus analysis using infinity restricted boltzmann machines,” Journal of Visual Communication and Image Representation, vol. 59, pp. 475–485, 2019. https://doi.org/10.1016/j.jvcir.2019.01.043
P. Khojasteh, L. A. P. Júnior, T. Carvalho, E. Rezende, B. Aliahmad, J. P. Papa, and D. K. Kumar, “Exudate detection in fundus images using deeply-learnable features,” Computers in biology and medicine, vol. 104, pp. 62–69, 2019. https://doi.org/10.1016/j.compbiomed.2018.10.031
L. A. Passos, D. Rodrigues, and J. P. Papa, “Quaternion-based backtracking search optimization algorithm,” in 2019 IEEE Congress on Evolutionary Computation. IEEE, 2019. https://doi.org/10.1109/CEC.2019.8790209
L. A. Passos, M. C. Santana, T. Moreira, and J. P. Papa, “κ-entropy based restricted boltzmann machines,” in The 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.
Publicado
28/10/2019
Como Citar
PASSOS, Leandro Aparecido; PAPA, João Paulo.
On the Training Algorithms for Restricted Boltzmann Machines. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 32. , 2019, Rio de Janeiro.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 1-7.
DOI: https://doi.org/10.5753/sibgrapi.est.2019.8294.
