Multi-Objective Optimization of Sequence of Layers in Deep Learning Architectures

• Mayara Castro UFRPE
• Giuseppe F. Neto UFRPE
• Péricles de Miranda UFRPE
• Filipe Cordeiro UFRPE
• Ricardo Prudêncio UFPE

DOI: https://doi.org/10.5753/eniac.2019.9269

Abstract

Selecting the best architecture for a Deep Neural Network (DNN) is a non-trivial task since there is a massive amount of possible configurations (layers and parameters) and great difficulty in how to choose them. In order to make this task more independent of human interaction, this paper proposes an intelligent method to optimize the architecture (sequence of layers) of a chain-structured DNN, taking into account multiple criteria: accuracy and F1 score. The method was evaluated for performance and compared to the exhaustive and random approaches. The results obtained showed the potential of the proposed method from a computational point of view.

Keywords: Deeplearning, multi-objective optimization

References

Assunção, F., Lourenço, N., Machado, P., and Ribeiro, B. (2018). Evolving the topology of large scale deep neural networks. In European Conference on Genetic Programming, pages 19–34. Springer.

Black, D. (1976). Partial justication of the borda count. Public Choice, 28(1):1–15.

Chollet, F. et al. (2015). Keras. https://keras.io/. Accessed: 2019-06-12.

David, O. E. and Greental, I. (2014). Genetic algorithms for evolving deep neural In Proceedings of the Annual Conference on Genetic and Evolutionary networks. Computation, pages 1451–1452. ACM.

Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2):182–197.

Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142.

Diniz, J. B., Cordeiro, F. R., Miranda, P. B., and da Silva, L. A. T. (2018). A grammarbased genetic programming approach to optimize convolutional neural network architectures. In Anais do XV Encontro Nacional de Inteligência Articial e Computacional, pages 82–93. SBC.

Elsken, T., Metzen, J. H., and Hutter, F. (2018). Neural architecture search: A survey.arXiv preprint arXiv:1808.05377.

Hadka, D. (2017). Platypus: A free and open source python library for multiobjective optimization. Available on Github, vol. https://github.com/Project-Platypus/Platypus.

Ishibuchi, H., Tsukamoto, N., and Nojima, Y. (2008). Evolutionary many-objective optimization: A short review. In IEEE World Congress on Computational Intelligence)., pages 2419–2426. IEEE.

Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical report, Citeseer.

LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. nature, 521(7553):436.

Liu, S., Yang, J., Huang, C., and Yang, M.-H. (2015). Multi-objective convolutional learning for face labeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3451–3459.

Lorenzo, P. R., Nalepa, J., Kawulok, M., Ramos, L. S., and Pastor, J. R. (2017). Particle swarm optimization for hyper-parameter selection in deep neural networks. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 481–488. ACM.

Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. (2019). Evolving deep neural networks. In Articial Intelligence in the Age of Neural Networks and Brain Computing, pages 293–312. Elsevier.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830.

Schaffer, J. D., Whitley, D., and Eshelman, L. J. (1992). Combinations of genetic alIn [Proceedings] gorithms and neural networks: A survey of the state of the art. COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks, pages 1–37. IEEE.

Wang, W., Zhang, L.-L., Chen, J.-J., and Wang, J.-H. (2015). Parameter estimation for coupled hydromechanical simulation of dynamic compaction based on pareto multiobjective optimization. Shock and Vibration, 2015.

Yang, C., An, Z., Li, C., Diao, B., and Xu, Y. (2019). Multi-objective pruning for cnns using genetic algorithm. arXiv preprint arXiv:1906.00399.

Young, S. R., Rose, D. C., Karnowski, T. P., Lim, S.-H., and Patton, R. M. (2015). Optimizing deep learning hyper-parameters through an evolutionary algorithm. In Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, page 4. ACM.