Optimizing Complex Neural Networks with Population-Based Genetic Algorithms

• Mateus de Freitas Rosa UFU
• Murillo Guimarães Carneiro UFU

DOI: https://doi.org/10.5753/eniac.2025.14265

Abstract

The design of Convolutional Neural Network (CNN) architectures is a complex and costly task, requiring considerable expertise and experimentation. Neural Architecture Search (NAS) aims to automate this process. This work presents a NAS system based on Genetic Algorithms (GA) that employs a graph-based representation for network architectures. The system uses customized genetic operators to directly manipulate the network topology, including convolutional and dense layers. The fitness function considers both performance metrics and the number of parameters, aiming to balance effectiveness and efficiency. To reduce training time for unpromising architectures, an early stopping mechanism based on batches was developed. Mechanisms such as elitism and automatic correction of invalid graphs (e.g., cycles, incompatible dimensions) are also integrated into the evolutionary process. Experiments conducted on the CIFAR-10 and SVHN datasets demonstrate the system’s ability to evolve competitive architectures. The main contribution is a robust and adaptable framework for exploring the neural architecture search space.

References

Covington, P., Adams, J., and Sargin, E. (2016). Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pages 191–198.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.

Ding, Y., Wu, Y., Huang, C., Tang, S., Wu, F., Yang, Y., Zhu, W., and Zhuang, Y. (2022). Nap: Neural architecture search with pruning. Neurocomputing, 477:85–95.

Elsken, T., Metzen, J. H., and Hutter, F. (2019). Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.

Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.

Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2019). Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32.

Liu, H., Simonyan, K., and Yang, Y. (2018). Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055.

Paszke, A. (2019). Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703.

Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. (2019). Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780–4789.

Tan, H., Cheng, R., Huang, S., He, C., Qiu, C., Yang, F., and Luo, P. (2021). Relativenas: Relative neural architecture search via slow-fast learning. IEEE Transactions on Neural Networks and Learning Systems, 34(1):475–489.

Wang, X., Guo, W., Su, J., Yang, X., and Yan, J. (2022). Zarts: On zero-order optimization for neural architecture search. Advances in Neural Information Processing Systems, 35:12868–12880.

Yu, H., Peng, H., Huang, Y., Fu, J., Du, H., Wang, L., and Ling, H. (2022). Cyclic differentiable architecture search. IEEE transactions on pattern analysis and machine intelligence, 45(1):211–228.

Zagoruyko, S. and Komodakis, N. (2016). Wide residual networks. arXiv preprint arXiv:1605.07146.

Zoph, B. and Le, Q. V. (2016). Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.