Surrogate-based constrained multi-objective optimization for the compression of CNNs
Resumo
A implantação de redes neurais profundas em dispositivos com recursos limitados apresenta desafios significativos. Para enfrentar esse problema, técnicas como poda e quantização são frequentemente utilizadas. No entanto, essas técnicas exigem ajustes cuidadosos para garantir que o modelo final seja computacionalmente eficiente sem sacrificar muita qualidade. Para automatizar o processo de compressão e melhorar sua robustez e acessibilidade, propomos um Algoritmo Evolucionário Multiobjetivo com Restrições Baseado em Surrogates para a compressão de redes neurais artificiais. Resultados experimentais obtidos para o conjunto de dados CIFAR10 utilizando as arquiteturas ResNet50 e VGG16 indicam que a abordagem proposta permite um ajuste mais eficiente em relação aos algoritmos de poda e quantização, resultando em conjuntos não dominados de maior qualidade em comparação com o método de otimização sem Surrogates. Além disso, nosso método produziu versões mais enxutas das arquiteturas testadas mantendo a precisão.
Palavras-chave:
Convolutional Neural Networks, Multiobjective Optimization, Neural Network Compression
Referências
Bendjillali, R. I., Beladgham, M., Merit, K., and Taleb-Ahmed, A. (2020). Illumination-robust face recognition based on deep convolutional neural networks architectures. Indonesian Journal of Electrical Engineering and Computer Science, 18(2):1015–1027.
Blank, J. and Deb, K. (2020). pymoo: Multi-objective optimization in python. IEEE Access, 8:89497–89509.
Chen, J., Xu, Y., Sun, W., and Huang, L. (2021). Joint sparse neural network compression via multi-application multi-objective optimization. Applied Intelligence, pages 1–18.
Chen, T. and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. https://xgboost.readthedocs.io. Acess in: 04/20/2023.
Coello Coello, C. A., Gonzalez Brambila, S., Figueroa Gamboa, J., Castillo Tapia, M. G., and Hernández G omez, R. (2020). Evolutionary multiobjective optimization: open research areas and some challenges lying ahead. Complex & Intelligent Systems, 6:221–236.
CStoronto (s.d). The CIFAR-10 dataset. Accessed: 2023-04-04.
Fernandes Jr., F. E. and Yen, G. G. (2021). Pruning deep convolutional neural networks architectures with evolution strategy. Information Sciences, 552:29–47.
Goldbarg, M. A. S. d. S. (2021). Análise de técnicas de compressão em redes neurais profundas por poda em dataset de imagens. B.S. thesis, Universidade Federal do Rio Grande do Norte.
Han, S., Mao, H., and Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
Hirsch, L. and Katz, G. (2022). Multi-objective pruning of dense neural networks using deep reinforcement learning. Information Sciences, 610:381–400.
Hong, W., Yang, P., Wang, Y., and Tang, K. (2020). Multi-objective magnitude-based pruning for latency-aware deep neural network compression. In Bäck, T., Preuss, M., Deutz, A., Wang, H., Doerr, C., Emmerich, M., and Trautmann, H., editors, Parallel Problem Solving from Nature – PPSN XVI, pages 470–483, Cham. Springer International Publishing.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., and Ye, Q. (2017). Light-GBM: A highly efficient gradient boosting decision tree. https://lightgbm.readthedocs.io. Acess in: 04/20/2023.
Li, K., Chen, R., Fu, G., and Yao, X. (2018). Two-archive evolutionary algorithm for constrained multiobjective optimization. IEEE Transactions on Evolutionary Computation, 23(2):303–315.
Li, L., Fan, Y., Tse, M., and Lin, K.-Y. (2020). A review of applications in federated learning. Computers & Industrial Engineering, 149:106854.
Liang, T., Glossner, J., Wang, L., Shi, S., and Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830.
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., and Iyengar, S. S. (2018). A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR), 51(5):1–36.
PyTorch, A. D. I. (2018). Pytorch. PyTorch, Q. (2023). QUANTIZATION TUTORIAL. Accessed: 2023-06-29.
Runarsson, T. P. (2004). Constrained evolutionary optimization by approximate ranking and surrogate models. In Yao, X., Burke, E. K., Lozano, J. A., Smith, J., Merelo-Guervós, J. J., Bullinaria, J. A., Rowe, J. E., Tiňo, P., Kabán, A., and Schwefel, H.-P., editors, Parallel Problem Solving from Nature - PPSN VIII, pages 401–410, Berlin, Heidelberg. Springer Berlin Heidelberg.
Silva, R. C. P. (2018). Surrogate problem evaluation and selection for optimization with expensive function evaluations. McGill University (Canada).
Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Sugata, T. and Yang, C. (2017). Leaf app: Leaf recognition with deep convolutional neural networks. In IOP Conference Series: Materials Science and Engineering, volume 273, page 012004. IOP Publishing.
Tong, H., Huang, C., Minku, L. L., and Yao, X. (2021). Surrogate models in evolutionary single-objective optimization: A new taxonomy and experimental study. Information Sciences, 562:414–437.
Vadera, S. and Ameen, S. (2022). Methods for pruning deep neural networks. IEEE Access, 10:63280–63300.
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., and Rellermeyer, J. S. (2020). A survey on distributed machine learning. ACM Comput. Surv., 53(2).
Wang, Z., Luo, T., Li, M., Zhou, J. T., Goh, R. S. M., and Zhen, L. (2021). Evolutionary multi-objective model compression for deep neural networks. IEEE Computational Intelligence Magazine, 16(3):10–21.
Xu, K., Zhang, D., An, J., Liu, L., Liu, L., and Wang, D. (2021). Genexp: Multi-objective pruning for deep neural network based on genetic algorithm. Neurocomputing, 451:81–94.
Blank, J. and Deb, K. (2020). pymoo: Multi-objective optimization in python. IEEE Access, 8:89497–89509.
Chen, J., Xu, Y., Sun, W., and Huang, L. (2021). Joint sparse neural network compression via multi-application multi-objective optimization. Applied Intelligence, pages 1–18.
Chen, T. and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. https://xgboost.readthedocs.io. Acess in: 04/20/2023.
Coello Coello, C. A., Gonzalez Brambila, S., Figueroa Gamboa, J., Castillo Tapia, M. G., and Hernández G omez, R. (2020). Evolutionary multiobjective optimization: open research areas and some challenges lying ahead. Complex & Intelligent Systems, 6:221–236.
CStoronto (s.d). The CIFAR-10 dataset. Accessed: 2023-04-04.
Fernandes Jr., F. E. and Yen, G. G. (2021). Pruning deep convolutional neural networks architectures with evolution strategy. Information Sciences, 552:29–47.
Goldbarg, M. A. S. d. S. (2021). Análise de técnicas de compressão em redes neurais profundas por poda em dataset de imagens. B.S. thesis, Universidade Federal do Rio Grande do Norte.
Han, S., Mao, H., and Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
Hirsch, L. and Katz, G. (2022). Multi-objective pruning of dense neural networks using deep reinforcement learning. Information Sciences, 610:381–400.
Hong, W., Yang, P., Wang, Y., and Tang, K. (2020). Multi-objective magnitude-based pruning for latency-aware deep neural network compression. In Bäck, T., Preuss, M., Deutz, A., Wang, H., Doerr, C., Emmerich, M., and Trautmann, H., editors, Parallel Problem Solving from Nature – PPSN XVI, pages 470–483, Cham. Springer International Publishing.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., and Ye, Q. (2017). Light-GBM: A highly efficient gradient boosting decision tree. https://lightgbm.readthedocs.io. Acess in: 04/20/2023.
Li, K., Chen, R., Fu, G., and Yao, X. (2018). Two-archive evolutionary algorithm for constrained multiobjective optimization. IEEE Transactions on Evolutionary Computation, 23(2):303–315.
Li, L., Fan, Y., Tse, M., and Lin, K.-Y. (2020). A review of applications in federated learning. Computers & Industrial Engineering, 149:106854.
Liang, T., Glossner, J., Wang, L., Shi, S., and Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830.
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., and Iyengar, S. S. (2018). A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR), 51(5):1–36.
PyTorch, A. D. I. (2018). Pytorch. PyTorch, Q. (2023). QUANTIZATION TUTORIAL. Accessed: 2023-06-29.
Runarsson, T. P. (2004). Constrained evolutionary optimization by approximate ranking and surrogate models. In Yao, X., Burke, E. K., Lozano, J. A., Smith, J., Merelo-Guervós, J. J., Bullinaria, J. A., Rowe, J. E., Tiňo, P., Kabán, A., and Schwefel, H.-P., editors, Parallel Problem Solving from Nature - PPSN VIII, pages 401–410, Berlin, Heidelberg. Springer Berlin Heidelberg.
Silva, R. C. P. (2018). Surrogate problem evaluation and selection for optimization with expensive function evaluations. McGill University (Canada).
Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Sugata, T. and Yang, C. (2017). Leaf app: Leaf recognition with deep convolutional neural networks. In IOP Conference Series: Materials Science and Engineering, volume 273, page 012004. IOP Publishing.
Tong, H., Huang, C., Minku, L. L., and Yao, X. (2021). Surrogate models in evolutionary single-objective optimization: A new taxonomy and experimental study. Information Sciences, 562:414–437.
Vadera, S. and Ameen, S. (2022). Methods for pruning deep neural networks. IEEE Access, 10:63280–63300.
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., and Rellermeyer, J. S. (2020). A survey on distributed machine learning. ACM Comput. Surv., 53(2).
Wang, Z., Luo, T., Li, M., Zhou, J. T., Goh, R. S. M., and Zhen, L. (2021). Evolutionary multi-objective model compression for deep neural networks. IEEE Computational Intelligence Magazine, 16(3):10–21.
Xu, K., Zhang, D., An, J., Liu, L., Liu, L., and Wang, D. (2021). Genexp: Multi-objective pruning for deep neural network based on genetic algorithm. Neurocomputing, 451:81–94.
Publicado
25/09/2023
Como Citar
FERREIRA, Gabriel Bicalho; BARROS, Antônio de; IBRAHIM, Issah; SILVA, Rodrigo.
Surrogate-based constrained multi-objective optimization for the compression of CNNs. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 20. , 2023, Belo Horizonte/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 184-198.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2023.233874.
