Surrogate-based constrained multi-objective optimization for the compression of CNNs
Abstract
The deployment of deep neural networks on devices with limited resources presents significant challenges. To address this issue, techniques such as pruning and quantization are frequently used. However, these techniques require careful tuning to ensure that the final model is computationally efficient without sacrificing too much quality.To automate the compression process and improve its robustness and affordability, we propose a Surrogate-based Multiobjective Constrained Evolutionary Algorithm for compressing artificial neural networks. Experimental results obtained for the CIFAR10 dataset using the ResNet50 and VGG16 architectures indicate that the proposed approach enables more efficient tuning related to pruning and quantization algorithms, resulting in higher quality non-dominated sets compared to the plain optimization method. Furthermore, our method produced leaner versions of the tested models while maintaining accuracy.
Keywords:
Convolutional Neural Networks, Multiobjective Optimization, Neural Network Compression
References
Bendjillali, R. I., Beladgham, M., Merit, K., and Taleb-Ahmed, A. (2020). Illumination-robust face recognition based on deep convolutional neural networks architectures. Indonesian Journal of Electrical Engineering and Computer Science, 18(2):1015–1027.
Blank, J. and Deb, K. (2020). pymoo: Multi-objective optimization in python. IEEE Access, 8:89497–89509.
Chen, J., Xu, Y., Sun, W., and Huang, L. (2021). Joint sparse neural network compression via multi-application multi-objective optimization. Applied Intelligence, pages 1–18.
Chen, T. and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. https://xgboost.readthedocs.io. Acess in: 04/20/2023.
Coello Coello, C. A., Gonzalez Brambila, S., Figueroa Gamboa, J., Castillo Tapia, M. G., and Hernández G omez, R. (2020). Evolutionary multiobjective optimization: open research areas and some challenges lying ahead. Complex & Intelligent Systems, 6:221–236.
CStoronto (s.d). The CIFAR-10 dataset. Accessed: 2023-04-04.
Fernandes Jr., F. E. and Yen, G. G. (2021). Pruning deep convolutional neural networks architectures with evolution strategy. Information Sciences, 552:29–47.
Goldbarg, M. A. S. d. S. (2021). Análise de técnicas de compressão em redes neurais profundas por poda em dataset de imagens. B.S. thesis, Universidade Federal do Rio Grande do Norte.
Han, S., Mao, H., and Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
Hirsch, L. and Katz, G. (2022). Multi-objective pruning of dense neural networks using deep reinforcement learning. Information Sciences, 610:381–400.
Hong, W., Yang, P., Wang, Y., and Tang, K. (2020). Multi-objective magnitude-based pruning for latency-aware deep neural network compression. In Bäck, T., Preuss, M., Deutz, A., Wang, H., Doerr, C., Emmerich, M., and Trautmann, H., editors, Parallel Problem Solving from Nature – PPSN XVI, pages 470–483, Cham. Springer International Publishing.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., and Ye, Q. (2017). Light-GBM: A highly efficient gradient boosting decision tree. https://lightgbm.readthedocs.io. Acess in: 04/20/2023.
Li, K., Chen, R., Fu, G., and Yao, X. (2018). Two-archive evolutionary algorithm for constrained multiobjective optimization. IEEE Transactions on Evolutionary Computation, 23(2):303–315.
Li, L., Fan, Y., Tse, M., and Lin, K.-Y. (2020). A review of applications in federated learning. Computers & Industrial Engineering, 149:106854.
Liang, T., Glossner, J., Wang, L., Shi, S., and Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830.
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., and Iyengar, S. S. (2018). A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR), 51(5):1–36.
PyTorch, A. D. I. (2018). Pytorch. PyTorch, Q. (2023). QUANTIZATION TUTORIAL. Accessed: 2023-06-29.
Runarsson, T. P. (2004). Constrained evolutionary optimization by approximate ranking and surrogate models. In Yao, X., Burke, E. K., Lozano, J. A., Smith, J., Merelo-Guervós, J. J., Bullinaria, J. A., Rowe, J. E., Tiňo, P., Kabán, A., and Schwefel, H.-P., editors, Parallel Problem Solving from Nature - PPSN VIII, pages 401–410, Berlin, Heidelberg. Springer Berlin Heidelberg.
Silva, R. C. P. (2018). Surrogate problem evaluation and selection for optimization with expensive function evaluations. McGill University (Canada).
Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Sugata, T. and Yang, C. (2017). Leaf app: Leaf recognition with deep convolutional neural networks. In IOP Conference Series: Materials Science and Engineering, volume 273, page 012004. IOP Publishing.
Tong, H., Huang, C., Minku, L. L., and Yao, X. (2021). Surrogate models in evolutionary single-objective optimization: A new taxonomy and experimental study. Information Sciences, 562:414–437.
Vadera, S. and Ameen, S. (2022). Methods for pruning deep neural networks. IEEE Access, 10:63280–63300.
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., and Rellermeyer, J. S. (2020). A survey on distributed machine learning. ACM Comput. Surv., 53(2).
Wang, Z., Luo, T., Li, M., Zhou, J. T., Goh, R. S. M., and Zhen, L. (2021). Evolutionary multi-objective model compression for deep neural networks. IEEE Computational Intelligence Magazine, 16(3):10–21.
Xu, K., Zhang, D., An, J., Liu, L., Liu, L., and Wang, D. (2021). Genexp: Multi-objective pruning for deep neural network based on genetic algorithm. Neurocomputing, 451:81–94.
Blank, J. and Deb, K. (2020). pymoo: Multi-objective optimization in python. IEEE Access, 8:89497–89509.
Chen, J., Xu, Y., Sun, W., and Huang, L. (2021). Joint sparse neural network compression via multi-application multi-objective optimization. Applied Intelligence, pages 1–18.
Chen, T. and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. https://xgboost.readthedocs.io. Acess in: 04/20/2023.
Coello Coello, C. A., Gonzalez Brambila, S., Figueroa Gamboa, J., Castillo Tapia, M. G., and Hernández G omez, R. (2020). Evolutionary multiobjective optimization: open research areas and some challenges lying ahead. Complex & Intelligent Systems, 6:221–236.
CStoronto (s.d). The CIFAR-10 dataset. Accessed: 2023-04-04.
Fernandes Jr., F. E. and Yen, G. G. (2021). Pruning deep convolutional neural networks architectures with evolution strategy. Information Sciences, 552:29–47.
Goldbarg, M. A. S. d. S. (2021). Análise de técnicas de compressão em redes neurais profundas por poda em dataset de imagens. B.S. thesis, Universidade Federal do Rio Grande do Norte.
Han, S., Mao, H., and Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
Hirsch, L. and Katz, G. (2022). Multi-objective pruning of dense neural networks using deep reinforcement learning. Information Sciences, 610:381–400.
Hong, W., Yang, P., Wang, Y., and Tang, K. (2020). Multi-objective magnitude-based pruning for latency-aware deep neural network compression. In Bäck, T., Preuss, M., Deutz, A., Wang, H., Doerr, C., Emmerich, M., and Trautmann, H., editors, Parallel Problem Solving from Nature – PPSN XVI, pages 470–483, Cham. Springer International Publishing.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., and Ye, Q. (2017). Light-GBM: A highly efficient gradient boosting decision tree. https://lightgbm.readthedocs.io. Acess in: 04/20/2023.
Li, K., Chen, R., Fu, G., and Yao, X. (2018). Two-archive evolutionary algorithm for constrained multiobjective optimization. IEEE Transactions on Evolutionary Computation, 23(2):303–315.
Li, L., Fan, Y., Tse, M., and Lin, K.-Y. (2020). A review of applications in federated learning. Computers & Industrial Engineering, 149:106854.
Liang, T., Glossner, J., Wang, L., Shi, S., and Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830.
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., and Iyengar, S. S. (2018). A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR), 51(5):1–36.
PyTorch, A. D. I. (2018). Pytorch. PyTorch, Q. (2023). QUANTIZATION TUTORIAL. Accessed: 2023-06-29.
Runarsson, T. P. (2004). Constrained evolutionary optimization by approximate ranking and surrogate models. In Yao, X., Burke, E. K., Lozano, J. A., Smith, J., Merelo-Guervós, J. J., Bullinaria, J. A., Rowe, J. E., Tiňo, P., Kabán, A., and Schwefel, H.-P., editors, Parallel Problem Solving from Nature - PPSN VIII, pages 401–410, Berlin, Heidelberg. Springer Berlin Heidelberg.
Silva, R. C. P. (2018). Surrogate problem evaluation and selection for optimization with expensive function evaluations. McGill University (Canada).
Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Sugata, T. and Yang, C. (2017). Leaf app: Leaf recognition with deep convolutional neural networks. In IOP Conference Series: Materials Science and Engineering, volume 273, page 012004. IOP Publishing.
Tong, H., Huang, C., Minku, L. L., and Yao, X. (2021). Surrogate models in evolutionary single-objective optimization: A new taxonomy and experimental study. Information Sciences, 562:414–437.
Vadera, S. and Ameen, S. (2022). Methods for pruning deep neural networks. IEEE Access, 10:63280–63300.
Verbraeken, J., Wolting, M., Katzy, J., Kloppenburg, J., Verbelen, T., and Rellermeyer, J. S. (2020). A survey on distributed machine learning. ACM Comput. Surv., 53(2).
Wang, Z., Luo, T., Li, M., Zhou, J. T., Goh, R. S. M., and Zhen, L. (2021). Evolutionary multi-objective model compression for deep neural networks. IEEE Computational Intelligence Magazine, 16(3):10–21.
Xu, K., Zhang, D., An, J., Liu, L., Liu, L., and Wang, D. (2021). Genexp: Multi-objective pruning for deep neural network based on genetic algorithm. Neurocomputing, 451:81–94.
Published
2023-09-25
How to Cite
FERREIRA, Gabriel Bicalho; BARROS, Antônio de; IBRAHIM, Issah; SILVA, Rodrigo.
Surrogate-based constrained multi-objective optimization for the compression of CNNs. In: NATIONAL MEETING ON ARTIFICIAL AND COMPUTATIONAL INTELLIGENCE (ENIAC), 20. , 2023, Belo Horizonte/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 184-198.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2023.233874.
