Partial Least Squares: A Deep Space Odyssey

• Artur Jordão UFMG
• William Robson Schwartz UFMG

DOI: https://doi.org/10.5753/sibgrapi.est.2021.20014

Resumo

Modern visual pattern recognition models are based on deep convolutional networks. Such models are computationally expensive, hindering applicability on resource-constrained devices. To handle this problem, we propose three strategies. The first removes unimportant structures (neurons or layers) of convolutional networks, reducing their computational cost. The second inserts structures to design architectures automatically, enabling us to build high-performance networks. The third combines multiple layers of convolutional networks, enhancing data representation at negligible additional cost. These strategies are based on Partial Least Squares (PLS) which, despite promising results, is infeasible on large datasets due to memory constraints. To address this issue, we also propose a discriminative and lowcomplexity incremental PLS that learns a compact representation of the data using a single sample at a time, thus enabling applicability on large datasets. We assess the effectiveness of our approaches on several convolutional architectures and computer vision tasks, which include image classification, face verification and activity recognition. Our approaches reduce the resource overhead of both convolutional networks and Partial Least Squares, promoting energyand hardware-friendly models for the academy and industry scenarios. Compared to state-of-theart methods for the same purpose, we obtain one of the best trade-offs between predictive ability and computational cost.

Referências

A. P. Badia, B. P. S. K. P. S. A. V. Z. Guoand, and C. Blundell, “Agent57: Outperforming the atari human benchmark,” in International Conference on International Conference on Machine Learning (ICML), 2020.

M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine Learning (ICML), 2019.

A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, “Big transfer (bit): General visual representation learning,” in European Conference on Computer Vision (ECCV), 2020.

Y. Li, M. Yang, and Z. Zhang, “A survey of multi-view representation learning,” Transactions on Knowledge and Data Engineering, vol. 31, no. 10, pp. 1863–1883, 2019.

M. A. Diniz and W. R. Schwartz, “Face attributes as cues for deep face recognition understanding,” in International Conference on Automatic Face and Gesture Recognition(FG), 2020.

X. Suau, L. Zappella, and N. Apostoloff, “Filter distillation for network compression,” in Winter Conference on Applications of Computer Vision (WACV), 2020.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Computer Vision and Pattern Recognition (CVPR), 2016.

J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit, “A constructive prediction of the generalization error across scales,” in International Conference on Learning Representations (ICLR), 2020.

K. Han, Y. Wang, Q. Zhang, W. Zhang, C. XU, and T. Zhang, “Model rubiks cube: Twisting resolution, depth and width for tinynets,” in Neural Information Processing Systems (NeurIPS), 2020.

E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in Conference of the Association for Computational Linguistics, 2019.

A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres, “Quantifying the carbon emissions of machine learning,” in Neural Information Processing Systems (NeurIPS), 2019.

R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Communications of the ACM, vol. 63, no. 12, pp. 54–63, 2020.

J.-H. Luo and J. Wu, “Neural network pruning with residual-connections and limited-data,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

L. Yang, Y. Han, X. Chen, S. Song, J. Dai, and G. Huang, “Resolution adaptive networks for efficient inference,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

V. Sindagi and V. M. Patel, “Multi-level bottom-top and top-bottom feature fusion for crowd counting,” in International Conference on Computer Vision (ICCV), 2019.

A. Sharma and D. W. Jacobs, “Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2011.

R. Hasegawa and K. Hotta, “Plsnet: A simple network using partial least squares regression for image classification,” in International Conference on Pattern Recognition (ICPR), 2016.

R. B. Kloss, A. Jord˜ao, and W. R. Schwartz, “Boosted projection: An ensemble of transformation models,” in Iberoamerican Congress on Pattern Recognition (CIARP), 2017.

X. Zeng and G. Li, “Incremental partial least squares analysis of big streaming data,” Pattern Recognition, vol. 47, pp. 3726–3735, 2014.

A. E. Stott, S. Kanna, D. P. Mandic, and W. T. Pike, “An online NIPALS algorithm for partial least squares,” in International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2017.

S. Alakkari and J. Dingliana, “An acceleration scheme for mini-batch, streaming PCA,” in British Machine Vision Conference (BMVC), 2019.

P. Geladi and B. Kowalski, “Partial least-squares regression: a tutorial,” Analytica Chimica Acta, vol. 185, pp. 1–17, 1986.

H. Abdi, “Partial least squares regression and projection on latent structure regression (pls regression),” Wiley Interdisciplinary Reviews: Computational Statistics, vol. 2, no. 1, pp. 97–106, 2010.

T. Mehmood, K. H. Liland, L. Snipen, and S. Saebo, “A review of variable selection methods in partial least squares regression,” Chemometrics and Intelligent Laboratory Systems, 2012.

G. Roffo, S. Melzi, and M. Cristani, “Infinite feature selection,” in International Conference on Computer Vision (ICCV), 2015.

G. Roffo, S. Melzi, U. Castellani, and A. Vinciarelli, “Infinite latent feature selection: A probabilistic latent graph-based ranking approach,” in International Conference on Computer Vision (ICCV), 2017.

G. Roffo, S. Melzi, U. Castellani, A. Vinciarelli, and M. Cristani, “Infinite feature selection: a graph-based feature filtering approach,” Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2020.

M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao, “Hrank: Filter pruning using high-rank feature map,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

C. M. J. Tan and M. Motani, “Dropnet: Reducing neural network complexity via iterative pruning,” in International Conference on International Conference on Machine Learning (ICML), 2020.

T. Elsken, J. H. Metzen, and F. Hutter, “Simple and efficient architecture search for convolutional neural networks,” in International Conference on Learning Representations (ICLR), 2018.

K. Kandasamy, W. Neiswanger, J. Schneider, B. Póczos, and E. P. Xing, “Neural architecture search with bayesian optimisation and optimal transport,” in Neural Information Processing Systems (NeurIPS), 2018.

H. Jin, Q. Song, and X. Hu, “Auto-keras: An efficient neural architecture search system,” in International Conference on Knowledge Discovery & Data Mining (SIGKDD), 2019.

B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural network architectures using reinforcement learning,” in International Conference on Learning Representations (ICLR), 2017.

E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,” in International Conference on International Conference on Machine Learning (ICML), 2017.

A. Brock, T. Lim, J. M. Ritchie, and N. Weston, “SMASH: oneshot model architecture search through hypernetworks,” in International Conference for Learning Representations(ICLR), 2018.

X. Dong and Y. Yang, “Searching for a robust neural architecture in four GPU hours,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

Z. Yang, Y. Wang, X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, and C. Xu, “CARS: continuous evolution for efficient neural architecture search,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

X. Chen, L. Xie, J. Wu, and Q. Tian, “Progressive differentiable architecture search: Bridging the depth gap between search and evaluation,” in International Conference on Computer Vision (ICCV), 2019.

Z. Li, T. Xi, J. Deng, G. Zhang, S. Wen, and R. He, “GP-NAS: gaussian process based neural architecture search,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

T. Kong, A. Yao, Y. Chen, and F. Sun, “Hypernet: Towards accurate region proposal generation and joint object detection,” in Computer Vision and Pattern Recognition (CVPR), 2016.

B. Hariharan, P. A. Arbeláez, R. B. Girshick, and J. Malik, “Hypercolumns for object segmentation and fine-grained localization,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2015.

J. Weng, Y. Zhang, and W. Hwang, “Candid covariance-free incremental principal component analysis,” Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 25, no. 8, pp. 1034–1040, 2003.

A. L. Andrew and R. C. E. Tan, “Computation of derivatives of repeated eigenvalues and the corresponding eigenvectors of symmetric matrix pencils,” SIAM Journal on Matrix Analysis and Applications, vol. 20, no. 1, pp. 78–100, 1998.

L. W. Mackey, “Deflation methods for sparse PCA,” in Neural Information Processing Systems (NIPS), D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, Eds., 2008.

R. Arora, P. Mianjy, and T. V. Marinov, “Stochastic optimization for multiview representation learning using partial least squares,” in International Conference on International Conference on Machine Learning (ICML), 2016.