Features transfer learning for image and video recognition tasks
Resumo
Feature transfer learning aims to reuse knowledge previously acquired in some source dataset to apply it in another target data and/or task. A requirement for the transfer of knowledge is the quality of feature spaces obtained, in which deep learning methods are widely applied since those provide discriminative and general descriptors. In this context, the main questions include: what to transfer; how to transfer; and when to transfer. Hence, we address these questions through distinct learning paradigms, transfer learning techniques, and several datasets and tasks. Therefore, our contributions are: an analysis of multiple descriptors contained in supervised deep networks; a new generalization metric that can be applied to any model and evaluation system; and a new architecture with a loss function for semi-supervised deep networks, in which all available data provide the learning.Referências
Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 35, no. 8, pp. 1798–1828, 2013.
J. Lu, V. Behbood, P. Hao, H. Zuo, S. Xue, and G. Zhang, "Transfer learning using computational intelligence: a survey," Knowledge-Based Systems, vol. 80, pp. 14–23, 2015.
L. Shao, F. Zhu, and X. Li, "Transfer learning for visual categorization: A survey," IEEE transactions on neural networks and learning systems, vol. 26, no. 5, pp. 1019–1034, 2015.
S. J. Pan, Q. Yang et al., "A survey on transfer learning," IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2010.
E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko, "Simultaneous deep transfer across domains and tasks," in Computer Vision (ICCV), 2015 IEEE International Conference on. IEEE, 2015, pp. 4068–4076.
J. Hu, J. Lu, and Y.-P. Tan, "Deep transfer metric learning," in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015, pp. 325–333.
B. Sengupta and K. J. Friston, "How robust are deep neural networks?" arXiv preprint arXiv:1804.11313, 2018.
L. Torrey and J. Shavlik, "Transfer learning," in Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques. IGI Global, 2010, pp. 242–264.
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?" in Advances in neural information processing systems, 2014, pp. 3320–3328.
A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "Cnn features off-the-shelf: an astounding baseline for recognition," in Pro- the IEEE conference on computer vision and pattern ceedings of recognition workshops, 2014, pp. 806–813.
M. Ponti, L. S. Ribeiro, T. S. Nazare, T. Bui, and J. Collomosse, "Everything you wanted to know about deep learning for computer vision but were afraid to ask," in 30th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T 2017), 2017, pp. 17–41.
S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, "Domain adaptation via transfer component analysis," IEEE Transactions on Neural Networks, vol. 22, no. 2, pp. 199–210, 2011.
R. F. Mello and M. A. Ponti, Machine Learning: A Practical Approach on the Statistical Learning Theory. Springer, 2018.
V. Pomponiu, H. Nejati, and N.-M. Cheung, "Deepmole: Deep neural networks for skin mole lesion classification," in Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2016, pp. 2623– 2627.
A. Mahbod, R. Ecker, and I. Ellinger, "Skin lesion classification using hybrid deep neural networks," arXiv preprint arXiv:1702.08434, 2017.
T. Majtner, S. Yildirim-Yayilgan, and J. Y. Hardeberg, "Combining deep learning and hand-crafted features for skin lesion classification," in Image Processing Theory Tools and Applications (IPTA), 2016 6th International Conference on. IEEE, 2016, pp. 1–6.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., "Imagenet large scale visual recognition challenge," International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
T. Mendonc¸a, P. M. Ferreira, J. S. Marques, A. R. Marcal, and J. Rozeira, "Ph 2-a dermoscopic image database for research and benchmarking," in 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, 2013, pp. 5437–5440.
F. P. dos Santos and M. A. Ponti, "Robust feature spaces from pre- trained deep network layers for skin lesion classification," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2018, pp. 189–196.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications," arXiv preprint arXiv:1704.04861, 2017.
K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
M. Ponti, T. S. Nazare, and G. S. Thum ´ e, "Image quantization as a ´ dimensionality reduction procedure in color and texture feature extraction," Neurocomputing, vol. 173, pp. 385–396, 2016.
L. Bi, J. Kim, E. Ahn, D. Feng, and M. Fulham, "Automatic melanoma detection via multi-scale lesion-biased representation and joint reverse classification," in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE, 2016, pp. 1055–1058.
F. P. dos Santos and M. A. Ponti, "Alignment of local and global features from multiple layers of convolutional neural network for image classification," in 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2019, pp. 241–248.
H. Mures¸an and M. Oltean, "Fruit recognition from images using deep learning," Acta Universitatis Sapientiae, Informatica, vol. 10, no. 1, pp. 26–42, 2018.
A. Rocha, D. C. Hauagge, J. Wainer, and S. Goldenstein, "Automatic produce classification from images using color, texture and appearance cues," in 2008 XXI Brazilian Symposium on Computer Graphics and Image Processing. IEEE, 2008, pp. 3–10.
K. Saenko, B. Kulis, M. Fritz, and T. Darrell, "Adapting visual category models to new domains," in European conference on computer vision. Springer, 2010, pp. 213–226.
P. Tschandl, C. Rosendahl, and H. Kittler, "The ham10000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions," arXiv preprint arXiv:1803.10417, 2018.
F. P. dos Santos, L. S. Ribeiro, and M. A. Ponti, "Generalization of feature embeddings transferred from different video anomaly detection domains," Journal of Visual Communication and Image Representation, vol. 60, pp. 407–416, 2019.
V. Mahadevan, W. Li, V. Bhalodia, and N. Vasconcelos, "Anomaly detection in crowded scenes," in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 1975–1981.
A. Zaharescu and R. Wildes, "Anomalous behaviour detection using spatiotemporal oriented energies, subset inclusion histogram comparison and event-driven processing," in European Conference on Computer Vision. Springer, 2010, pp. 563–576.
F. P. dos Santos, C. Zor, J. Kittler, and M. A. Ponti, "Learning image features with fewer labels using a semi-supervised deep convolutional network," Neural Networks, 2020.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
Y. Ren, K. Hu, X. Dai, L. Pan, S. C. Hoi, and Z. Xu, "Semi-supervised deep embedded clustering," Neurocomputing, vol. 325, pp. 121–130, 2019.
Y. Kuznietsov, J. Stuckler, and B. Leibe, "Semi-supervised deep learning ¨ for monocular depth map prediction," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017, pp. 2215–2223.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," in NIPS workshop on deep learning and unsupervised feature learning, vol. 2011, 2011, p. 5.
M. A. Ponti, G. B. P. da Costa, F. P. Santos, and K. U. Silveira, "Supervised and unsupervised relevance sampling in handcrafted and deep learning features obtained from image collections," Applied Soft Computing, vol. 80, pp. 414–424, 2019.
F. P. dos Santos and M. A. Ponti, "Homogeneity index as stopping criterion for anisotropic diffusion filter," in International Conference on Computer Analysis of Images and Patterns. Springer, 2019, pp. 269– 280.
J. Lu, V. Behbood, P. Hao, H. Zuo, S. Xue, and G. Zhang, "Transfer learning using computational intelligence: a survey," Knowledge-Based Systems, vol. 80, pp. 14–23, 2015.
L. Shao, F. Zhu, and X. Li, "Transfer learning for visual categorization: A survey," IEEE transactions on neural networks and learning systems, vol. 26, no. 5, pp. 1019–1034, 2015.
S. J. Pan, Q. Yang et al., "A survey on transfer learning," IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2010.
E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko, "Simultaneous deep transfer across domains and tasks," in Computer Vision (ICCV), 2015 IEEE International Conference on. IEEE, 2015, pp. 4068–4076.
J. Hu, J. Lu, and Y.-P. Tan, "Deep transfer metric learning," in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015, pp. 325–333.
B. Sengupta and K. J. Friston, "How robust are deep neural networks?" arXiv preprint arXiv:1804.11313, 2018.
L. Torrey and J. Shavlik, "Transfer learning," in Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques. IGI Global, 2010, pp. 242–264.
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?" in Advances in neural information processing systems, 2014, pp. 3320–3328.
A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "Cnn features off-the-shelf: an astounding baseline for recognition," in Pro- the IEEE conference on computer vision and pattern ceedings of recognition workshops, 2014, pp. 806–813.
M. Ponti, L. S. Ribeiro, T. S. Nazare, T. Bui, and J. Collomosse, "Everything you wanted to know about deep learning for computer vision but were afraid to ask," in 30th SIBGRAPI Conference on Graphics, Patterns and Images Tutorials (SIBGRAPI-T 2017), 2017, pp. 17–41.
S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, "Domain adaptation via transfer component analysis," IEEE Transactions on Neural Networks, vol. 22, no. 2, pp. 199–210, 2011.
R. F. Mello and M. A. Ponti, Machine Learning: A Practical Approach on the Statistical Learning Theory. Springer, 2018.
V. Pomponiu, H. Nejati, and N.-M. Cheung, "Deepmole: Deep neural networks for skin mole lesion classification," in Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2016, pp. 2623– 2627.
A. Mahbod, R. Ecker, and I. Ellinger, "Skin lesion classification using hybrid deep neural networks," arXiv preprint arXiv:1702.08434, 2017.
T. Majtner, S. Yildirim-Yayilgan, and J. Y. Hardeberg, "Combining deep learning and hand-crafted features for skin lesion classification," in Image Processing Theory Tools and Applications (IPTA), 2016 6th International Conference on. IEEE, 2016, pp. 1–6.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., "Imagenet large scale visual recognition challenge," International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
T. Mendonc¸a, P. M. Ferreira, J. S. Marques, A. R. Marcal, and J. Rozeira, "Ph 2-a dermoscopic image database for research and benchmarking," in 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, 2013, pp. 5437–5440.
F. P. dos Santos and M. A. Ponti, "Robust feature spaces from pre- trained deep network layers for skin lesion classification," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2018, pp. 189–196.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications," arXiv preprint arXiv:1704.04861, 2017.
K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
M. Ponti, T. S. Nazare, and G. S. Thum ´ e, "Image quantization as a ´ dimensionality reduction procedure in color and texture feature extraction," Neurocomputing, vol. 173, pp. 385–396, 2016.
L. Bi, J. Kim, E. Ahn, D. Feng, and M. Fulham, "Automatic melanoma detection via multi-scale lesion-biased representation and joint reverse classification," in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE, 2016, pp. 1055–1058.
F. P. dos Santos and M. A. Ponti, "Alignment of local and global features from multiple layers of convolutional neural network for image classification," in 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2019, pp. 241–248.
H. Mures¸an and M. Oltean, "Fruit recognition from images using deep learning," Acta Universitatis Sapientiae, Informatica, vol. 10, no. 1, pp. 26–42, 2018.
A. Rocha, D. C. Hauagge, J. Wainer, and S. Goldenstein, "Automatic produce classification from images using color, texture and appearance cues," in 2008 XXI Brazilian Symposium on Computer Graphics and Image Processing. IEEE, 2008, pp. 3–10.
K. Saenko, B. Kulis, M. Fritz, and T. Darrell, "Adapting visual category models to new domains," in European conference on computer vision. Springer, 2010, pp. 213–226.
P. Tschandl, C. Rosendahl, and H. Kittler, "The ham10000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions," arXiv preprint arXiv:1803.10417, 2018.
F. P. dos Santos, L. S. Ribeiro, and M. A. Ponti, "Generalization of feature embeddings transferred from different video anomaly detection domains," Journal of Visual Communication and Image Representation, vol. 60, pp. 407–416, 2019.
V. Mahadevan, W. Li, V. Bhalodia, and N. Vasconcelos, "Anomaly detection in crowded scenes," in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 1975–1981.
A. Zaharescu and R. Wildes, "Anomalous behaviour detection using spatiotemporal oriented energies, subset inclusion histogram comparison and event-driven processing," in European Conference on Computer Vision. Springer, 2010, pp. 563–576.
F. P. dos Santos, C. Zor, J. Kittler, and M. A. Ponti, "Learning image features with fewer labels using a semi-supervised deep convolutional network," Neural Networks, 2020.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
Y. Ren, K. Hu, X. Dai, L. Pan, S. C. Hoi, and Z. Xu, "Semi-supervised deep embedded clustering," Neurocomputing, vol. 325, pp. 121–130, 2019.
Y. Kuznietsov, J. Stuckler, and B. Leibe, "Semi-supervised deep learning ¨ for monocular depth map prediction," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017, pp. 2215–2223.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," in NIPS workshop on deep learning and unsupervised feature learning, vol. 2011, 2011, p. 5.
M. A. Ponti, G. B. P. da Costa, F. P. Santos, and K. U. Silveira, "Supervised and unsupervised relevance sampling in handcrafted and deep learning features obtained from image collections," Applied Soft Computing, vol. 80, pp. 414–424, 2019.
F. P. dos Santos and M. A. Ponti, "Homogeneity index as stopping criterion for anisotropic diffusion filter," in International Conference on Computer Analysis of Images and Patterns. Springer, 2019, pp. 269– 280.
Publicado
07/11/2020
Como Citar
DOS SANTOS, Fernando Pereira; PONTI, Moacir Antonelli.
Features transfer learning for image and video recognition tasks. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 33. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 29-35.
DOI: https://doi.org/10.5753/sibgrapi.est.2020.12980.
