Going Deep into Remote Sensing Spatial Feature Learning
Resumo
A lot of information may be extracted from the Earth’s surface through aerial images. This information may assist in myriad applications, such as urban planning, crop and forest management, disaster relief, etc. However, the process of distilling this information is strongly based on efficiently encoding the spatial features, a challenging task. Facing this, Deep Learning is able to learn specific data-driven features. This PhD thesis1 introduces deep learning into the remote sensing domain. Specifically, we tackled two main tasks, scene and pixel classification, using Deep Learning to encode spatial features over high-resolution remote sensing images. First, we proposed an architecture and analyze different strategies to exploit Convolutional Networks for image classification. Second, we introduced a network and proposed a new strategy to better exploit multi-context information in order to improve pixelwise classification. Finally, we proposed a new network based on morphological operations towards better learning of some relevant visual features.
Palavras-chave:
deep learning, machine learning, remote sensing, image classification, image segmentation.
Referências
M. Volpi and D. Tuia, "Dense semantic labeling of subdecimeter reso- lution images with convolutional neural networks," IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 2, pp. 881–893, 2017.
J. A. d. dos Santos, P.-H. Gosselin, S. Philipp-Foliguet, R. d. S. Torres, and A. X. Falao, "Multiscale classification of remote sensing images," IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 10, pp. 3764–3775, 2012.
K. Nogueira, W. R. Schwartz, and J. A. dos Santos, "Coffee crop recog- nition using multi-scale convolutional neural networks," in Iberoameri- can Congress on Pattern Recognition. Springer, 2015, pp. 67–74.
D. Fustes, D. Cantorna, C. Dafonte, B. Arcay, A. Iglesias, and M. Man- teiga, "A cloud-integrated web platform for marine monitoring using gis and remote sensing," Future Generation Computer Systems, vol. 34, pp. 155–160, 2014.
K. Nogueira, S. G. Fadel, ´I. C. Dourado, R. O. Werneck, J. A. V. Munoz, O. A. Penatti, R. T. Calumby, L. T. Li, J. A. dos Santos, and R. S. Torres, "Exploiting convnet diversity for flooding identification," IEEE Geoscience and Remote Sensing Letters, vol. 15, no. 9, pp. 1446–1450, 2018.
K. Nogueira, J. A. Dos Santos, T. Fornazari, T. S. F. Silva, L. P. Morel- lato, and R. d. S. Torres, "Towards vegetation species discrimination by using data-driven descriptors," in Pattern Recogniton in Remote Sensing (PRRS), 2016 9th IAPR Workshop on. IEEE, 2016, pp. 1–6.
K. Nogueira, J. A. dos Santos, L. Cancian, B. D. Borges, T. S. F. Silva, L. P. Morellato, and R. S. Torres, "Semantic segmentation of vegetation images acquired by unmanned aerial vehicles using an ensemble of con- vnets," IEEE International Geoscience & Remote Sensing Symposium, 2017.
K. Nogueira, J. A. dos Santos, N. Menini, T. S. F. Silva, L. P. C. Morel- lato, and R. S. Torres, "Spatio-temporal vegetation pixel classification by using convolutional networks," IEEE Geoscience and Remote Sensing Letters, 2019.
J. R. Jensen and K. Lulla, "Introductory digital image processing: a remote sensing perspective," 1987.
G. Kumar and P. K. Bhatia, "A detailed review of feature extraction in image processing systems," in Advanced Computing & Communication Technologies. IEEE, 2014, pp. 5–12.
D. G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.
R. de O. Stehling, M. A. Nascimento, and A. X. Falcao, "A compact and efficient image retrieval approach based on border/interior pixel classi- fication," in International Conference on Information and Knowledge Management, 2002, pp. 102–109.
F. Hu, G.-S. Xia, J. Hu, Y. Zhong, and K. Xu, "Fast binary coding for the scene classification of high-resolution remote sensing imagery," Remote Sensing, vol. 8, no. 7, p. 555, 2016.
C.-h. Chen, L.-F. Pau, and P. S.-p. Wang, Handbook of pattern recog- nition and computer vision. World Scientific, 2010, vol. 27.
J. A. dos Santos, O. A. B. Penatti, and R. da Silva Torres, "Evaluating the potential of texture and color descriptors for remote sensing image retrieval and classification." in International Conference on Computer Vision Theory and Applications, 2010, pp. 203–208.
J. E. Ball, D. T. Anderson, and C. S. Chan, "Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community," Journal of Applied Remote Sensing, vol. 11, no. 4, p. 042609, 2017.
J. dos Santos, O. Penatti, P. Gosselin, A. Falcao, S. Philipp-Foliguet, and R. Torres, "Efficient and effective hierarchical feature propagation," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. PP, no. 99, pp. 1–12, 2014.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016, http://www.deeplearningbook.org.
Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436–444, 2015.
J. Sivic and A. Zisserman, "Video google: a text retrieval approach to object matching in videos," in International Conference on Computer Vision, vol. 2, 2003, pp. 1470–1477.
K. Nogueira, "Going deep into remote sensing spatial feature learning," 2019.
K. Nogueira, W. O. Miranda, and J. A. Dos Santos, "Improving spatial feature representation from aerial scenes by using convolutional net- works," in Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2015, pp. 289–296.
K. Nogueira, O. A. Penatti, and J. A. dos Santos, "Towards better exploiting convolutional neural networks for remote sensing scene classification," Pattern Recognition, vol. 61, pp. 539–556, 2017.
K. Nogueira, M. Dalla Mura, J. Chanussot, W. R. Schwartz, and J. A. dos Santos, "Learning to semantically segment high-resolution remote sensing images," in International Conference on Pattern Recognition. IEEE, 2016, pp. 3566–3571.
——, "Dynamic multicontext segmentation of remote sensing images based on convolutional networks," IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 10, pp. 7503–7520, 2019.
K. Nogueira, J. Chanussot, M. D. Mura, W. R. Schwartz, and J. A. d. Santos, "An introduction to deep morphological networks," arXiv preprint arXiv:1906.01751, 2019.
T. M. Santana, K. Nogueira, A. M. Machado, and J. A. dos Santos, "Deep contextual description of superpixels for aerial urban scenes classification," in IEEE International Geoscience & Remote Sensing Symposium, July 2017, pp. 3027–3031.
R. Baeta, K. Nogueira, D. Menotti, and J. A. dos Santos, "Learning deep features on multiple scales for coffee crop recognition," in Conference on Graphics, Patterns and Images (SIBGRAPI), 2017, pp. 262–268.
K. Nogueira, S. G. Fadel, I. C. Dourado, R. d. O. Werneck, J. A. V. Munoz, O. A. B. Penatti, R. T. Calumby, L. T. Li, J. A. dos Santos, and R. d. S. Torres, "Data-driven flood detection using neural networks," in Working Notes Proc. MediaEval Workshop, 2017, p. 2. [Online]. Available: http://slim-sig.irisa.fr/me17/Mediaeval 2017 paper 39.pdf
J. A. V. Múnoz, L. T. Li, I. C. Dourado, K. Nogueira, S. G. Fadel, O. A. B. Penatti, J. Almeida, L. A. M. Pereira, R. T. Calumby, J. A. dos Santos, and R. d. S. Torres, "A ranking fusion approach for geographic-location prediction of multimedia objects," in Working Notes Proc. MediaEval Workshop, 2016, p. 2. [Online]. Available: http://ceur-ws.org/Vol-1739/MediaEval 2016 paper 22.pdf
O. A. Penatti, K. Nogueira, and J. A. dos Santos, "Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?" in Conference on Computer Vision and Pattern Recognition Workshop, 2015, pp. 44–51.
L. T. Li, J. A. V. Múnoz, J. Almeida, R. T. Calumby, O. A. B. Penatti, I. C. Dourado, K. Nogueira, P. R. Mendes Júnior, A. M. L. Pereira, D. C. G. Pedronette, M. A. Gonçalves, J. A. dos Santos, and R. d. S. Torres, "Recod @ placing task of mediaeval 2015," in Working Notes Proc. MediaEval Workshop, 2015, p. 2. [Online]. Available: http://ceur-ws.org/Vol-1436/Paper49.pdf
F. Lateef and Y. Ruichek, "Survey on semantic segmentation using deep learning techniques," Neurocomputing, vol. 338, pp. 321–348, 2019.
R. Szeliski, Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
J. Serra and P. Soille, Mathematical morphology and its applications to image processing. Springer Science & Business Media, 2012, vol. 2.
Y. Seo, B. Park, S.-C. Yoon, K. C. Lawrence, and G. R. Gamble, "Morphological image analysis for foodborne bacteria classification," Transactions of the American Society of Agricultural and Biological Engineers, vol. 61, pp. 5–13, 2018.
F. Chollet, "Xception: Deep learning with depthwise separable convo- lutions," in Conference on Computer Vision and Pattern Recognition, 2017, pp. 1800–1807.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Neural Information Pro- cessing Systems, 2012, pp. 1106–1114.
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Conference on Computer Vision and Pattern Recognition, June 2017, pp. 2261–2269.
J. A. d. dos Santos, P.-H. Gosselin, S. Philipp-Foliguet, R. d. S. Torres, and A. X. Falao, "Multiscale classification of remote sensing images," IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 10, pp. 3764–3775, 2012.
K. Nogueira, W. R. Schwartz, and J. A. dos Santos, "Coffee crop recog- nition using multi-scale convolutional neural networks," in Iberoameri- can Congress on Pattern Recognition. Springer, 2015, pp. 67–74.
D. Fustes, D. Cantorna, C. Dafonte, B. Arcay, A. Iglesias, and M. Man- teiga, "A cloud-integrated web platform for marine monitoring using gis and remote sensing," Future Generation Computer Systems, vol. 34, pp. 155–160, 2014.
K. Nogueira, S. G. Fadel, ´I. C. Dourado, R. O. Werneck, J. A. V. Munoz, O. A. Penatti, R. T. Calumby, L. T. Li, J. A. dos Santos, and R. S. Torres, "Exploiting convnet diversity for flooding identification," IEEE Geoscience and Remote Sensing Letters, vol. 15, no. 9, pp. 1446–1450, 2018.
K. Nogueira, J. A. Dos Santos, T. Fornazari, T. S. F. Silva, L. P. Morel- lato, and R. d. S. Torres, "Towards vegetation species discrimination by using data-driven descriptors," in Pattern Recogniton in Remote Sensing (PRRS), 2016 9th IAPR Workshop on. IEEE, 2016, pp. 1–6.
K. Nogueira, J. A. dos Santos, L. Cancian, B. D. Borges, T. S. F. Silva, L. P. Morellato, and R. S. Torres, "Semantic segmentation of vegetation images acquired by unmanned aerial vehicles using an ensemble of con- vnets," IEEE International Geoscience & Remote Sensing Symposium, 2017.
K. Nogueira, J. A. dos Santos, N. Menini, T. S. F. Silva, L. P. C. Morel- lato, and R. S. Torres, "Spatio-temporal vegetation pixel classification by using convolutional networks," IEEE Geoscience and Remote Sensing Letters, 2019.
J. R. Jensen and K. Lulla, "Introductory digital image processing: a remote sensing perspective," 1987.
G. Kumar and P. K. Bhatia, "A detailed review of feature extraction in image processing systems," in Advanced Computing & Communication Technologies. IEEE, 2014, pp. 5–12.
D. G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.
R. de O. Stehling, M. A. Nascimento, and A. X. Falcao, "A compact and efficient image retrieval approach based on border/interior pixel classi- fication," in International Conference on Information and Knowledge Management, 2002, pp. 102–109.
F. Hu, G.-S. Xia, J. Hu, Y. Zhong, and K. Xu, "Fast binary coding for the scene classification of high-resolution remote sensing imagery," Remote Sensing, vol. 8, no. 7, p. 555, 2016.
C.-h. Chen, L.-F. Pau, and P. S.-p. Wang, Handbook of pattern recog- nition and computer vision. World Scientific, 2010, vol. 27.
J. A. dos Santos, O. A. B. Penatti, and R. da Silva Torres, "Evaluating the potential of texture and color descriptors for remote sensing image retrieval and classification." in International Conference on Computer Vision Theory and Applications, 2010, pp. 203–208.
J. E. Ball, D. T. Anderson, and C. S. Chan, "Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community," Journal of Applied Remote Sensing, vol. 11, no. 4, p. 042609, 2017.
J. dos Santos, O. Penatti, P. Gosselin, A. Falcao, S. Philipp-Foliguet, and R. Torres, "Efficient and effective hierarchical feature propagation," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. PP, no. 99, pp. 1–12, 2014.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016, http://www.deeplearningbook.org.
Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436–444, 2015.
J. Sivic and A. Zisserman, "Video google: a text retrieval approach to object matching in videos," in International Conference on Computer Vision, vol. 2, 2003, pp. 1470–1477.
K. Nogueira, "Going deep into remote sensing spatial feature learning," 2019.
K. Nogueira, W. O. Miranda, and J. A. Dos Santos, "Improving spatial feature representation from aerial scenes by using convolutional net- works," in Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2015, pp. 289–296.
K. Nogueira, O. A. Penatti, and J. A. dos Santos, "Towards better exploiting convolutional neural networks for remote sensing scene classification," Pattern Recognition, vol. 61, pp. 539–556, 2017.
K. Nogueira, M. Dalla Mura, J. Chanussot, W. R. Schwartz, and J. A. dos Santos, "Learning to semantically segment high-resolution remote sensing images," in International Conference on Pattern Recognition. IEEE, 2016, pp. 3566–3571.
——, "Dynamic multicontext segmentation of remote sensing images based on convolutional networks," IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 10, pp. 7503–7520, 2019.
K. Nogueira, J. Chanussot, M. D. Mura, W. R. Schwartz, and J. A. d. Santos, "An introduction to deep morphological networks," arXiv preprint arXiv:1906.01751, 2019.
T. M. Santana, K. Nogueira, A. M. Machado, and J. A. dos Santos, "Deep contextual description of superpixels for aerial urban scenes classification," in IEEE International Geoscience & Remote Sensing Symposium, July 2017, pp. 3027–3031.
R. Baeta, K. Nogueira, D. Menotti, and J. A. dos Santos, "Learning deep features on multiple scales for coffee crop recognition," in Conference on Graphics, Patterns and Images (SIBGRAPI), 2017, pp. 262–268.
K. Nogueira, S. G. Fadel, I. C. Dourado, R. d. O. Werneck, J. A. V. Munoz, O. A. B. Penatti, R. T. Calumby, L. T. Li, J. A. dos Santos, and R. d. S. Torres, "Data-driven flood detection using neural networks," in Working Notes Proc. MediaEval Workshop, 2017, p. 2. [Online]. Available: http://slim-sig.irisa.fr/me17/Mediaeval 2017 paper 39.pdf
J. A. V. Múnoz, L. T. Li, I. C. Dourado, K. Nogueira, S. G. Fadel, O. A. B. Penatti, J. Almeida, L. A. M. Pereira, R. T. Calumby, J. A. dos Santos, and R. d. S. Torres, "A ranking fusion approach for geographic-location prediction of multimedia objects," in Working Notes Proc. MediaEval Workshop, 2016, p. 2. [Online]. Available: http://ceur-ws.org/Vol-1739/MediaEval 2016 paper 22.pdf
O. A. Penatti, K. Nogueira, and J. A. dos Santos, "Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?" in Conference on Computer Vision and Pattern Recognition Workshop, 2015, pp. 44–51.
L. T. Li, J. A. V. Múnoz, J. Almeida, R. T. Calumby, O. A. B. Penatti, I. C. Dourado, K. Nogueira, P. R. Mendes Júnior, A. M. L. Pereira, D. C. G. Pedronette, M. A. Gonçalves, J. A. dos Santos, and R. d. S. Torres, "Recod @ placing task of mediaeval 2015," in Working Notes Proc. MediaEval Workshop, 2015, p. 2. [Online]. Available: http://ceur-ws.org/Vol-1436/Paper49.pdf
F. Lateef and Y. Ruichek, "Survey on semantic segmentation using deep learning techniques," Neurocomputing, vol. 338, pp. 321–348, 2019.
R. Szeliski, Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
J. Serra and P. Soille, Mathematical morphology and its applications to image processing. Springer Science & Business Media, 2012, vol. 2.
Y. Seo, B. Park, S.-C. Yoon, K. C. Lawrence, and G. R. Gamble, "Morphological image analysis for foodborne bacteria classification," Transactions of the American Society of Agricultural and Biological Engineers, vol. 61, pp. 5–13, 2018.
F. Chollet, "Xception: Deep learning with depthwise separable convo- lutions," in Conference on Computer Vision and Pattern Recognition, 2017, pp. 1800–1807.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Neural Information Pro- cessing Systems, 2012, pp. 1106–1114.
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Conference on Computer Vision and Pattern Recognition, June 2017, pp. 2261–2269.
Publicado
07/11/2020
Como Citar
NOGUEIRA, Keiller; SCHWARTZ, William Robson; DOS SANTOS, Jefersson Alex.
Going Deep into Remote Sensing Spatial Feature Learning. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 33. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 98-104.
DOI: https://doi.org/10.5753/sibgrapi.est.2020.12990.
