Open-set semantic segmentation for remote sensing images

• Ian Nunes IBGE / PUC-Rio
• Hugo Oliveira UFV
• Marcus Poggi PUC-Rio

DOI: https://doi.org/10.5753/sibgrapi.est.2024.31640

Resumo

Collecting samples that exhaust all possible classes for real-world tasks is usually difficult or impossible due to many different factors. In a realistic/feasible scenario, methods should be aware that the training data is incomplete and that not all knowledge is available. Therefore all developed methods should be able to identify the unknown samples while correctly executing the proposed task to the known classes in the tests phase. Open-Set Recognition and Semantic Segmentation models emerge to handle this kind of scenario for, respectively, visual recognition and dense labeling tasks. Initially, this work proposes a novel taxonomy aiming to organize the literature and provide an understanding of the theoretical trends that guided the existing approaches that may influence future methods. This work also proposes two distinct techniques to perform open-set semantic segmentation. First, a method called Open Gaussian Mixture of Models (OpenGMM) extends the Open Principal Component Scoring (OpenPCS) framework using a Gaussian Mixture of Models to model the distribution of pixels for each class in a multimodal manner. Second, the Conditional Reconstruction for Open-set Semantic Segmentation (CoReSeg) method tackles the issue using class-conditioned reconstruction of the input images according to their pixel-wise mask. The third proposed approach is a general post-processing procedure that uses superpixels to enforce highly homogeneous regions to behave equally, rectifying erroneously classified pixels within these regions. We also proposed a novel superpixel generation method called Fusing Superpixels for Semantic Consistency (FuSC). All proposed approaches produce better semantic consistency and outperformed state-of-the-art baseline methods on Vaihingen and Potsdam ISPRS dataset. The official implementation of all proposed approaches is available at https://github.com/iannunes.

Referências

C. Geng, S.-j. Huang, and S. Chen, “Recent advances in open set recognition: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 10, pp. 3614–3631, 2020.

A. Bendale and T. E. Boult, “Towards open set deep networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1563–1572.

X. Sun, Z. Yang, C. Zhang, K.-V. Ling, and G. Peng, “Conditional gaussian distribution learning for open set recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13 480–13 489.

P. Oza and V. M. Patel, “C2ae: Class conditioned auto-encoder for open-set recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2307–2316.

Z. Cui, W. Longshi, and R. Wang, “Open set semantic segmentation with statistical test and adaptive threshold,” in 2020 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2020, pp. 1–6.

Y. Guo, G. Camporese, W. Yang, A. Sperduti, and L. Ballan, “Conditional variational capsule network for open set recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 103–111.

H. Oliveira, C. Silva, G. L. Machado, K. Nogueira, and J. A. dos Santos, “Fully convolutional open set segmentation,” Machine Learning, pp. 1–52, 2021.

A. Brilhador, M. Gutoski, A. E. Lazzaretti, and H. S. . Lopes, “A comparative study for open set semantic segmentation methods,” in Anais do 15 Congresso Brasileiro de Inteligência Computacional, C. J. A. B. Filho, H. V. Siqueira, D. D. Ferreira, D. W. Bertol, and R. C. L. ao de Oliveira, Eds. Joinville, SC: SBIC, 2021, pp. 1–8.

I. Nunes, H. Oliveira, M. B. Pereira, J. A. d. Santos, and M. Poggi, “Deep open-set segmentation in visual learning,” in Proceedings... Conference on Graphics, Patterns and Images, 35. (SIBGRAPI), 2022. [Online]. Available: [link]

I. Nunes, C. Laranjeira, H. Oliveira, and J. A. dos Santos, “A systematic review on open-set segmentation,” Computers & Graphics, 2023. [Online]. Available: [link]

I. Nunes, M. B. Pereira, H. Oliveira, and J. A. Dos Santos, “Fusc: Fusing superpixels for improved semantic consistency,” IEEE Access, 2024.

I. Nunes, M. B. Pereira, H. Oliveira, J. A. dos Santos, and M. Poggi, “Conditional reconstruction for open-set semantic segmentation,” in 2022 IEEE International Conference on Image Processing (ICIP), 2022, pp. 946–950.

Z. Ge, S. Demyanov, Z. Chen, and R. Garnavi, “Generative openmax for multi-class open set classification,” arXiv preprint arXiv:1707.07418, 2017.

D. Hendrycks, M. Mazeika, and T. Dietterich, “Deep anomaly detection with outlier exposure,” arXiv preprint arXiv:1812.04606, 2018.

C. C. da Silva, K. Nogueira, H. N. Oliveira, and J. A. dos Santos, “Towards open-set semantic segmentation of aerial images,” in 2020 IEEE Latin American GRSS & ISPRS Remote Sensing Conference (LAGIRS). IEEE, 2020, pp. 16–21.

M. Vendramini, H. Oliveira, A. Machado, and J. A. dos Santos, “Opening Deep Neural Networks With Generative Models,” in ICIP. IEEE, 2021, pp. 1314–1318.

J. A. C. Martinez, H. Oliveira, J. A. dos Santos, and R. Q. Feitosa, “Open set semantic segmentation for multitemporal crop recognition,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2021.

S. Yan, J. Zhou, J. Xie, S. Zhang, and X. He, “An em framework for online incremental learning of semantic segmentation,” in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 3052–3060.

J. Cen, P. Yun, J. Cai, M. Y. Wang, and M. Liu, “Deep metric learning for open world semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 15 333–15 342.

M. Grcić, P. Bevandić, and S. Šegvić, “Dense anomaly detection by robust learning on synthetic negative data,” arXiv preprint arXiv:2112.12833, 2021.

R. Chan, M. Rottmann, and H. Gottschalk, “Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation,” in Proceedings of the ieee/cvf international conference on computer vision, 2021, pp. 5128–5137.

J. Gawlikowski, S. Saha, A. Kruspe, and X. X. Zhu, “An advanced dirichlet prior network for out-of-distribution detection in remote sensing,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–19, 2022.

J. Hong, W. Li, J. Han, J. Zheng, P. Fang, M. Harandi, and L. Petersson, “Goss: Towards generalized open-set semantic segmentation,” arXiv preprint arXiv:2203.12116, 2022.

H. Dong, Z. Chen, M. Yuan, Y. Xie, J. Zhao, F. Yu, B. Dong, and L. Zhang, “Region-aware metric learning for open world semantic segmentation via meta-channel aggregation,” arXiv preprint arXiv:2205.08083, 2022.

R. Yoshihashi, W. Shao, R. Kawakami, S. You, M. Iida, and T. Naemura, “Classification-reconstruction learning for open-set recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4016–4025.

M. Grcić, P. Bevandić, and S. Šegvić, “Dense open-set recognition with synthetic outliers generated by real nvp,” arXiv preprint arXiv:2011.11094, 2020.

S. Kong and D. Ramanan, “Opengan: Open-set recognition via open data generation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 813–822.

P. Bevandić, I. Krešo, M. Oršić, and S. Šegvić, “Dense open-set recognition based on training with noisy negative images,” Image and Vision Computing, p. 104490, 2022.

M. Grcić, P. Bevandić, and S. Šegvić, “Densehybrid: Hybrid anomaly detection for dense open-set recognition,” arXiv preprint arXiv:2207.02606, 2022.

R. Bharadwaj, G. Jaswal, A. Nigam, and K. Tiwari, “Mobile based human identification using forehead creases: Application and assessment under covid-19 masked face scenarios,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 3693–3701.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.

S. Zagoruyko and N. Komodakis, “Wide residual networks,” in British Machine Vision Conference 2016. British Machine Vision Association, 2016.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.

E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018.

J. Ji, X. Lu, M. Luo, M. Yin, Q. Miao, and X. Liu, “Parallel fully convolutional network for semantic segmentation,” IEEE Access, vol. 9, pp. 673–682, 2020.

L. Melas-Kyriazi and A. K. Manrai, “Pixmatch: Unsupervised domain adaptation via pixelwise consistency training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12 435–12 445.

J. Kang, Z. Wang, R. Zhu, X. Sun, R. Fernandez-Beltran, and A. Plaza, “Picoco: Pixelwise contrast and consistency learning for semisupervised building footprint segmentation,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 10 548–10 559, 2021.

R. Ratajczak, C. Crispim, B. Fervers, E. Faure, and L. Tougne, “Semantic segmentation post-processing with colorized pairwise potentials and deep edges,” in 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA). IEEE, 2020, pp. 1–6.

H. Zhang, K. Jiang, Y. Zhang, Q. Li, C. Xia, and X. Chen, “Discriminative feature learning for video semantic segmentation,” in 2014 International Conference on Virtual Reality and Visualization. IEEE, 2014, pp. 321–326.