A survey on the state-of-the-art superpixel segmentation
Resumo
In contrast to the rapid progress of superpixel segmentation, their methods are often compared only with classical approaches. Also, the previous superpixel methods categorizations are insufficient to cover the recent literature. In addition, although the inner color similarity usually underlies superpixel methods, both color homogeneity measures have important drawbacks. In this work, we fill these gaps by providing a new taxonomy for superpixel segmentation, a new color homogeneity measure, and an extensive comparison among 20 superpixel methods. Experiments show that the proposed measure, named Similarity between Image and Reconstruction from Superpixels (SIRS), is more robust to slight color variations than Explained Variation. Using SIRS and the commonly used superpixel metrics, we evaluated 20 superpixel segmentation methods and provided insights into the different approaches based on the clustering categories in our taxonomy.Referências
Y. Liang, Y. Zhang, Y. Wu, S. Tu, and C. Liu, “Robust video object segmentation via propagating seams and matching superpixels,” IEEE Access, vol. 8, pp. 53 766–53 776, 2020.
I. Borlido Barcelos, F. Belém, P. Miranda, A. X. Falcão, Z. K. G. do Patrocínio, and S. J. F. Guimarães, “Towards interactive image segmentation by dynamic and iterative spanning forest,” in Discrete Geometry and Mathematical Morphology, J. Lindblad, F. Malmberg, and N. Sladoje, Eds. Cham: Springer International Publishing, 2021, pp. 351–364.
W. Zhao, Y. Fu, X. Wei, and H. Wang, “An improved image semantic segmentation method based on superpixels and conditional random fields,” Applied Sciences, vol. 8, no. 5, p. 837, 2018.
L. Ren, L. Zhao, and Y. Wang, “A superpixel-based dual window rx for hyperspectral anomaly detection,” IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 7, pp. 1233–1237, 2019.
J. Zhang, J. Chen, Q. Wang, and S. Chen, “Spatiotemporal saliency detection based on maximum consistency superpixels merging for video analysis,” IEEE Transactions on Industrial Informatics, vol. 16, no. 1, pp. 606–614, 2019.
X. Zhou, Y. Wang, Q. Zhu, C. Xiao, and X. Lu, “Ssg: superpixel segmentation and grabcut-based salient object segmentation,” The Visual Computer, vol. 35, no. 3, pp. 385–398, 2019.
P. Sellars, A. I. Aviles-Rivero, and C.-B. Schönlieb, “Superpixel contracted graph-based learning for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 6, pp. 4180–4193, 2020.
D. Stutz, A. Hermans, and B. Leibe, “Superpixels: An evaluation of the state-of-the-art,” Computer Vision and Image Understanding, vol. 166, pp. 1–27, 2018.
M. Wang, X. Liu, Y. Gao, X. Ma, and N. Q. Soomro, “Superpixel segmentation: A benchmark,” Signal Processing: Image Communication, vol. 56, pp. 28–39, 2017.
A. Schick, M. Fischer, and R. Stiefelhagen, “Measuring and evaluating the compactness of superpixels,” in Proceedings of the 21st international conference on pattern recognition (ICPR2012). IEEE, 2012, pp. 930–934.
——, “An evaluation of the compactness of superpixels,” Pattern Recognition Letters, vol. 43, pp. 71–80, 2014.
W. Benesova and M. Kottman, “Fast superpixel segmentation using morphological processing,” in Conference on Machine Vision and Machine Learning, 2014, pp. 67–1.
A. P. Moore, S. J. Prince, J. Warrell, U. Mohammed, and G. Jones, “Superpixel lattices,” in 2008 IEEE conference on computer vision and pattern recognition. IEEE, 2008, pp. 1–8.
P. Neubert and P. Protzel, “Superpixel benchmark and comparison,” in Proc. Forum Bildverarbeitung, vol. 6, 2012, pp. 1–12.
D. Stutz, “Superpixel segmentation: An evaluation,” in German conference on pattern recognition. Springer, 2015, pp. 555–562.
B. Mathieu, A. Crouzil, and J. B. Puel, “Oversegmentation methods: a new evaluation,” in Iberian Conference on Pattern Recognition and Image Analysis. Springer, 2017, pp. 185–193.
L. A. C. Mansilla and P. A. V. Miranda, “Oriented image foresting transform segmentation: Connectivity constraints with adjustable width,” in 2016 29th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 2016, pp. 289–296.
E. B. Alexandre, A. S. Chowdhury, A. X. Falcao, and P. A. V. Miranda, “Ift-slic: A general framework for superpixel generation based on simple linear iterative clustering and image foresting transform,” in 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, 2015, pp. 337–344.
J. Shi, Q. Yan, L. Xu, and J. Jia, “Hierarchical image saliency detection on extended cssd,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 4, pp. 717–729, 2015.
F. C. Belém, S. J. F. Guimarães, and A. X. Falcão, “Superpixel segmentation using dynamic and iterative spanning forest,” IEEE Signal Processing Letters, vol. 27, pp. 1440–1444, 2020.
X. Wei, Q. Yang, Y. Gong, N. Ahuja, and M.-H. Yang, “Superpixel hierarchy,” IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 4838–4849, 2018.
S. Bobbia, R. Macwan, Y. Benezeth, K. Nakamura, R. Gomez, and J. Dubois, “Iterative boundaries implicit identification for superpixels segmentation: a real-time approach,” IEEE Access, vol. 9, pp. 77 250–77 263, 2021.
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2274–2282, 2012.
D. Chai, “Rooted spanning superpixels,” International Journal of Computer Vision, vol. 128, no. 12, pp. 2962–2978, 2020.
F. C. Belém, B. Perret, J. Cousty, S. J. Guimarães, and A. X. Falcão, “Towards a simple and efficient object-based superpixel delineation framework,” in 2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2021, pp. 346–353.
X. Kang, L. Zhu, and A. Ming, “Dynamic random walk for superpixel segmentation,” IEEE Transactions on Image Processing, vol. 29, pp. 3871–3884, 2020.
H. Peng, A. I. Aviles-Rivero, and C.-B. Schönlieb, “Hers superpixels: Deep affinity learning for hierarchical entropy rate segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 217–226.
L. Zhu, Q. She, B. Zhang, Y. Lu, Z. Lu, D. Li, and J. Hu, “Learning the superpixel in a non-iterative and lifelong manner,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1225–1234.
J. E. Vargas-Mu˜noz, A. S. Chowdhury, E. B. Alexandre, F. L. Galvão, P. A. V. Miranda, and A. X. Falcão, “An iterative spanning forest framework for superpixel segmentation,” IEEE Transactions on Image Processing, vol. 28, no. 7, pp. 3477–3489, 2019.
R. Achanta and S. Susstrunk, “Superpixels and polygons using simple non-iterative clustering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7 2017.
Z. Ban, J. Liu, and L. Cao, “Superpixel segmentation using gaussian mixture model,” IEEE Transactions on Image Processing, vol. 27, no. 8, pp. 4105–4117, 2018.
Z. Li and J. Chen, “Superpixel segmentation using linear spectral clustering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6 2015.
R. Giraud, V.-T. Ta, and N. Papadakis, “Robust superpixels using color and contour features along linear path,” Computer Vision and Image Understanding, vol. 170, pp. 1–13, 2018.
M. Van den Bergh, X. Boix, G. Roig, B. de Capitani, and L. Van Gool, “Seeds: Superpixels extracted via energy-driven sampling,” in Computer Vision – ECCV 2012, A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 13–26.
M.-Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, “Entropy rate superpixel segmentation,” in CVPR 2011, 2011, pp. 2097–2104.
J. Yao, M. Boben, S. Fidler, and R. Urtasun, “Real-time coarse-tofine topologically preserving segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2947–2955.
C. Conrad, M. Mertz, and R. Mester, “Contour-relaxed superpixels,” in Energy Minimization Methods in Computer Vision and Pattern Recognition. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 280–293.
P. Buyssens, I. Gardin, and S. Ruan, “Eikonal based region growing for superpixels generation: Application to semi-supervised real time organ segmentation in ct images,” IRBM, vol. 35, no. 1, pp. 20–26, 2014, biomedical image segmentation using variational and statistical approaches. [Online]. Available: [link].
D. R. Martin, C. C. Fowlkes, and J. Malik, “Learning to detect natural image boundaries using local brightness, color, and texture cues,” IEEE transactions on pattern analysis and machine intelligence, vol. 26, no. 5, pp. 530–549, 2004.
I. B. Barcelos, F. D. C. Belém, L. D. M. João, A. X. Falcão, and G. S. JF, “Improving color homogeneity measure in superpixel segmentation assessment,” in 2022 35th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), vol. 1. IEEE, 2022, pp. 79–84.
F. Belém, I. Borlido, L. João, B. Perret, J. Cousty, S. J. F. Guimarães, and A. Falcão, “Fast and effective superpixel segmentation using accurate saliency estimation,” in Discrete Geometry and Mathematical Morphology, ´ E. Baudrier, B. Naegel, A. Krähenbühl, and M. Tajine, Eds. Cham: Springer International Publishing, 2022, pp. 261–273.
I. Borlido Barcelos, F. Belém, P. Miranda, A. X. Falcão, Z. K. G. do Patrocínio, and S. J. F. Guimarães, “Towards interactive image segmentation by dynamic and iterative spanning forest,” in Discrete Geometry and Mathematical Morphology, J. Lindblad, F. Malmberg, and N. Sladoje, Eds. Cham: Springer International Publishing, 2021, pp. 351–364.
W. Zhao, Y. Fu, X. Wei, and H. Wang, “An improved image semantic segmentation method based on superpixels and conditional random fields,” Applied Sciences, vol. 8, no. 5, p. 837, 2018.
L. Ren, L. Zhao, and Y. Wang, “A superpixel-based dual window rx for hyperspectral anomaly detection,” IEEE Geoscience and Remote Sensing Letters, vol. 17, no. 7, pp. 1233–1237, 2019.
J. Zhang, J. Chen, Q. Wang, and S. Chen, “Spatiotemporal saliency detection based on maximum consistency superpixels merging for video analysis,” IEEE Transactions on Industrial Informatics, vol. 16, no. 1, pp. 606–614, 2019.
X. Zhou, Y. Wang, Q. Zhu, C. Xiao, and X. Lu, “Ssg: superpixel segmentation and grabcut-based salient object segmentation,” The Visual Computer, vol. 35, no. 3, pp. 385–398, 2019.
P. Sellars, A. I. Aviles-Rivero, and C.-B. Schönlieb, “Superpixel contracted graph-based learning for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 6, pp. 4180–4193, 2020.
D. Stutz, A. Hermans, and B. Leibe, “Superpixels: An evaluation of the state-of-the-art,” Computer Vision and Image Understanding, vol. 166, pp. 1–27, 2018.
M. Wang, X. Liu, Y. Gao, X. Ma, and N. Q. Soomro, “Superpixel segmentation: A benchmark,” Signal Processing: Image Communication, vol. 56, pp. 28–39, 2017.
A. Schick, M. Fischer, and R. Stiefelhagen, “Measuring and evaluating the compactness of superpixels,” in Proceedings of the 21st international conference on pattern recognition (ICPR2012). IEEE, 2012, pp. 930–934.
——, “An evaluation of the compactness of superpixels,” Pattern Recognition Letters, vol. 43, pp. 71–80, 2014.
W. Benesova and M. Kottman, “Fast superpixel segmentation using morphological processing,” in Conference on Machine Vision and Machine Learning, 2014, pp. 67–1.
A. P. Moore, S. J. Prince, J. Warrell, U. Mohammed, and G. Jones, “Superpixel lattices,” in 2008 IEEE conference on computer vision and pattern recognition. IEEE, 2008, pp. 1–8.
P. Neubert and P. Protzel, “Superpixel benchmark and comparison,” in Proc. Forum Bildverarbeitung, vol. 6, 2012, pp. 1–12.
D. Stutz, “Superpixel segmentation: An evaluation,” in German conference on pattern recognition. Springer, 2015, pp. 555–562.
B. Mathieu, A. Crouzil, and J. B. Puel, “Oversegmentation methods: a new evaluation,” in Iberian Conference on Pattern Recognition and Image Analysis. Springer, 2017, pp. 185–193.
L. A. C. Mansilla and P. A. V. Miranda, “Oriented image foresting transform segmentation: Connectivity constraints with adjustable width,” in 2016 29th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 2016, pp. 289–296.
E. B. Alexandre, A. S. Chowdhury, A. X. Falcao, and P. A. V. Miranda, “Ift-slic: A general framework for superpixel generation based on simple linear iterative clustering and image foresting transform,” in 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, 2015, pp. 337–344.
J. Shi, Q. Yan, L. Xu, and J. Jia, “Hierarchical image saliency detection on extended cssd,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 4, pp. 717–729, 2015.
F. C. Belém, S. J. F. Guimarães, and A. X. Falcão, “Superpixel segmentation using dynamic and iterative spanning forest,” IEEE Signal Processing Letters, vol. 27, pp. 1440–1444, 2020.
X. Wei, Q. Yang, Y. Gong, N. Ahuja, and M.-H. Yang, “Superpixel hierarchy,” IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 4838–4849, 2018.
S. Bobbia, R. Macwan, Y. Benezeth, K. Nakamura, R. Gomez, and J. Dubois, “Iterative boundaries implicit identification for superpixels segmentation: a real-time approach,” IEEE Access, vol. 9, pp. 77 250–77 263, 2021.
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2274–2282, 2012.
D. Chai, “Rooted spanning superpixels,” International Journal of Computer Vision, vol. 128, no. 12, pp. 2962–2978, 2020.
F. C. Belém, B. Perret, J. Cousty, S. J. Guimarães, and A. X. Falcão, “Towards a simple and efficient object-based superpixel delineation framework,” in 2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2021, pp. 346–353.
X. Kang, L. Zhu, and A. Ming, “Dynamic random walk for superpixel segmentation,” IEEE Transactions on Image Processing, vol. 29, pp. 3871–3884, 2020.
H. Peng, A. I. Aviles-Rivero, and C.-B. Schönlieb, “Hers superpixels: Deep affinity learning for hierarchical entropy rate segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 217–226.
L. Zhu, Q. She, B. Zhang, Y. Lu, Z. Lu, D. Li, and J. Hu, “Learning the superpixel in a non-iterative and lifelong manner,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1225–1234.
J. E. Vargas-Mu˜noz, A. S. Chowdhury, E. B. Alexandre, F. L. Galvão, P. A. V. Miranda, and A. X. Falcão, “An iterative spanning forest framework for superpixel segmentation,” IEEE Transactions on Image Processing, vol. 28, no. 7, pp. 3477–3489, 2019.
R. Achanta and S. Susstrunk, “Superpixels and polygons using simple non-iterative clustering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7 2017.
Z. Ban, J. Liu, and L. Cao, “Superpixel segmentation using gaussian mixture model,” IEEE Transactions on Image Processing, vol. 27, no. 8, pp. 4105–4117, 2018.
Z. Li and J. Chen, “Superpixel segmentation using linear spectral clustering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6 2015.
R. Giraud, V.-T. Ta, and N. Papadakis, “Robust superpixels using color and contour features along linear path,” Computer Vision and Image Understanding, vol. 170, pp. 1–13, 2018.
M. Van den Bergh, X. Boix, G. Roig, B. de Capitani, and L. Van Gool, “Seeds: Superpixels extracted via energy-driven sampling,” in Computer Vision – ECCV 2012, A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 13–26.
M.-Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, “Entropy rate superpixel segmentation,” in CVPR 2011, 2011, pp. 2097–2104.
J. Yao, M. Boben, S. Fidler, and R. Urtasun, “Real-time coarse-tofine topologically preserving segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2947–2955.
C. Conrad, M. Mertz, and R. Mester, “Contour-relaxed superpixels,” in Energy Minimization Methods in Computer Vision and Pattern Recognition. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 280–293.
P. Buyssens, I. Gardin, and S. Ruan, “Eikonal based region growing for superpixels generation: Application to semi-supervised real time organ segmentation in ct images,” IRBM, vol. 35, no. 1, pp. 20–26, 2014, biomedical image segmentation using variational and statistical approaches. [Online]. Available: [link].
D. R. Martin, C. C. Fowlkes, and J. Malik, “Learning to detect natural image boundaries using local brightness, color, and texture cues,” IEEE transactions on pattern analysis and machine intelligence, vol. 26, no. 5, pp. 530–549, 2004.
I. B. Barcelos, F. D. C. Belém, L. D. M. João, A. X. Falcão, and G. S. JF, “Improving color homogeneity measure in superpixel segmentation assessment,” in 2022 35th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), vol. 1. IEEE, 2022, pp. 79–84.
F. Belém, I. Borlido, L. João, B. Perret, J. Cousty, S. J. F. Guimarães, and A. Falcão, “Fast and effective superpixel segmentation using accurate saliency estimation,” in Discrete Geometry and Mathematical Morphology, ´ E. Baudrier, B. Naegel, A. Krähenbühl, and M. Tajine, Eds. Cham: Springer International Publishing, 2022, pp. 261–273.
Publicado
06/11/2023
Como Citar
BARCELOS, Isabela Borlido; FALCÃO, Alexandre X.; GUIMARÃES, Silvio J. F..
A survey on the state-of-the-art superpixel segmentation. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 36. , 2023, Rio Grande/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 14-20.
DOI: https://doi.org/10.5753/sibgrapi.est.2023.27446.
