Hierarchical Graph Neural Networks Based on Multi-Scale Image Representations

• João Pedro Oliveira Batisteli PUC Minas
• Zenilton K. G. Patrocínio Jr PUC Minas

DOI: https://doi.org/10.5753/sibgrapi.est.2024.31646

Resumo

Image representation as graphs can enhance the understanding of image semantics and facilitate multi-scale image representation. However, existing methods often overlook the significance of the relationship between elements at each scale or fail to encode the hierarchical relationship between graph elements. To cope with that, we introduce four novel approaches for graph construction from images. These approaches utilize hierarchical image segmentation techniques to generate segmentations at multiple scales, and one of them incorporates edges to encode the relationships at each scale. Leveraging these representations, we present two new models: the Hierarchical Graph Convolutional Network for Image Classification (HGCIC) and the Hierarchical Image Graph with Scale Importance (HIGSI). HGCIC uses an adaptive depth to capture significant features and patterns at different scales, while HIGSI employs a novel readout function that weighs the importance of each scale when generating a fixed-size graph representation. Experimental results with CIFAR-10 and STL-10 datasets show that the HIGSI model outperforms (or closely matches) state-of-the-art models. The model also utilizes smaller graphs, reaching the point of using graphs with 50% of the number of nodes compared to other approaches. Additionally, HIGSI outperforms models trained with only the base graph used to create the hierarchy, achieving up to 11.54% better performance while using fewer parameters.

Referências

J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3668–3678.

W. Liang, Y. Jiang, and Z. Liu, “GraghVQA: Language-guided graph neural networks for graph-based visual question answering,” in Proceedings of the Third Workshop on Multimodal Artificial Intelligence. Mexico City, Mexico: Association for Computational Linguistics, Jun. 2021, pp. 79–86. [Online]. Available: [link]

B. Knyazev, X. Lin, M. Amer, and G. Taylor, “Image classification with hierarchical multigraph networks,” in Proceedings of the British Machine Vision Conference (BMVC), K. Sidorov and Y. Hicks, Eds. BMVA Press, September 2019, pp. 223.1–223.13.

S. Wan, C. Gong, P. Zhong, B. Du, L. Zhang, and J. Yang, “Multiscale dynamic graph convolutional network for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 5, pp. 3162–3177, 2019.

Q. Liu, L. Xiao, J. Yang, and Z. Wei, “Cnn-enhanced graph convolutional network with pixel-and superpixel-level feature fusion for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 10, pp. 8657–8671, 2020.

P. H. Avelar, A. R. Tavares, T. L. da Silveira, C. R. Jung, and L. C. Lamb, “Superpixel image classification with graph attention networks,” in 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 2020, pp. 203–209.

V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson, “Benchmarking graph neural networks,” Journal of Machine Learning Research, vol. 24, no. 43, pp. 1–48, 2023.

V. Vasudevan, M. Bassenne, M. T. Islam, and L. Xing, “Image classification using graph neural network and multiscale wavelet superpixels,” Pattern Recognition Letters, 2023.

R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2274–2282, 2012.

S. Guimarães, Y. Kenmochi, J. Cousty, Z. Patrocinio, and L. Najman, “Hierarchizing graph-based image segmentation algorithms relying on region dissimilarity,” Mathematical Morphology-Theory and Applications, vol. 2, no. 1, pp. 55–75, 2017.

L. Guigues, J. P. Cocquerez, and H. Le Men, “Scale-sets image analysis,” International Journal of Computer Vision, vol. 68, no. 3, pp. 289–317, 2006.

X. Bresson and T. Laurent, “Residual gated graph convnets,” arXiv preprint arXiv:1711.07553, 2017.

G. Corso, L. Cavalleri, D. Beaini, P. Liò, and P. Veličković, “Principal neighbourhood aggregation for graph nets,” Advances in Neural Information Processing Systems, vol. 33, pp. 13 260–13 271, 2020.

K. Han, Y. Wang, J. Guo, Y. Tang, and E. Wu, “Vision gnn: An image is worth graph of nodes,” arXiv preprint arXiv:2206.00272, 2022.

F. C. Belém, S. J. F. Guimaraes, and A. X. Falcao, “Superpixel segmentation using dynamic and iterative spanning forest,” IEEE Signal Processing Letters, vol. 27, pp. 1440–1444, 2020.

J. Cousty and L. Najman, “Incremental algorithm for hierarchical minimum spanning forests and saliency of watershed cuts,” in International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing, 2011, pp. 272–283.

L. Rampášek, M. Galkin, V. P. Dwivedi, A. T. Luu, G. Wolf, and D. Beaini, “Recipe for a general, powerful, scalable graph transformer,” Advances in Neural Information Processing Systems, vol. 35, pp. 14 501–14 515, 2022.

R. A. Cosma, L. Knobel, P. van der Linden, D. M. Knigge, and E. J. Bekkers, “Geometric superpixel representations for efficient image classification with graph neural networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 109–118.

P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.

J. P. O. Batisteli, S. J. F. Guimarães, and Z. K. G. Patrocínio, Jr, “Hierarchical graph convolutional networks for image classification,” in Brazilian Conference on Intelligent Systems – BRACIS’23, 2023, pp. 63–76.

J. P. O. Batisteli, S. J. F. Guimarães, and Z. K. G. Patrocínio, “Multi-scale image graph representation: A novel gnn approach for image classification through scale importance estimation,” in 2023 IEEE International Symposium on Multimedia (ISM), 2023, pp. 62–68.

J. P. O. Batisteli, S. J. F. Guimarães, and Z. K. G. Patrocínio, Jr, “Hierarchical graph neural networks with scale-aware readout for image classification,” International Journal of Semantic Computing, 2024.