Unsupervised Selective Rank Fusion on Content-Based Image Retrieval

  • Lucas Pascotti Valem UNESP
  • Daniel Carlos Guimarães Pedronette UNESP

Resumo


Mainly due to the evolution of technologies to store and share images, the growth of image collections have been remarkable for years. Therefore, developing effective methods to index and retrieve such extensive available visual information is indispensable. The CBIR (Content-Based Image Retrieval) systems are one of the main solutions for image retrieval tasks. These systems are mainly supported by the use of different visual descriptors and machine learning methods. Despite the relevant advances in the area, mainly driven by deep learning technologies, accurately computing the similarity between images remains a complex task in various scenarios due to the well known semantic gap problem. As distinct features produce complementary ranking results with different effectiveness performance, a promising solution consists in combining them. However, how to decide which visual features to combine is a very challenging task. This work proposes three novel methods for selecting and combining ranked lists by estimating their effectiveness in an unsupervised way. The approaches were evaluated in five different image collections and several descriptors, achieving results comparable or superior to the state-of-the-art in most of the evaluated scenarios.

Referências

L. P. Valem and D. C. G. Pedronette, “Combinação Seletiva Não Supervisionada de Listas Ranqueadas Aplicada à Busca de Imagens pelo Conteúdo.” 2019, Dissertation (M.Sc. in Computer Science), UNESP (Universidade Estadual Paulista Júlio de Mesquita Filho), Rio Claro, São Paulo, Brazil.

L. P. Valem and D. C. G. Pedronette, “Unsupervised selective rank fusion for image retrieval tasks,” Neurocomputing, 2019 (Accept with Minor Revision).

L. P. Valem and D. C. G. Pedronette, “Graph-based selective rank fusion for unsupervised image retrieval,” Pattern Recognition Letters, 2019 (Submitted).

L. P. Valem and D. C. G. Pedronette, “An unsupervised genetic algorithm framework for rank selection and fusion on image retrieval,” in Proceedings of the 2019 on International Conference on Multimedia Retrieval, ser. ICMR ’19. New York, NY, USA: ACM, 2019, pp. 58–62. https://doi.org/10.1145/3323873.3325022

R. Datta, D. Joshi, J. Li, and J. Z. Wang, “Image retrieval: Ideas, influences, and trends of the new age,” ACM Computing Surveys, vol. 40, no. 2, pp. 5:1–5:60, 2008. https://doi.org/10.1145/1348246.1348248

R. D. S. Torres and A. X. Falcão, “Content-based image retrieval: Theory and applications,” Revista de Informática Teórica e Aplicada, vol. 13, pp. 161–185, 2006.

L. Piras and G. Giacinto, “Information fusion in content based image retrieval: A comprehensive overview,” vol. 37, no. Supplement C, 2017, pp. 50 – 60. https://doi.org/10.1016/j.inffus.2017.01.003

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proceedings of the 25th International Conference on Neural Information Processing Systems -Volume 1, ser. NIPS’12, 2012, pp. 1097–1105.

L. Deng, “A tutorial survey of architectures, algorithms, and applications for deep learning,” APSIPA Transactions on Signal and Information Processing, vol. 3, 2014. https://doi.org/10.1017/atsip.2013.9

F. F. Faria, A. Veloso, H. M. Almeida, E. Valle, R. d. S. Torres, M. A. Gonçalves, and W. Meira, Jr., “Learning to rank for content-based image retrieval,” in Proceedings of the International Conference on Multimedia Information Retrieval, ser. MIR ’10, 2010, pp. 285–294. https://doi.org/10.1145/1743384.1743434

L. P. Valem and D. C. G. Pedronette, “Unsupervised selective rank fusion for image retrieval tasks,” in Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, ser. ICMR ’17. New York, NY, USA: ACM, 2017, pp. 107–111.

L. P. Valem, D. C. G. Pedronette, and J. Almeida, “Unsupervised similarity learning through Cartesian product of ranking references,” Pattern Recognition Letters, vol. 114, pp. 41 – 52, 2018.

J. A. Vargas Muñoz, R. da Silva Torres, and M. A. Gonçalves, “A soft computing approach for learning to aggregate rankings,” in Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, ser. CIKM ’15, 2015, pp. 83–92. https://doi.org/10.1145/2806416.2806478

X. He, D. Cai, and P. Niyogi, “Laplacian score for feature selection,” in Proceedings of the 18th International Conference on Neural Information Processing Systems, ser. NIPS’05, 2005, pp. 507–514.

Z. Zhao and H. Liu, “Spectral feature selection for supervised and unsupervised learning,” in Proceedings of the 24th International Conference on Machine Learning, ser. ICML ’07, 2007, pp. 1151–1157. https://doi.org/10.1145/1273496.1273641

D. C. G. Pedronette and R. da S. Torres, “A correlation graph approach for unsupervised manifold learning in image retrieval tasks,” Neurocomputing, vol. 208, no. Sup C, pp. 66 – 79, 2016. https://doi.org/10.1016/j.neucom.2016.03.081

L. Zheng, S. Wang, L. Tian, F. He, Z. Liu, and Q. Tian, “Query-adaptive late fusion for image search and person re-identification,” in CVPR,2015. https://doi.org/10.1109/CVPR.2015.7298783

S. Zhang, M. Yang, T. Cour, K. Yu, and D. Metaxas, “Query specific rank fusion for image retrieval,” IEEE TPAMI, vol. 37, no. 4, pp. 803–815, April 2015. https://doi.org/10.1109/TPAMI.2014.2346201

C. G. M. Snoek, M. Worring, and A. W. M. Smeulders, “Early versus late fusion in semantic video analysis,” in Proceedings of the 13th Annual ACM International Conference on Multimedia, ser. MULTIMEDIA ’05, 2005, pp. 399–402. https://doi.org/10.1145/1101149.1101236

P. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, “Multimodal fusion for multimedia analysis: a survey,” Multimedia Systems, vol. 16, no. 6, pp. 345–379, Nov 2010. https://doi.org/10.1007/s00530-010-0182-0

D. C. G. Pedronette, O. A. Penatti, and R. da S. Torres, “Unsupervised manifold learning using reciprocal kNN graphs in image re-ranking and rank aggregation tasks,” Image and Vision Computing, vol. 32, no. 2, pp. 120 – 130, 2014. https://doi.org/10.1016/j.imavis.2013.12.009

D. C. G. Pedronette and R. d. S. Torres, “Unsupervised effectiveness estimation for image retrieval using reciprocal rank information,” in 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, 2015, pp. 321–328. https://doi.org/10.1109/SIBGRAPI.2015.28

H. Jegou, M. Douze, and C. Schmid, “Hamming embedding and weak geometric consistency for large scale image search,” in European Conference on Computer Vision, ser. ECCV ’08, 2008, pp. 304–317. https://doi.org/10.1007/978-3-540-88682-2_24

G. Tolias, Y. Avrithis, and H. Jégou, “To aggregate or not to aggregate: Selective match kernels for image search,” in IEEE International Conference on Computer Vision (ICCV’2013), Dec 2013, pp. 1401–1408. https://doi.org/10.1109/ICCV.2013.177

M. Paulin, J. Mairal, M. Douze, Z. Harchaoui, F. Perronnin, and C. Schmid, “Convolutional patch representations for image retrieval: An unsupervised approach,” Int. Journal of Computer Vision, 2017. https://doi.org/10.1007/s11263-016-0924-3

D. Qin, C. Wengert, and L. V. Gool, “Query adaptive similarity for large scale object retrieval,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2013), June 2013, pp. 1610–1617. https://doi.org/10.1109/CVPR.2013.211

L. Zheng, S. Wang, and Q. Tian, “Coupled binary embedding for large-scale image retrieval,” IEEE Transactions on Image Processing (TIP),vol. 23, no. 8, pp. 3368–3380, 2014. https://doi.org/10.1109/TIP.2014.2330763

S. Sun, Y. Li, W. Zhou, Q. Tian, and H. Li, “Local residual similarity for image re-ranking,” Information Sciences, vol. 417, no. Sup. C, pp. 143 – 153, 2017. https://doi.org/10.1016/j.ins.2017.07.004

L. Zheng, S. Wang, Z. Liu, and Q. Tian, “Packing and padding: Coupled multi-index for accurate image retrieval,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2014), June 2014, pp. 1947–1954. https://doi.org/10.1109/CVPR.2014.250

D. C. G. Pedronette, F. M. F. Gonçalves, and I. R. Guilherme, “Unsupervised manifold learning through reciprocal kNN graph and Connected Components for image retrieval tasks,” Pattern Recognition, vol. 75, pp. 161 – 174, 2018. https://doi.org/10.1016/j.patcog.2017.05.009

X. Li, M. Larson, and A. Hanjalic, “Pairwise geometric matching for large-scale object retrieval,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2015), June 2015, pp. 5153–5161. https://doi.org/10.1109/CVPR.2015.7299151

Z. Liu, S. Wang, L. Zheng, and Q. Tian, “Robust imagegraph: Rank-level feature fusion for image search,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3128–3141, 2017. https://doi.org/10.1109/TIP.2017.2660244

D. Nistér and H. Stewénius, “Scalable recognition with a vocabulary tree,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2006), vol. 2, 2006, pp. 2161–2168. https://doi.org/10.1109/CVPR.2006.264

L. Zheng, S. Wang, and Q. Tian, “Lp-norm idf for scalable image retrieval,” IEEE TIP, vol. 23, no. 8, pp. 3604–3617, Aug 2014. https://doi.org/10.1109/TIP.2014.2329182

B. Wang, J. Jiang, W. Wang, Z.-H. Zhou, and Z. Tu, “Unsupervised metric fusion by cross diffusion,” in CVPR, 2012, pp. 3013 –3020. https://doi.org/10.1109/CVPR.2012.6248029

S. Bai and X. Bai, “Sparse contextual activation for efficient visual re-ranking,” IEEE Trans. on Image Processing (TIP), vol. 25, no. 3, pp. 1056–1069, 2016. https://doi.org/10.1109/TIP.2016.2514498

L. Xie, R. Hong, B. Zhang, and Q. Tian, “Image classification and retrieval are one,” in ACM ICMR’2015, 2015, pp. 3–10. https://doi.org/10.1145/2671188.2749289

S. Bai, X. Bai, Q. Tian, and L. J. Latecki, “Regularized diffusion process for visual retrieval,” in Conf. on Artificial Intelligence (AAAI), 2017, pp. 3967–3973.

M.-E. Nilsback and A. Zisserman, “A visual vocabulary for flower classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 2006, pp. 1447–1454. https://doi.org/10.1109/CVPR.2006.42

G.-H. Liu and J.-Y. Yang, “Content-based image retrieval using color difference histogram,” Pattern Recognition, vol. 46, no. 1, pp. 188 – 198, 2013. https://doi.org/10.1016/j.patcog.2012.06.001

L. P. Valem and D. C. G. Pedronette, “Selection and combination of unsupervised learning methods for image retrieval,” in Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing, ser. CBMI ’17, 2017, pp. 27:1–27:6. https://doi.org/10.1145/3095713.3095741

J. Almeida, L. P. Valem, and D. C. G. Pedronette, “A rank aggregation framework for video interestingness prediction,” in Proceedings of the 9th International Conference on Image Analysis and Processing, ser. ICIAP ’17, 2017. https://doi.org/10.1007/978-3-319-68560-1_1

L. P. Valem, D. C. G. Pedronette, F. Breve, and I. R. Guilherme, “Manifold correlation graph for semi-supervised learning,” in 2018 International Joint Conference on Neural Networks (IJCNN), 2018, pp. 1–7. https://doi.org/10.1109/IJCNN.2018.8489487

L. P. Valem, C. R. D. Oliveira, D. C. G. Pedronette, and J. Almeida, “Unsupervised similarity learning through rank correlation and knn sets,” in ACM Trans. Multimedia Comput. Commun. Appl., vol. 14, no. 4. New York, NY, USA: ACM, Oct. 2018, pp. 80:1–80:23. https://doi.org/10.1145/3241053

D. C. G. Pedronette, L. P. Valem, J. Almeida, and R. da Silva Torres, “Multimedia retrieval through unsupervised hypergraph-based manifold ranking,” IEEE Transactions on Image Processing (TIP), 2019, On-line, to appear. https://doi.org/10.1109/TIP.2019.2920526
Publicado
28/10/2019
Como Citar

Selecione um Formato
VALEM, Lucas Pascotti; PEDRONETTE, Daniel Carlos Guimarães. Unsupervised Selective Rank Fusion on Content-Based Image Retrieval. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 32. , 2019, Rio de Janeiro. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2019 . p. 63-69. DOI: https://doi.org/10.5753/sibgrapi.est.2019.8303.