AEIMPS: Deep Autoencoder for Image Retargeting Quality Assessment
Resumo
Evaluating retargeting image operators is a subjective task and, therefore, challenging to execute without human interference. Image Retargeting Quality Algorithms execute this task, giving some score to the retargeted image and, usually, trying to get a result similar to a human opinion since humans generally agree with each other on the quality of a resized image. Therefore, we propose an Autoencoder-based IRQA named AutoEncoder Information MaP Similarity (AEIMPS) to address this task using the NVAE architecture. In our experiments, besides the retargeting ratio, we use the latent space and the reconstructed image in the IRQA. AIEMPS achieved an average performance compared to other IRQAs in the literature.
Referências
A. Shamir and O. Sorkine, "Visual media retargeting," in ACM SIGGRAPH ASIA 2009 Courses, ser. SIGGRAPH ASIA '09. New York, NY, USA: Association for Computing Machinery, 2009. [Online]. Available: https://doi.org/10.1145/1665817.1665828
D. Vaquero, M. Turk, K. Pulli, M. Tico, and N. Gelfand, "A survey of image retargeting techniques," in Applications of Digital Image Processing XXXIII, vol. 7798. SPIE, 2010, pp. 328-342.
M. Rubinstein, A. Shamir, and S. Avidan, "Improved seam carving for video retargeting," ACM TRANSACTIONS ON GRAPHICS, vol. 27, no. 3, AUG 2008, aCM SIGGRAPH Conference 2008, Singapore, SINGAPORE, AUG 11-15, 2008.
Z. Karni, D. Freedman, and C. Gotsman, "Energy-based image deformation," COMPUTER GRAPHICS FORUM, vol. 28, no. 5, SI, pp. 1257-1268, JUL 2009, 7th Eurographics Symposium on Geometry Processing (SGP), Berlin, GERMANY, JUL 15-17, 2009.
Y. Pritch, E. Kav-Venaki, and S. Peleg, "Shift-map image editing," in 2009 IEEE 12TH INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), ser. IEEE International Conference on Computer Vision. IEEE; IEEE Comp Soc, 2009, pp. 151-158, 12th IEEE International Conference on Computer Vision, Kyoto, JAPAN, SEP 29-OCT 02, 2009.
M. Rubinstein, A. Shamir, and S. Avidan, "Multi-operator media retargeting," ACM TRANSACTIONS ON GRAPHICS, vol. 28, no. 3, AUG 2009, aCM SIGGRAPH Conference 2009, New Orleans, LA, 2009.
L. Wolf, M. Guttmann, and D. Cohen-Or, "Non-homogeneous content-driven video-retargeting," in 2007 IEEE 11TH INTERNATIONAL CONFERENCE ON COMPUTER VISION, VOLS 1-6, ser. IEEE International Conference on Computer Vision. IEEE, 2007, pp. 1418-1423, 11th IEEE International Conference on Computer Vision, Rio de Janeiro, BRAZIL, OCT 14-21, 2007.
Y.-S. Wang, C.-L. Tai, O. Sorkine, and T.-Y. Lee, "Optimized scale-and-stretch for image resizing," ACM TRANSACTIONS ON GRAPHICS, vol. 27, no. 5, DEC 2008, aCM SIGGRAPH Conference 2008, Singapore, SINGAPORE, AUG 11-15, 2008.
Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 13, no. 4, pp. 600-612, APR 2004.
H. Sheikh and A. Bovik, "Image information and visual quality," in 2004 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL III, PROCEEDINGS: IMAGE AND MULTIDIMENSIONAL SIGNAL PROCESSING SPECIAL SESSIONS. IEEE Signal Proc Soc; IEEE, 2004, pp. 709-712, iEEE International Conference on Acoustics, Speech, and Signal Processing, Montreal, CANADA, MAY 17-21, 2004.
C. Liu, J. Yuen, and A. Torralba, "Sift flow: Dense correspondence across scenes and its applications," IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 33, no. 5, pp. 978-994, MAY 2011.
J. Zhang and C. C. J. Kuo, "An objective quality of experience (qoe) assessment index for retargeted images," in PROCEEDINGS OF THE 2014 ACM CONFERENCE ON MULTIMEDIA (MM'14). Assoc Comp Machinery; ACM SIGMM; FXPAL; Google; IBM; Microsoft Res; NASA Florida Space Grant Consortium; Yahoo Lab; Yandex, 2014, pp. 257-266, aCM Conference on Multimedia (MM), Univ Cent Florida, Orlando, FL, NOV 03-07, 2014.
C.-C. Hsu, C.-W. Lin, Y. Fang, and W. Lin, "Objective quality assessment for image retargeting based on perceptual distortion and information loss," in 2013 IEEE INTERNATIONAL CONFERENCE ON VISUAL COMMUNICATIONS AND IMAGE PROCESSING (IEEE VCIP 2013). IEEE; Sarawak Convent Bur; Malaysia Convent & Exhibit Bur; IEEE Circuits & Syst Soc; Neuramatix, 2013, iEEE International Conference on Visual Communications and Image Processing (VCIP), Kuching, MALAYSIA, NOV 17-20, 2013.
Y. Zhang, Y. Fang, W. Lin, X. Zhang, and L. Li, "Backward registration-based aspect ratio similarity for image retargeting quality assessment," IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 25, no. 9, pp. 4286-4297, SEP 2016.
S. A. F. Oliveira, S. S. A. Alves, J. P. P. Gomes, and A. R. Rocha Neto, "A bi-directional evaluation-based approach for image retargeting quality assessment," COMPUTER VISION AND IMAGE UNDERSTANDING, vol. 168, no. SI, pp. 172-181, MAR 2018.
A. Liu, W. Lin, H. Chen, and P. Zhang, "Image retargeting quality assessment based on support vector regression," SIGNAL PROCESSINGIMAGE COMMUNICATION, vol. 39, no. B, SI, pp. 444-456, NOV 2015.
A. Vahdat and J. Kautz, "NVAE: A deep hierarchical variational autoencoder," in Neural Information Processing Systems (NeurIPS), 2020.
L. Ma, W. Lin, C. Deng, and K. N. Ngan, "Image retargeting quality assessment: A study of subjective scores and objective metrics," IEEE Journal of Selected Topics in Signal Processing, vol. 6, no. 6, pp. 626-639, 2012.
Y. Zhang, Y. Fang, W. Lin, X. Zhang, and L. Li, "Backward registration-based aspect ratio similarity for image retargeting quality assessment," IEEE Transactions on Image Processing, vol. 25, no. 9, pp. 4286-4297, 2016.
Y.-J. Liu, X. Luo, Y.-M. Xuan, W.-F. Chen, and X.-L. Fu, "Image retargeting quality assessment," COMPUTER GRAPHICS FORUM, vol. 30, no. 2, pp. 583-592, 2011.
Y. Zhang, K. N. Ngan, L. Ma, and H. Li, "Objective quality assessment of image retargeting by incorporating fidelity measures and inconsistency detection," IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 26, no. 12, pp. 5980-5993, DEC 2017.
L. Ma, C. Deng, W. Lin, K. N. Ngan, and L. Xu, Retargeted Image Quality Assessment: Current Progresses and Future Trends. Cham: Springer International Publishing, 2015, pp. 213-242. [Online]. Available: https://doi.org/10.1007/978-3-319-10368-6_8
A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," INTERNATIONAL JOURNAL OF COMPUTER VISION, vol. 42, no. 3, pp. 145-175, 2001.
L. Ma, L. Xu, Y. Zhang, Y. Yan, and K. N. Ngan, "No-reference retargeted image quality assessment based on pairwise rank learning," IEEE TRANSACTIONS ON MULTIMEDIA, vol. 18, no. 11, pp. 2228-2237, NOV 2016.
D. Messing, P. van Beek, and J. Errico, "The mpeg-7 colour structure descriptor: Image description using colour and local spatial information," in 2001 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOL I, PROCEEDINGS, ser. IEEE International Conference on Image Processing ICIP. IEEE Signal Processing Soc; IEEE, 2001, pp. 670-673, international Conference on Image Processing (ICIP 2001), THESSALONIKI, GREECE, OCT 07-10, 2001.
O. Pele and M. Werman, "Fast and robust earth mover's distances," in 2009 IEEE 12TH INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), ser. IEEE International Conference on Computer Vision. IEEE; IEEE Comp Soc, 2009, pp. 460-467, 12th IEEE International Conference on Computer Vision, Kyoto, JAPAN, SEP 29-OCT 02, 2009.
D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, "Summarizing visual data using bidirectional similarity," in 2008 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, VOLS 1-12, ser. IEEE Conference on Computer Vision and Pattern Recognition. IEEE Comp Soc, 2008, pp. 3887+, iEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, JUN 23-28, 2008.
A. Bosch, A. Zisserman, and X. Munoz, "Image classification using random forests and ferns," in 2007 IEEE 11TH INTERNATIONAL CONFERENCE ON COMPUTER VISION, VOLS 1-6, ser. IEEE International Conference on Computer Vision. IEEE, 2007, pp. 1863-1870, 11th IEEE International Conference on Computer Vision, Rio de Janeiro, BRAZIL, OCT 14-21, 2007.
S. Zhao, J. Song, and S. Ermon, "Infovae: Balancing learning and inference in variational autoencoders," in Proceedings of the aaai conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 5885-5892.
D. Vaquero, M. Turk, K. Pulli, M. Tico, and N. Gelfand, "A survey of image retargeting techniques," in Applications of Digital Image Processing XXXIII, vol. 7798. SPIE, 2010, pp. 328-342.
M. Rubinstein, A. Shamir, and S. Avidan, "Improved seam carving for video retargeting," ACM TRANSACTIONS ON GRAPHICS, vol. 27, no. 3, AUG 2008, aCM SIGGRAPH Conference 2008, Singapore, SINGAPORE, AUG 11-15, 2008.
Z. Karni, D. Freedman, and C. Gotsman, "Energy-based image deformation," COMPUTER GRAPHICS FORUM, vol. 28, no. 5, SI, pp. 1257-1268, JUL 2009, 7th Eurographics Symposium on Geometry Processing (SGP), Berlin, GERMANY, JUL 15-17, 2009.
Y. Pritch, E. Kav-Venaki, and S. Peleg, "Shift-map image editing," in 2009 IEEE 12TH INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), ser. IEEE International Conference on Computer Vision. IEEE; IEEE Comp Soc, 2009, pp. 151-158, 12th IEEE International Conference on Computer Vision, Kyoto, JAPAN, SEP 29-OCT 02, 2009.
M. Rubinstein, A. Shamir, and S. Avidan, "Multi-operator media retargeting," ACM TRANSACTIONS ON GRAPHICS, vol. 28, no. 3, AUG 2009, aCM SIGGRAPH Conference 2009, New Orleans, LA, 2009.
L. Wolf, M. Guttmann, and D. Cohen-Or, "Non-homogeneous content-driven video-retargeting," in 2007 IEEE 11TH INTERNATIONAL CONFERENCE ON COMPUTER VISION, VOLS 1-6, ser. IEEE International Conference on Computer Vision. IEEE, 2007, pp. 1418-1423, 11th IEEE International Conference on Computer Vision, Rio de Janeiro, BRAZIL, OCT 14-21, 2007.
Y.-S. Wang, C.-L. Tai, O. Sorkine, and T.-Y. Lee, "Optimized scale-and-stretch for image resizing," ACM TRANSACTIONS ON GRAPHICS, vol. 27, no. 5, DEC 2008, aCM SIGGRAPH Conference 2008, Singapore, SINGAPORE, AUG 11-15, 2008.
Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 13, no. 4, pp. 600-612, APR 2004.
H. Sheikh and A. Bovik, "Image information and visual quality," in 2004 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL III, PROCEEDINGS: IMAGE AND MULTIDIMENSIONAL SIGNAL PROCESSING SPECIAL SESSIONS. IEEE Signal Proc Soc; IEEE, 2004, pp. 709-712, iEEE International Conference on Acoustics, Speech, and Signal Processing, Montreal, CANADA, MAY 17-21, 2004.
C. Liu, J. Yuen, and A. Torralba, "Sift flow: Dense correspondence across scenes and its applications," IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 33, no. 5, pp. 978-994, MAY 2011.
J. Zhang and C. C. J. Kuo, "An objective quality of experience (qoe) assessment index for retargeted images," in PROCEEDINGS OF THE 2014 ACM CONFERENCE ON MULTIMEDIA (MM'14). Assoc Comp Machinery; ACM SIGMM; FXPAL; Google; IBM; Microsoft Res; NASA Florida Space Grant Consortium; Yahoo Lab; Yandex, 2014, pp. 257-266, aCM Conference on Multimedia (MM), Univ Cent Florida, Orlando, FL, NOV 03-07, 2014.
C.-C. Hsu, C.-W. Lin, Y. Fang, and W. Lin, "Objective quality assessment for image retargeting based on perceptual distortion and information loss," in 2013 IEEE INTERNATIONAL CONFERENCE ON VISUAL COMMUNICATIONS AND IMAGE PROCESSING (IEEE VCIP 2013). IEEE; Sarawak Convent Bur; Malaysia Convent & Exhibit Bur; IEEE Circuits & Syst Soc; Neuramatix, 2013, iEEE International Conference on Visual Communications and Image Processing (VCIP), Kuching, MALAYSIA, NOV 17-20, 2013.
Y. Zhang, Y. Fang, W. Lin, X. Zhang, and L. Li, "Backward registration-based aspect ratio similarity for image retargeting quality assessment," IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 25, no. 9, pp. 4286-4297, SEP 2016.
S. A. F. Oliveira, S. S. A. Alves, J. P. P. Gomes, and A. R. Rocha Neto, "A bi-directional evaluation-based approach for image retargeting quality assessment," COMPUTER VISION AND IMAGE UNDERSTANDING, vol. 168, no. SI, pp. 172-181, MAR 2018.
A. Liu, W. Lin, H. Chen, and P. Zhang, "Image retargeting quality assessment based on support vector regression," SIGNAL PROCESSINGIMAGE COMMUNICATION, vol. 39, no. B, SI, pp. 444-456, NOV 2015.
A. Vahdat and J. Kautz, "NVAE: A deep hierarchical variational autoencoder," in Neural Information Processing Systems (NeurIPS), 2020.
L. Ma, W. Lin, C. Deng, and K. N. Ngan, "Image retargeting quality assessment: A study of subjective scores and objective metrics," IEEE Journal of Selected Topics in Signal Processing, vol. 6, no. 6, pp. 626-639, 2012.
Y. Zhang, Y. Fang, W. Lin, X. Zhang, and L. Li, "Backward registration-based aspect ratio similarity for image retargeting quality assessment," IEEE Transactions on Image Processing, vol. 25, no. 9, pp. 4286-4297, 2016.
Y.-J. Liu, X. Luo, Y.-M. Xuan, W.-F. Chen, and X.-L. Fu, "Image retargeting quality assessment," COMPUTER GRAPHICS FORUM, vol. 30, no. 2, pp. 583-592, 2011.
Y. Zhang, K. N. Ngan, L. Ma, and H. Li, "Objective quality assessment of image retargeting by incorporating fidelity measures and inconsistency detection," IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 26, no. 12, pp. 5980-5993, DEC 2017.
L. Ma, C. Deng, W. Lin, K. N. Ngan, and L. Xu, Retargeted Image Quality Assessment: Current Progresses and Future Trends. Cham: Springer International Publishing, 2015, pp. 213-242. [Online]. Available: https://doi.org/10.1007/978-3-319-10368-6_8
A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," INTERNATIONAL JOURNAL OF COMPUTER VISION, vol. 42, no. 3, pp. 145-175, 2001.
L. Ma, L. Xu, Y. Zhang, Y. Yan, and K. N. Ngan, "No-reference retargeted image quality assessment based on pairwise rank learning," IEEE TRANSACTIONS ON MULTIMEDIA, vol. 18, no. 11, pp. 2228-2237, NOV 2016.
D. Messing, P. van Beek, and J. Errico, "The mpeg-7 colour structure descriptor: Image description using colour and local spatial information," in 2001 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOL I, PROCEEDINGS, ser. IEEE International Conference on Image Processing ICIP. IEEE Signal Processing Soc; IEEE, 2001, pp. 670-673, international Conference on Image Processing (ICIP 2001), THESSALONIKI, GREECE, OCT 07-10, 2001.
O. Pele and M. Werman, "Fast and robust earth mover's distances," in 2009 IEEE 12TH INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), ser. IEEE International Conference on Computer Vision. IEEE; IEEE Comp Soc, 2009, pp. 460-467, 12th IEEE International Conference on Computer Vision, Kyoto, JAPAN, SEP 29-OCT 02, 2009.
D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, "Summarizing visual data using bidirectional similarity," in 2008 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, VOLS 1-12, ser. IEEE Conference on Computer Vision and Pattern Recognition. IEEE Comp Soc, 2008, pp. 3887+, iEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, JUN 23-28, 2008.
A. Bosch, A. Zisserman, and X. Munoz, "Image classification using random forests and ferns," in 2007 IEEE 11TH INTERNATIONAL CONFERENCE ON COMPUTER VISION, VOLS 1-6, ser. IEEE International Conference on Computer Vision. IEEE, 2007, pp. 1863-1870, 11th IEEE International Conference on Computer Vision, Rio de Janeiro, BRAZIL, OCT 14-21, 2007.
S. Zhao, J. Song, and S. Ermon, "Infovae: Balancing learning and inference in variational autoencoders," in Proceedings of the aaai conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 5885-5892.
Publicado
24/10/2022
Como Citar
CARVALHO, Levi C.; OLIVEIRA, Saulo A. F..
AEIMPS: Deep Autoencoder for Image Retargeting Quality Assessment. In: WORKSHOP DE TRABALHOS EM ANDAMENTO - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 35. , 2022, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 72-75.
DOI: https://doi.org/10.5753/sibgrapi.est.2022.23263.
