A multi-stream dense network with different receptive fields to assess visual quality

Luan A. Gonçalves; Ronaldo F. Zampolo; Fabrício B. Barros

doi:10.5753/kdmile.2019.8790

Luan A. Gonçalves Universidade Federal do Pará
Ronaldo F. Zampolo Universidade Federal do Pará
Fabrício B. Barros Universidade Federal do Pará

DOI: https://doi.org/10.5753/kdmile.2019.8790

Resumo

The prediction of visual quality is crucial in image and video systems. Image quality metrics based on the mean square error prevail in the field, due to their mathematical straightforwardness, even though they do not correlate well with the visual human perception. Latest achievements in the area support that the use of convolutional neural networks (CNN) to assess perceptual visual quality is a clear trend. Results in other applications, like blur detection and de-raining, indicate the combination of different receptive fields (i.e., convolutional kernels with different dimensions) improves a CNN performance. However, to the best of our knowledge, the role of different receptive fields in visual quality characterization is still an open issue. Thus, in this paper, we investigate the influence of using different receptive fields to predict image distortion. Specifically, we propose a multi-stream dense network that estimates a spatially-varying quality metric parameter from either reference or distorted images. The performance of the proposed method is compared with a competing state-of-the-art approach by using a public image database. Results show the proposed strategy outperforms the competing technique when the quality metric parameter is estimated from degraded images.

Palavras-chave: Convolution neural network, different receptive fields, differential mean opinion score, multi-stream dense network, peak signal-to-noise ratio, visual quality assessment

Referências

Bell, A. J. and Sejnowski, T. J. The “independent components” of natural scenes are edge filters. Vision Research 37 (23): 3327-3338, 1997.

Bosse, S., Becker, S., Fisches, Z. V., Samek, W., and Wiegand, T. Neural Network-Based Estimation of Distortion Sensitivity for Image Quality Prediction. In 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, Athens, Greece, pp. 629–633, 2018.

Bosse, S., Becker, S., Müller, K.-R., Samek, W., and Wiegand, T. Estimation of distortion sensitivity for visual quality prediction using a convolutional neural network. Digital Signal Processing vol. 91, pp. 54–64, dec, 2019.

Bosse, S., Maniry, D., Müller, K.-R., Wiegand, T., and Samek, W. Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment. IEEE Transactions on Image Processing vol. 17, pp. 2016–219, dec, 2016.

Gillibert, L., Chabardès, T., and Marcotegui, B. Local multiscale blur estimation based on toggle mapping for sharp region extraction. IET Image Processing 12 (12): 2138–2146, dec, 2018.

Girod, B. Whats Wrong with Mean-squared Error? In A. B. Wattson (Ed.), Digital Images and Human Visions. MIT press, Cambridge,MA, pp. 207–220, 1993.

Huang, R., Feng, W., Fan, M., Wan, L., and Sun, J. Multiscale blur detection by learning discriminative deep features. Neurocomputing vol. 285, pp. 154–166, apr, 2018.

Kang, L., Ye, P., Li, Y., and Doermann, D. Convolutional Neural Networks for No-Reference Image Quality Assessment. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Columbus, OH, USA, pp. 1733–1740, 2014.

Lin Zhang, Lei Zhang, Xuanqin Mou, and Zhang, D. FSIM: A Feature Similarity Index for Image Quality Assessment. IEEE Transactions on Image Processing 20 (8): 2378–2386, aug, 2011.

Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. In 31st Conference on Neural Information Processing Systems (NIPS 2017). NIPS, Long Beach, CA, USA., 2017.

Reisenhofer, R., Bosse, S., Kutyniok, G., and Wiegand, T. A Haar wavelet-based perceptual similarity index for image quality assessment. Signal Processing: Image Communication vol. 61, pp. 33–43, feb, 2018.

Ruderman, D. L. The statistics of natural images. Network: Computation in Neural Systems 5 (4): 517–548, 1994.

Sheikh, H. R., Sabir, M. F., and Bovik, A. C. A statistical evaluation of recent full reference image quality assessment algorithms. Trans. Img. Proc. 15 (11): 3440–3451, Nov., 2006.

Wang, Z., B. A. S. H. S. E. Modern image quality assessment. Synthesis Lectures on Image, Video, and Multimedia Processing 2 (1): 1–156, jan, 2006.

Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13 (4): 600–612, apr, 2004.

Wang, Z., Simoncelli, E., and Bovik, A. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. IEEE, Pacific Grove, CA, USA, pp. 1398–1402, 2003.

Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., and Yan, S. Deep Joint Rain Detection and Removal from a Single Image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, USA, 2016.

Zhang, H. and Patel, V. M. Density-aware Single Image De-raining using a Multi-stream Dense Network. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, USA, 2018.