Optimizing Super Resolution for Face Recognition
Resumo
Face Super-Resolution is a subset of Super Resolution (SR) that aims to retrieve a high-resolution (HR) image of a face from a lower resolution input. Recently, Deep Learning (DL) methods have improved drastically the quality of SR generated images. However, these qualitative improvements are not always followed by quantitative improvements in the traditional metrics of the area, namely PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index). In some cases, models that perform better in opinion scores and qualitative evaluation have worse performance in these metrics, indicating they are not sufficiently informative. To address this issue we propose a task-basedevaluation procedure based on the comparative performance of face recognition algorithms on HR and SR images to evaluate how well the models retrieve high-frequency and identity defining information. Furthermore, as our face recognition model is differentiable, this leads to a novel loss function that can be optimized to improve performance in these tasks. We successfully apply our evaluation method to validate this training method, yielding promising results
Palavras-chave:
deep learning, super resolution, face recognition, face super resolution
Referências
P. Hanhart P. Korshunov T. Ebrahimi "Benchmarking of quality metrics on ultra-high definition video sequences" 2013 18th International Conference on Digital Signal Processing (DSP) pp. 1-8 2013.
Q. Huynh-Thu M. Ghanbari "Scope of validity of psnr in image/video quality assessment" Electronics letters vol. 44 no. 13 pp. 800-2008.
C. Ledig L. Theis F. Huszár J. Caballero A. Cunningham A. Acosta A. Aitken A. Tejani J. Totz Z. Wang et al. "Photo-realistic single image super-resolution using a generative adversarial network" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 4681-42017.
J. Johnson A. Alahi L. Fei-Fei "Perceptual losses for real-time style transfer and super-resolution" European conference on computer vision pp. 694-2016.
"Mean opinion score interpretation and reporting" ITU Recommendation pp. 800.2 2013.
Y. Chen Y. Tai X. Liu C. Shen J. Yang "Fsrnet: End-to-end learning face super-resolution with facial priors" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 2492-22018.
P. Rasti T. Uiboupin S. Escalera G. Anbarjafari "Convolutional neural network super resolution for face recognition in surveillance monitoring" International conference on articulated motion and deformable objects pp. 175-2016.
P. Li L. Prieto D. Mery P. J. Flynn "On low-resolution face recognition in the wild: Comparisons and new techniques" IEEE Transactions on Information Forensics and Security 2019.
P. J. Phillips P. Grother R. Micheals D. M. Blackburn E. Tabassi M. Bone "Face recognition vendor test 2002" 2003 IEEE International SOI Conference. Proceedings (Cat. No. 03CH374pp. 44 2003.
Y. Taigman M. Yang M. Ranzato L. Wolf "Deepface: Closing the gap to human-level performance in face verification" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 1701-12014.
O. M. Parkhi A. Vedaldi A. Zisserman et al. "Deep face recognition" bmvc vol. 1 no. 3 pp. 6 2015.
C. Dong C. C. Loy K. He X. Tang "Learning a deep convolutional network for image super-resolution" European conference on computer vision pp. 184-2014.
W. Shi J. Caballero F. Huszár J. Totz A. P. Aitken R. Bishop D. Rueckert Z. Wang "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 1874-1883 2016.
Z. Wang J. Chen S. C. H. Hoi "Deep learning for image super-resolution: A survey" CoRR vol. abs/1902.06068 2019.
H. Huang R. He Z. Sun T. Tan "Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution" Proceedings of the IEEE International Conference on Computer Vision pp. 1689-12017.
W. Liu Y. Wen Z. Yu M. Li B. Raj L. Song "Sphereface: Deep hypersphere embedding for face recognition" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 212-2017.
C. Szegedy V. Vanhoucke S. Ioffe J. Shlens Z. Wojna "Rethinking the inception architecture for computer vision" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 2818-22016.
T. Salimans I. J. Goodfellow W. Zaremba V. Cheung A. Radford X. Chen "Improved techniques for training gans" CoRR vol. abs/1606.032016.
M. Heusel H. Ramsauer T. Unterthiner B. Nessler S. Hochreiter "Gans trained by a two time-scale update rule converge to a local nash equilibrium" Advances in Neural Information Processing Systems pp. 6626-62017.
D. Dai Y. Wang Y. Chen L. van Gool "Is image super-resolution helpful for other vision tasks?" 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) pp. 1-9 2016.
S. Hu R. Maschal S. S. Young T. H. Hong P. J. Phillips "Face recognition performance with superresolution" Applied optics vol. 51 no. 18 pp. 4250-42012.
A. Bulat G. Tzimiropoulos "Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 109-2018.
M. Haris G. Shakhnarovich N. Ukita Task-driven super resolution: Object detection in low-resolution images 2018.
K. Zhang Z. Zhang C.-W. Cheng W. H. Hsu Y. Qiao W. Liu T. Zhang "Super-identity convolutional neural network for face hallucination" Proceedings of the European Conference on Computer Vision (ECCV) pp. 183-2018.
K. He X. Zhang S. Ren J. Sun "Deep residual learning for image recognition" CoRR vol. abs/1512.032015.
Z. Liu P. Luo X. Wang X. Tang "Deep learning face attributes in the wild" Proceedings of International Conference on Computer Vision (ICCV) 2015.
G. B. Huang M. Ramesh T. Berg E. Learned-Miller "Labeled faces in the wild: A database for studying face recognition in unconstrained environments" University of Massachusetts Amherst Tech. Rep. pp. 07-49 October 2007.
Q. Cao L. Shen W. Xie O. M. Parkhi A. Zisserman "Vggface2: A dataset for recognising faces across pose and age" 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 20pp. 67-74 2018.
Z. Wang A. C. Bovik H. R. Sheikh E. P. Simoncelli et al. "Image quality assessment: from error visibility to structural similarity" IEEE transactions on image processing vol. 13 no. 4 pp. 600-2004.
J.-M. Cheng H.-C. Wang "A method of estimating the equal error rate for automatic speaker verification" 2004 International Symposium on Chinese Spoken Language Processing pp. 285-2004.
T. Fawcett "An introduction to roc analysis" Pattern recognition letters vol. 27 no. 8 pp. 861-874 2006.
K. Zhang Z. Zhang Z. Li Y. Qiao "Joint face detection and alignment using multitask cascaded convolutional networks" IEEE Signal Processing Letters vol. 23 no. 10 pp. 1499-12016.
D. P. Kingma J. Ba Adam: A method for stochastic optimization 2014.
Q. Huynh-Thu M. Ghanbari "Scope of validity of psnr in image/video quality assessment" Electronics letters vol. 44 no. 13 pp. 800-2008.
C. Ledig L. Theis F. Huszár J. Caballero A. Cunningham A. Acosta A. Aitken A. Tejani J. Totz Z. Wang et al. "Photo-realistic single image super-resolution using a generative adversarial network" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 4681-42017.
J. Johnson A. Alahi L. Fei-Fei "Perceptual losses for real-time style transfer and super-resolution" European conference on computer vision pp. 694-2016.
"Mean opinion score interpretation and reporting" ITU Recommendation pp. 800.2 2013.
Y. Chen Y. Tai X. Liu C. Shen J. Yang "Fsrnet: End-to-end learning face super-resolution with facial priors" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 2492-22018.
P. Rasti T. Uiboupin S. Escalera G. Anbarjafari "Convolutional neural network super resolution for face recognition in surveillance monitoring" International conference on articulated motion and deformable objects pp. 175-2016.
P. Li L. Prieto D. Mery P. J. Flynn "On low-resolution face recognition in the wild: Comparisons and new techniques" IEEE Transactions on Information Forensics and Security 2019.
P. J. Phillips P. Grother R. Micheals D. M. Blackburn E. Tabassi M. Bone "Face recognition vendor test 2002" 2003 IEEE International SOI Conference. Proceedings (Cat. No. 03CH374pp. 44 2003.
Y. Taigman M. Yang M. Ranzato L. Wolf "Deepface: Closing the gap to human-level performance in face verification" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 1701-12014.
O. M. Parkhi A. Vedaldi A. Zisserman et al. "Deep face recognition" bmvc vol. 1 no. 3 pp. 6 2015.
C. Dong C. C. Loy K. He X. Tang "Learning a deep convolutional network for image super-resolution" European conference on computer vision pp. 184-2014.
W. Shi J. Caballero F. Huszár J. Totz A. P. Aitken R. Bishop D. Rueckert Z. Wang "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 1874-1883 2016.
Z. Wang J. Chen S. C. H. Hoi "Deep learning for image super-resolution: A survey" CoRR vol. abs/1902.06068 2019.
H. Huang R. He Z. Sun T. Tan "Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution" Proceedings of the IEEE International Conference on Computer Vision pp. 1689-12017.
W. Liu Y. Wen Z. Yu M. Li B. Raj L. Song "Sphereface: Deep hypersphere embedding for face recognition" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 212-2017.
C. Szegedy V. Vanhoucke S. Ioffe J. Shlens Z. Wojna "Rethinking the inception architecture for computer vision" Proceedings of the IEEE conference on computer vision and pattern recognition pp. 2818-22016.
T. Salimans I. J. Goodfellow W. Zaremba V. Cheung A. Radford X. Chen "Improved techniques for training gans" CoRR vol. abs/1606.032016.
M. Heusel H. Ramsauer T. Unterthiner B. Nessler S. Hochreiter "Gans trained by a two time-scale update rule converge to a local nash equilibrium" Advances in Neural Information Processing Systems pp. 6626-62017.
D. Dai Y. Wang Y. Chen L. van Gool "Is image super-resolution helpful for other vision tasks?" 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) pp. 1-9 2016.
S. Hu R. Maschal S. S. Young T. H. Hong P. J. Phillips "Face recognition performance with superresolution" Applied optics vol. 51 no. 18 pp. 4250-42012.
A. Bulat G. Tzimiropoulos "Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 109-2018.
M. Haris G. Shakhnarovich N. Ukita Task-driven super resolution: Object detection in low-resolution images 2018.
K. Zhang Z. Zhang C.-W. Cheng W. H. Hsu Y. Qiao W. Liu T. Zhang "Super-identity convolutional neural network for face hallucination" Proceedings of the European Conference on Computer Vision (ECCV) pp. 183-2018.
K. He X. Zhang S. Ren J. Sun "Deep residual learning for image recognition" CoRR vol. abs/1512.032015.
Z. Liu P. Luo X. Wang X. Tang "Deep learning face attributes in the wild" Proceedings of International Conference on Computer Vision (ICCV) 2015.
G. B. Huang M. Ramesh T. Berg E. Learned-Miller "Labeled faces in the wild: A database for studying face recognition in unconstrained environments" University of Massachusetts Amherst Tech. Rep. pp. 07-49 October 2007.
Q. Cao L. Shen W. Xie O. M. Parkhi A. Zisserman "Vggface2: A dataset for recognising faces across pose and age" 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 20pp. 67-74 2018.
Z. Wang A. C. Bovik H. R. Sheikh E. P. Simoncelli et al. "Image quality assessment: from error visibility to structural similarity" IEEE transactions on image processing vol. 13 no. 4 pp. 600-2004.
J.-M. Cheng H.-C. Wang "A method of estimating the equal error rate for automatic speaker verification" 2004 International Symposium on Chinese Spoken Language Processing pp. 285-2004.
T. Fawcett "An introduction to roc analysis" Pattern recognition letters vol. 27 no. 8 pp. 861-874 2006.
K. Zhang Z. Zhang Z. Li Y. Qiao "Joint face detection and alignment using multitask cascaded convolutional networks" IEEE Signal Processing Letters vol. 23 no. 10 pp. 1499-12016.
D. P. Kingma J. Ba Adam: A method for stochastic optimization 2014.
Publicado
28/10/2019
Como Citar
ABELLO, Antonio A. ; HIRATA JUNIOR, Roberto.
Optimizing Super Resolution for Face Recognition. In: CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 32. , 2019, Rio de Janeiro.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
DOI: https://doi.org/10.5753/sibgrapi.2019.9809.
