Super-Resolution Towards License Plate Recognition
Resumo
Recent years have seen significant developments in license plate recognition through the integration of deep learning techniques and the increasing availability of training data. Nevertheless, reconstructing license plates from low-resolution surveillance footage remains challenging. To address this issue, we propose an attention-based super-resolution approach that incorporates sub-pixel convolution layers and an Optical Character Recognition (OCR)-based loss function. We trained the proposed architecture on synthetic images created by applying heavy Gaussian noise followed by bicubic downsampling to high-resolution license plate images. Our results show that the proposed approach for reconstructing these low-resolution images substantially outperforms existing methods in both quantitative and qualitative measures. Our source code is publicly available at https://github.com/valfride/lpr-rsr-ext/.
Referências
V. Nascimento, “Super-resolution towards license plate recognition,” Master’s thesis, Federal University of Paraná (UFPR), 2023.
L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang, and L. Zhang, “Image superresolution: The techniques, applications, and future,” Signal Processing, vol. 128, pp. 389–408, 2016.
A. Liu, Y. Liu, J. Gu, Y. Qiao, and C. Dong, “Blind image superresolution: A survey and beyond,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 5461–5480, 2023.
Z. Wang, J. Chen, and S. C. H. Hoi, “Deep learning for image superresolution: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 10, pp. 3365–3387, 2021.
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2480–2495, 2021.
M. Santos et al., “Face super-resolution using stochastic differential equations,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2022, pp. 216–221.
G. R. Gonçalves et al., “Multi-task learning for low-resolution license plate recognition,” in Iberoamerican Congress on Pattern Recognition (CIARP), Oct 2019, pp. 251–261.
A. Maier et al., “Reliability scoring for the recognition of degraded license plates,” in IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2022, pp. 1–8.
D. Moussa et al., “Forensic license plate recognition with compressioninformed transformers,” in IEEE International Conference on Image Processing (ICIP), 2022, pp. 406–410.
R. Laroca, L. A. Zanlorensi, G. R. Gonçalves, E. Todt, W. R. Schwartz, and D. Menotti, “An efficient and layout-independent automatic license plate recognition system based on the YOLO detector,” IET Intelligent Transport Systems, vol. 15, no. 4, pp. 483–503, 2021.
Y. Gong et al., “Unified Chinese license plate detection and recognition with high efficiency,” Journal of Visual Communication and Image Representation, vol. 86, p. 103541, 2022.
S. M. Silva and C. R. Jung, “A flexible approach for automatic license plate recognition in unconstrained scenarios,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 5693–5703, 2022.
Y. Wang et al., “Rethinking and designing a high-performing automatic license plate recognition approach,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 8868–8880, 2022.
R. Laroca, L. A. Zanlorensi, V. Estevam, R. Minetto, and D. Menotti, “Leveraging model fusion for improved license plate recognition,” in Iberoamerican Congress on Pattern Recognition, Nov 2023, pp. 1–15.
A. Lucas et al., “Generative adversarial networks and perceptual losses for video super-resolution,” IEEE Transactions on Image Processing, vol. 28, no. 7, pp. 3312–3327, 2019.
A. Mehri, P. B. Ardakani, and A. D. Sappa, “MPRNet: Multi-path residual network for lightweight image super resolution,” in IEEE Winter Conference on Applications of Computer Vision, 2021, pp. 2703–2712.
G. R. Gonçalves et al., “Real-time automatic license plate recognition through deep multi-task networks,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2018, pp. 110–117.
C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016.
Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep networks for image super-resolution with sparse prior,” in IEEE International Conference on Computer Vision (ICCV), 2015, pp. 370–378.
Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration,” IEEE Trans. on Pattern Analysis and Machine Intel., vol. 39, pp. 1256–1272, 2017.
C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European Conference on Computer Vision (ECCV), 2016, pp. 391–407.
W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1874–1883.
Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image superresolution using very deep residual channel attention networks,” in European Conference on Computer Vision (ECCV), 2018, pp. 294–310.
T. Dai et al., “Second-order attention network for single image superresolution,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11 057–11 066.
Y. Huang, J. Li, X. Gao, Y. Hu, and W. Lu, “Interpretable detail-fidelity attention network for single image super-resolution,” IEEE Transactions on Image Processing, vol. 30, pp. 2325–2339, 2021.
X. He et al., “Ode-inspired network design for single image superresolution,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019, pp. 1732–1741.
X. Luo, Y. Xie, Y. Zhang, Y. Qu, C. Li, and Y. Fu, “LatticeNet: Towards lightweight image super-resolution with lattice block,” in European Conference on Computer Vision (ECCV), 2020, pp. 272–289.
A. Muqeet, J. Hwang, S. Yang, J. Kang, Y. Kim, and S.-H. Bae, “Multiattention based ultra lightweight image super-resolution,” in European Conference on Computer Vision Workshops, 2020, pp. 103–118.
Y. Zhang, Y. Huang, K. Wang, G. Qi, and J. Zhu, “Single image super-resolution reconstruction with preservation of structure and texture details,” Mathematics, vol. 11, p. 216, 01 2023.
R. Laroca, E. V. Cardoso, D. R. Lucio, V. Estevam, and D. Menotti, “On the cross-dataset generalization in license plate recognition,” in International Conference on Computer Vision Theory and Applications (VISAPP), Feb 2022, pp. 166–178.
C.-K. Liang, L.-W. Chang, and H. H. Chen, “Analysis and compensation of rolling shutter effect,” IEEE Transactions on Image Processing, vol. 17, no. 8, pp. 1323–1330, 2008.
K. V. Suresh, G. M. Kumar, and A. N. Rajagopalan, “Superresolution of license plates in real traffic videos,” IEEE Transactions on Intelligent Transportation Systems, vol. 8, no. 2, pp. 321–331, 2007.
J. Yuan, S.-D. Du, and X. Zhu, “Fast super-resolution for license plate image reconstruction,” in International Conference on Pattern Recognition (ICPR), 2008, pp. 1–4.
M. Lin, L. Liu, F. Wang, J. Li, and J. Pan, “License plate image reconstruction based on generative adversarial networks,” Remote Sensing, vol. 13, no. 15, p. 3018, 2021.
A. Hamdi, Y. K. Chan, and V. C. Koo, “A new image enhancement and super resolution technique for license plate recognition,” Heliyon, vol. 7, no. 11, p. e08341, 2021.
S. Lee, J.-H. Kim, and J.-P. Heo, “Super-resolution of license plate images via character-based perceptual loss,” in IEEE International Conference on Big Data and Smart Computing, 2020, pp. 560–563.
J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for realtime style transfer and super-resolution,” in European Conference on Computer Vision (ECCV), 2016, pp. 694–711.
R. Zhang et al., “The unreasonable effectiveness of deep features as a perceptual metric,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 586–595.
V. Nascimento, R. Laroca, J. A. Lambert, W. R. Schwartz, and D. Menotti, “Combining attention module and pixel shuffle for license plate super-resolution,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2022, pp. 228–233.
Y. Yuan, W. Zou, Y. Zhao, X. Wang, X. Hu, and N. Komodakis, “A robust and efficient approach to license plate detection,” IEEE Transactions on Image Processing, vol. 26, no. 3, pp. 1102–1114, 2017.
R. Laroca, M. Santos, V. Estevam, E. Luz, and D. Menotti, “A first look at dataset bias in license plate recognition,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2022, pp. 234–239.
L. Zhang et al., “A robust attentional framework for license plate recognition in the wild,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 11, pp. 6967–6976, 2021.
R. Laroca et al., “Do we train on test data? The impact of near-duplicates on license plate recognition,” in International Joint Conference on Neural Networks (IJCNN), June 2023, pp. 1–8.
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 2472–2481.
W. Li, L. Fan, Z. Wang, C. Ma, and X. Cui, “Tackling mode collapse in multi-generator GANs with orthogonal vectors,” Pattern Recognition, vol. 110, p. 107646, 2021.
C. Saharia, J. Ho, W. Chan, T. Salimans, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4713–4726, 2023.
S. Qin and S. Liu, “Towards end-to-end car license plate location and recognition in unconstrained scenarios,” Neural Computing and Applications, vol. 34, p. 21551–21566, 2022.
V. Nascimento et al., “Super-resolution of license plate images using attention modules and sub-pixel convolution layers,” Computers & Graphics, vol. 113, pp. 69–76, 2023.
L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang, and L. Zhang, “Image superresolution: The techniques, applications, and future,” Signal Processing, vol. 128, pp. 389–408, 2016.
A. Liu, Y. Liu, J. Gu, Y. Qiao, and C. Dong, “Blind image superresolution: A survey and beyond,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 5461–5480, 2023.
Z. Wang, J. Chen, and S. C. H. Hoi, “Deep learning for image superresolution: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 10, pp. 3365–3387, 2021.
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2480–2495, 2021.
M. Santos et al., “Face super-resolution using stochastic differential equations,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2022, pp. 216–221.
G. R. Gonçalves et al., “Multi-task learning for low-resolution license plate recognition,” in Iberoamerican Congress on Pattern Recognition (CIARP), Oct 2019, pp. 251–261.
A. Maier et al., “Reliability scoring for the recognition of degraded license plates,” in IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2022, pp. 1–8.
D. Moussa et al., “Forensic license plate recognition with compressioninformed transformers,” in IEEE International Conference on Image Processing (ICIP), 2022, pp. 406–410.
R. Laroca, L. A. Zanlorensi, G. R. Gonçalves, E. Todt, W. R. Schwartz, and D. Menotti, “An efficient and layout-independent automatic license plate recognition system based on the YOLO detector,” IET Intelligent Transport Systems, vol. 15, no. 4, pp. 483–503, 2021.
Y. Gong et al., “Unified Chinese license plate detection and recognition with high efficiency,” Journal of Visual Communication and Image Representation, vol. 86, p. 103541, 2022.
S. M. Silva and C. R. Jung, “A flexible approach for automatic license plate recognition in unconstrained scenarios,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 5693–5703, 2022.
Y. Wang et al., “Rethinking and designing a high-performing automatic license plate recognition approach,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 8868–8880, 2022.
R. Laroca, L. A. Zanlorensi, V. Estevam, R. Minetto, and D. Menotti, “Leveraging model fusion for improved license plate recognition,” in Iberoamerican Congress on Pattern Recognition, Nov 2023, pp. 1–15.
A. Lucas et al., “Generative adversarial networks and perceptual losses for video super-resolution,” IEEE Transactions on Image Processing, vol. 28, no. 7, pp. 3312–3327, 2019.
A. Mehri, P. B. Ardakani, and A. D. Sappa, “MPRNet: Multi-path residual network for lightweight image super resolution,” in IEEE Winter Conference on Applications of Computer Vision, 2021, pp. 2703–2712.
G. R. Gonçalves et al., “Real-time automatic license plate recognition through deep multi-task networks,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2018, pp. 110–117.
C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016.
Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep networks for image super-resolution with sparse prior,” in IEEE International Conference on Computer Vision (ICCV), 2015, pp. 370–378.
Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration,” IEEE Trans. on Pattern Analysis and Machine Intel., vol. 39, pp. 1256–1272, 2017.
C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in European Conference on Computer Vision (ECCV), 2016, pp. 391–407.
W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1874–1883.
Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image superresolution using very deep residual channel attention networks,” in European Conference on Computer Vision (ECCV), 2018, pp. 294–310.
T. Dai et al., “Second-order attention network for single image superresolution,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11 057–11 066.
Y. Huang, J. Li, X. Gao, Y. Hu, and W. Lu, “Interpretable detail-fidelity attention network for single image super-resolution,” IEEE Transactions on Image Processing, vol. 30, pp. 2325–2339, 2021.
X. He et al., “Ode-inspired network design for single image superresolution,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019, pp. 1732–1741.
X. Luo, Y. Xie, Y. Zhang, Y. Qu, C. Li, and Y. Fu, “LatticeNet: Towards lightweight image super-resolution with lattice block,” in European Conference on Computer Vision (ECCV), 2020, pp. 272–289.
A. Muqeet, J. Hwang, S. Yang, J. Kang, Y. Kim, and S.-H. Bae, “Multiattention based ultra lightweight image super-resolution,” in European Conference on Computer Vision Workshops, 2020, pp. 103–118.
Y. Zhang, Y. Huang, K. Wang, G. Qi, and J. Zhu, “Single image super-resolution reconstruction with preservation of structure and texture details,” Mathematics, vol. 11, p. 216, 01 2023.
R. Laroca, E. V. Cardoso, D. R. Lucio, V. Estevam, and D. Menotti, “On the cross-dataset generalization in license plate recognition,” in International Conference on Computer Vision Theory and Applications (VISAPP), Feb 2022, pp. 166–178.
C.-K. Liang, L.-W. Chang, and H. H. Chen, “Analysis and compensation of rolling shutter effect,” IEEE Transactions on Image Processing, vol. 17, no. 8, pp. 1323–1330, 2008.
K. V. Suresh, G. M. Kumar, and A. N. Rajagopalan, “Superresolution of license plates in real traffic videos,” IEEE Transactions on Intelligent Transportation Systems, vol. 8, no. 2, pp. 321–331, 2007.
J. Yuan, S.-D. Du, and X. Zhu, “Fast super-resolution for license plate image reconstruction,” in International Conference on Pattern Recognition (ICPR), 2008, pp. 1–4.
M. Lin, L. Liu, F. Wang, J. Li, and J. Pan, “License plate image reconstruction based on generative adversarial networks,” Remote Sensing, vol. 13, no. 15, p. 3018, 2021.
A. Hamdi, Y. K. Chan, and V. C. Koo, “A new image enhancement and super resolution technique for license plate recognition,” Heliyon, vol. 7, no. 11, p. e08341, 2021.
S. Lee, J.-H. Kim, and J.-P. Heo, “Super-resolution of license plate images via character-based perceptual loss,” in IEEE International Conference on Big Data and Smart Computing, 2020, pp. 560–563.
J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for realtime style transfer and super-resolution,” in European Conference on Computer Vision (ECCV), 2016, pp. 694–711.
R. Zhang et al., “The unreasonable effectiveness of deep features as a perceptual metric,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 586–595.
V. Nascimento, R. Laroca, J. A. Lambert, W. R. Schwartz, and D. Menotti, “Combining attention module and pixel shuffle for license plate super-resolution,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2022, pp. 228–233.
Y. Yuan, W. Zou, Y. Zhao, X. Wang, X. Hu, and N. Komodakis, “A robust and efficient approach to license plate detection,” IEEE Transactions on Image Processing, vol. 26, no. 3, pp. 1102–1114, 2017.
R. Laroca, M. Santos, V. Estevam, E. Luz, and D. Menotti, “A first look at dataset bias in license plate recognition,” in Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2022, pp. 234–239.
L. Zhang et al., “A robust attentional framework for license plate recognition in the wild,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 11, pp. 6967–6976, 2021.
R. Laroca et al., “Do we train on test data? The impact of near-duplicates on license plate recognition,” in International Joint Conference on Neural Networks (IJCNN), June 2023, pp. 1–8.
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 2472–2481.
W. Li, L. Fan, Z. Wang, C. Ma, and X. Cui, “Tackling mode collapse in multi-generator GANs with orthogonal vectors,” Pattern Recognition, vol. 110, p. 107646, 2021.
C. Saharia, J. Ho, W. Chan, T. Salimans, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4713–4726, 2023.
S. Qin and S. Liu, “Towards end-to-end car license plate location and recognition in unconstrained scenarios,” Neural Computing and Applications, vol. 34, p. 21551–21566, 2022.
V. Nascimento et al., “Super-resolution of license plate images using attention modules and sub-pixel convolution layers,” Computers & Graphics, vol. 113, pp. 69–76, 2023.
Publicado
06/11/2023
Como Citar
NASCIMENTO, Valfride; LAROCA, Rayson; MENOTTI, David.
Super-Resolution Towards License Plate Recognition. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 36. , 2023, Rio Grande/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 28-34.
DOI: https://doi.org/10.5753/sibgrapi.est.2023.27448.
