Automatic License Plate Recognition: An Efficient and Layout-Independent System Based on the YOLO Detector
Resumo
Automatic License Plate Recognition (ALPR) has been a frequent topic of research due to many practical applications, such as border control and traffic law enforcement. This work presents an efficient, robust and layout-independent ALPR system based on the YOLO object detector that contains a unified approach for license plate detection and layout classification and that leverages post-processing rules in the recognition stage to eliminate a major shortcoming of existing ALPR systems (being layout dependent). We also introduce a publicly available dataset for ALPR, called UFPR-ALPR, that has become very popular, having been downloaded more than 650 times by researchers from 80 different countries over the past two years. The proposed system, which performs in real time even when there are 4 vehicles in the scene, outperformed both previous works and commercial systems on four public datasets widely used in the literature. The entire ALPR system (i.e., the architectures and weights), along with all annotations made by us are publicly available at https://web.inf.ufpr.br/vri/publications/layout-independent-alpr/.Referências
R. Laroca, "An efficient and layout-independent automatic license plate recognition system based on the YOLO detector," Master’s thesis, Federal University of Paraná (UFPR), 2019.
R. A. Lotufo, A. D. Morgan, and A. S. Johnson, "Automatic number- plate recognition," in IEE Colloquium on Image Analysis for Transport Applications, 1990, pp. 1–6.
K. Kanayama, Y. Fujikawa, K. Fujimoto, and M. Horino, "Development of vehicle-license number recognition system using real-time image processing and its application to travel-time measurement," in IEEE Vehicular Technology Conference, 1991, pp. 798–804.
C. N. E. Anagnostopoulos, I. E. Anagnostopoulos, I. D. Psoroulas, V. Loumos, and E. Kayafas, "License plate recognition from still images and video sequences: A survey," IEEE Transactions on Intelligent Transportation Systems, vol. 9, no. 3, pp. 377–391, 2008.
S. Du, M. Ibrahim, M. Shehata, and W. Badawy, "Automatic license plate recognition (ALPR): A state-of-the-art review," IEEE Trans. on Circuits and Systems for Video Technology, vol. 23, pp. 311–325, 2013.
H. Li, P. Wang, M. You, and C. Shen, "Reading car license plates using deep neural networks," Image and Vision Computing, vol. 72, pp. 14–23, 2018.
H. Li, P. Wang, and C. Shen, "Toward end-to-end car license plate detection and recognition with deep neural networks," IEEE Trans. on Intelligent Transportation Systems, vol. 20, no. 3, pp. 1126–1136, 2019.
W. Weihong and T. Jiaoyang, "Research on license plate recognition algorithms based on deep learning in complex environment," IEEE Access, vol. 8, pp. 91 661–91 675, 2020.
G. R. Gonc¸alves, S. P. G. da Silva, D. Menotti, and W. R. Schwartz, "Benchmark for license plate character segmentation," Journal of Electronic Imaging, vol. 25, no. 5, p. 053034, 2016.
Y. Kessentini et al., "A two-stage deep neural network for multi- norm license plate detection and recognition," Expert Systems with Applications, vol. 136, pp. 159–170, 2019.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788.
G. R. Gonçalves, M. A. Diniz, R. Laroca, D. Menotti, and W. R. Schwartz, "Real-time automatic license plate recognition through deep multi-task networks," in Conference on Graphics, Patterns and Im- ages (SIBGRAPI), 2018, pp. 110–117.
R. Laroca, E. Severo, L. A. Zanlorensi, L. S. Oliveira, G. R. Gonc¸alves, W. R. Schwartz, and D. Menotti, "A robust real-time automatic license plate recognition based on the YOLO detector," in International Joint Conference on Neural Networks (IJCNN), 2018, pp. 1–10.
M. Dong, D. He, C. Luo, D. Liu, and W. Zeng, "A CNN-based approach for automatic license plate recognition in the wild," in British Machine Vision Conference (BMVC), 2017, pp. 1–12.
J. Redmon and A. Farhadi, "YOLO9000: Better, faster, stronger," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6517–6525.
——, "YOLOv3: An incremental improvement," arXiv preprint, 2018. [Online]. Available: http://arxiv.org/abs/1804.02767
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The pascal visual object classes (VOC) challenge," International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010.
T.-Y. Lin et al., "Microsoft COCO: Common objects in context," in European Conference on Computer Vision, 2014, pp. 740–755.
F. D. Kurpiel, R. Minetto, and B. T. Nassu, "Convolutional neural networks for license plate detection in images," in IEEE International Conference on Image Processing (ICIP), 2017, pp. 3395–3399.
M. S. Al-Shemarry et al., "Ensemble of adaboost cascades of 3L-LBPs classifiers for license plates detection with low quality images," Expert Systems with Applications, vol. 92, pp. 216–235, 2018.
G. S. Hsu, A. Ambikapathi, S. L. Chung, and C. P. Su, "Robust license plate detection in the wild," in IEEE International Conference on Advanced Video and Signal Based Surveillance, 2017, pp. 1–6.
L. Xie, T. Ahmad, L. Jin, Y. Liu, and S. Zhang, "A new CNN-based method for multi-directional car license plate detection," IEEE Trans. on Intelligent Transportation Systems, vol. 19, pp. 507–517, 2018.
D. Menotti, G. Chiachia, A. X. Falcao, and V. J. O. Neto, "Vehicle license plate recognition with random convolutional networks," in Conference on Graphics, Patterns and Images, 2014, pp. 298–303.
Y. Yang, D. Li, and Z. Duan, "Chinese vehicle license plate recognition using kernel-based extreme learning machine with deep convolutional features," IET Intelligent Transport Systems, vol. 12, pp. 213–219, 2018.
J. Zhuang, S. Hou, Z. Wang, and Z. Zha, "Towards human-level license plate recognition," in European Conference on Computer Vision (ECCV), 2018, pp. 314–329.
O. Bulan, V. Kozitsky, P. Ramesh, and M. Shreve, "Segmentation- and annotation-free license plate recognition with deep localization and failure identification," IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 9, pp. 2351–2363, 2017.
J. Wang, H. Huang, X. Qian, J. Cao, and Y. Dai, "Sequence recognition of chinese license plates," Neurocomputing, vol. 317, pp. 149–158, 2018.
C. Liu and F. Chang, "Hybrid cascade structure for license plate detection in large visual surveillance scenes," IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 6, pp. 2122–2135, 2019.
S. M. Silva and C. R. Jung, "Real-time license plate detection and recognition using deep convolutional neural networks," Journal of Visual Communication and Image Representation, p. 102773, 2020.
Z. Selmi et al., "Deep learning system for automatic license plate detection and recognition," in IAPR International Conference on Document Analysis and Recognition (ICDAR), 2017, pp. 1132–1138.
W. Liu et al., "SSD: Single shot multibox detector," in European Conference on Computer Vision (ECCV), 2016, pp. 21–37.
T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2999–3007.
G. R. Gonc¸alves, D. Menotti, and W. R. Schwartz, "License plate recognition based on temporal redundancy," in IEEE International Conference on Intelligent Transportation Systems, 2016, pp. 2577–2582.
S. M. Silva and C. R. Jung, "Real-time brazilian license plate detection and recognition using deep convolutional neural networks," in Confer- ence on Graphics, Patterns and Images (SIBGRAPI), 2017, pp. 55–62.
R. Laroca, V. Barroso, M. A. Diniz, G. R. Gonçalves, W. R. Schwartz, and D. Menotti, "Convolutional neural networks for automatic meter reading," Journal of Electronic Imaging, vol. 28, no. 1, p. 013023, 2019.
M. Weber, "Caltech Cars dataset," http://www.vision.caltech.edu/Image Datasets/cars markus/cars markus.tar, 1999, Accessed: July 3, 2020.
V. Srebríc, "EnglishLP database," http://www.zemris.fer.hr/projects/ LicensePlates/english/baza slika.zip, 2003, Accessed: July 3, 2020.
L. Dlagnekov and S. Belongie, "UCSD/Calit2 car license plate, make and model database," http://vision.ucsd.edu/belongie-grp/research/ carRec/car data.html, 2005, Accessed: July 3, 2020.
W. Zhou, H. Li, Y. Lu, and Q. Tian, "Principal visual word discovery for automatic license plate detection," IEEE Transactions on Image Processing, vol. 21, no. 9, pp. 4269–4279, 2012.
G. S. Hsu, J. C. Chen, and Y. Z. Chung, "Application-oriented license plate recognition," IEEE Transactions on Vehicular Technology, vol. 62, no. 2, pp. 552–561, 2013.
OpenALPR Inc., "OpenALPR-EU dataset," https://github.com/openalpr/ benchmarks/tree/master/endtoend/eu, 2016, Accessed: July 3, 2020.
R. Laroca, L. A. Zanlorensi, G. R. Gonçalves, E. Todt, W. R. Schwartz, and D. Menotti, "An efficient and layout-independent automatic license plate recognition system based on the YOLO detector," arXiv preprint, vol. arXiv:1909.01754, 2019, submitted to IET Intelligent Transport Systems (provisionally accepted subject to major revisions).
R. Panahi and I. Gholampour, "Accurate detection and recognition of dirty vehicle plate numbers for high-speed applications," IEEE Trans. on Intelligent Transportation Systems, vol. 18, no. 4, pp. 767–779, 2017.
S. M. Silva and C. R. Jung, "License plate detection and recognition in unconstrained scenarios," in European Conference on Computer Vision (ECCV), 2018, pp. 593–609.
S. Z. Masood, G. Shu, A. Dehghan, and E. G. Ortiz, "License plate detection and recognition using deeply learned convolutional neural networks," arXiv preprint, vol. arXiv:1703.07330, 2017.
OpenALPR Inc., "OpenALPR Cloud API," https://www.openalpr.com/ carcheck-api.html, 2020, Accessed: July 3, 2020.
G. R. Gonçalves, M. A. Diniz, R. Laroca, D. Menotti, and W. R. Schwartz, "Multi-task learning for low-resolution license plate recog- nition," in Iberoamerican Congress on Pattern Recognition (CIARP), 2019, pp. 251–261.
I. O. Oliveira, R. Laroca, D. Menotti, K. V. O. Fonseca, and R. Minetto, "Vehicle re-identification: exploring feature fusion using multi-stream convolutional networks," arXiv preprint, vol. arXiv:1911.05541, 2019, submitted to IEEE Transactions on Intelligent Transportation Systems.
R. A. Lotufo, A. D. Morgan, and A. S. Johnson, "Automatic number- plate recognition," in IEE Colloquium on Image Analysis for Transport Applications, 1990, pp. 1–6.
K. Kanayama, Y. Fujikawa, K. Fujimoto, and M. Horino, "Development of vehicle-license number recognition system using real-time image processing and its application to travel-time measurement," in IEEE Vehicular Technology Conference, 1991, pp. 798–804.
C. N. E. Anagnostopoulos, I. E. Anagnostopoulos, I. D. Psoroulas, V. Loumos, and E. Kayafas, "License plate recognition from still images and video sequences: A survey," IEEE Transactions on Intelligent Transportation Systems, vol. 9, no. 3, pp. 377–391, 2008.
S. Du, M. Ibrahim, M. Shehata, and W. Badawy, "Automatic license plate recognition (ALPR): A state-of-the-art review," IEEE Trans. on Circuits and Systems for Video Technology, vol. 23, pp. 311–325, 2013.
H. Li, P. Wang, M. You, and C. Shen, "Reading car license plates using deep neural networks," Image and Vision Computing, vol. 72, pp. 14–23, 2018.
H. Li, P. Wang, and C. Shen, "Toward end-to-end car license plate detection and recognition with deep neural networks," IEEE Trans. on Intelligent Transportation Systems, vol. 20, no. 3, pp. 1126–1136, 2019.
W. Weihong and T. Jiaoyang, "Research on license plate recognition algorithms based on deep learning in complex environment," IEEE Access, vol. 8, pp. 91 661–91 675, 2020.
G. R. Gonc¸alves, S. P. G. da Silva, D. Menotti, and W. R. Schwartz, "Benchmark for license plate character segmentation," Journal of Electronic Imaging, vol. 25, no. 5, p. 053034, 2016.
Y. Kessentini et al., "A two-stage deep neural network for multi- norm license plate detection and recognition," Expert Systems with Applications, vol. 136, pp. 159–170, 2019.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788.
G. R. Gonçalves, M. A. Diniz, R. Laroca, D. Menotti, and W. R. Schwartz, "Real-time automatic license plate recognition through deep multi-task networks," in Conference on Graphics, Patterns and Im- ages (SIBGRAPI), 2018, pp. 110–117.
R. Laroca, E. Severo, L. A. Zanlorensi, L. S. Oliveira, G. R. Gonc¸alves, W. R. Schwartz, and D. Menotti, "A robust real-time automatic license plate recognition based on the YOLO detector," in International Joint Conference on Neural Networks (IJCNN), 2018, pp. 1–10.
M. Dong, D. He, C. Luo, D. Liu, and W. Zeng, "A CNN-based approach for automatic license plate recognition in the wild," in British Machine Vision Conference (BMVC), 2017, pp. 1–12.
J. Redmon and A. Farhadi, "YOLO9000: Better, faster, stronger," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6517–6525.
——, "YOLOv3: An incremental improvement," arXiv preprint, 2018. [Online]. Available: http://arxiv.org/abs/1804.02767
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The pascal visual object classes (VOC) challenge," International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010.
T.-Y. Lin et al., "Microsoft COCO: Common objects in context," in European Conference on Computer Vision, 2014, pp. 740–755.
F. D. Kurpiel, R. Minetto, and B. T. Nassu, "Convolutional neural networks for license plate detection in images," in IEEE International Conference on Image Processing (ICIP), 2017, pp. 3395–3399.
M. S. Al-Shemarry et al., "Ensemble of adaboost cascades of 3L-LBPs classifiers for license plates detection with low quality images," Expert Systems with Applications, vol. 92, pp. 216–235, 2018.
G. S. Hsu, A. Ambikapathi, S. L. Chung, and C. P. Su, "Robust license plate detection in the wild," in IEEE International Conference on Advanced Video and Signal Based Surveillance, 2017, pp. 1–6.
L. Xie, T. Ahmad, L. Jin, Y. Liu, and S. Zhang, "A new CNN-based method for multi-directional car license plate detection," IEEE Trans. on Intelligent Transportation Systems, vol. 19, pp. 507–517, 2018.
D. Menotti, G. Chiachia, A. X. Falcao, and V. J. O. Neto, "Vehicle license plate recognition with random convolutional networks," in Conference on Graphics, Patterns and Images, 2014, pp. 298–303.
Y. Yang, D. Li, and Z. Duan, "Chinese vehicle license plate recognition using kernel-based extreme learning machine with deep convolutional features," IET Intelligent Transport Systems, vol. 12, pp. 213–219, 2018.
J. Zhuang, S. Hou, Z. Wang, and Z. Zha, "Towards human-level license plate recognition," in European Conference on Computer Vision (ECCV), 2018, pp. 314–329.
O. Bulan, V. Kozitsky, P. Ramesh, and M. Shreve, "Segmentation- and annotation-free license plate recognition with deep localization and failure identification," IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 9, pp. 2351–2363, 2017.
J. Wang, H. Huang, X. Qian, J. Cao, and Y. Dai, "Sequence recognition of chinese license plates," Neurocomputing, vol. 317, pp. 149–158, 2018.
C. Liu and F. Chang, "Hybrid cascade structure for license plate detection in large visual surveillance scenes," IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 6, pp. 2122–2135, 2019.
S. M. Silva and C. R. Jung, "Real-time license plate detection and recognition using deep convolutional neural networks," Journal of Visual Communication and Image Representation, p. 102773, 2020.
Z. Selmi et al., "Deep learning system for automatic license plate detection and recognition," in IAPR International Conference on Document Analysis and Recognition (ICDAR), 2017, pp. 1132–1138.
W. Liu et al., "SSD: Single shot multibox detector," in European Conference on Computer Vision (ECCV), 2016, pp. 21–37.
T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2999–3007.
G. R. Gonc¸alves, D. Menotti, and W. R. Schwartz, "License plate recognition based on temporal redundancy," in IEEE International Conference on Intelligent Transportation Systems, 2016, pp. 2577–2582.
S. M. Silva and C. R. Jung, "Real-time brazilian license plate detection and recognition using deep convolutional neural networks," in Confer- ence on Graphics, Patterns and Images (SIBGRAPI), 2017, pp. 55–62.
R. Laroca, V. Barroso, M. A. Diniz, G. R. Gonçalves, W. R. Schwartz, and D. Menotti, "Convolutional neural networks for automatic meter reading," Journal of Electronic Imaging, vol. 28, no. 1, p. 013023, 2019.
M. Weber, "Caltech Cars dataset," http://www.vision.caltech.edu/Image Datasets/cars markus/cars markus.tar, 1999, Accessed: July 3, 2020.
V. Srebríc, "EnglishLP database," http://www.zemris.fer.hr/projects/ LicensePlates/english/baza slika.zip, 2003, Accessed: July 3, 2020.
L. Dlagnekov and S. Belongie, "UCSD/Calit2 car license plate, make and model database," http://vision.ucsd.edu/belongie-grp/research/ carRec/car data.html, 2005, Accessed: July 3, 2020.
W. Zhou, H. Li, Y. Lu, and Q. Tian, "Principal visual word discovery for automatic license plate detection," IEEE Transactions on Image Processing, vol. 21, no. 9, pp. 4269–4279, 2012.
G. S. Hsu, J. C. Chen, and Y. Z. Chung, "Application-oriented license plate recognition," IEEE Transactions on Vehicular Technology, vol. 62, no. 2, pp. 552–561, 2013.
OpenALPR Inc., "OpenALPR-EU dataset," https://github.com/openalpr/ benchmarks/tree/master/endtoend/eu, 2016, Accessed: July 3, 2020.
R. Laroca, L. A. Zanlorensi, G. R. Gonçalves, E. Todt, W. R. Schwartz, and D. Menotti, "An efficient and layout-independent automatic license plate recognition system based on the YOLO detector," arXiv preprint, vol. arXiv:1909.01754, 2019, submitted to IET Intelligent Transport Systems (provisionally accepted subject to major revisions).
R. Panahi and I. Gholampour, "Accurate detection and recognition of dirty vehicle plate numbers for high-speed applications," IEEE Trans. on Intelligent Transportation Systems, vol. 18, no. 4, pp. 767–779, 2017.
S. M. Silva and C. R. Jung, "License plate detection and recognition in unconstrained scenarios," in European Conference on Computer Vision (ECCV), 2018, pp. 593–609.
S. Z. Masood, G. Shu, A. Dehghan, and E. G. Ortiz, "License plate detection and recognition using deeply learned convolutional neural networks," arXiv preprint, vol. arXiv:1703.07330, 2017.
OpenALPR Inc., "OpenALPR Cloud API," https://www.openalpr.com/ carcheck-api.html, 2020, Accessed: July 3, 2020.
G. R. Gonçalves, M. A. Diniz, R. Laroca, D. Menotti, and W. R. Schwartz, "Multi-task learning for low-resolution license plate recog- nition," in Iberoamerican Congress on Pattern Recognition (CIARP), 2019, pp. 251–261.
I. O. Oliveira, R. Laroca, D. Menotti, K. V. O. Fonseca, and R. Minetto, "Vehicle re-identification: exploring feature fusion using multi-stream convolutional networks," arXiv preprint, vol. arXiv:1911.05541, 2019, submitted to IEEE Transactions on Intelligent Transportation Systems.
Publicado
07/11/2020
Como Citar
LAROCA, Rayson; MENOTTI, David.
Automatic License Plate Recognition: An Efficient and Layout-Independent System Based on the YOLO Detector. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 33. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 15-21.
DOI: https://doi.org/10.5753/sibgrapi.est.2020.12978.
