Brazilian Mercosur License Plate Detection: a Deep Learning Approach Relying on Synthetic Imagery
Resumo
Automated license plate recognition (ALPR) technology is a powerful technology enabling more efficient and effective law enforcement, security, payment collection, and research. A common license plate standard was adopted by the member states of the Mercosur trading bloc (Argentina, Brazil, Paraguay and Uruguay) and consequently requires an upgrade to the ALPR software used by law enforcement and industry. Due to the scarcity of real license plate images, training state-of-the-art supervised detectors is unfeasible unless data augmentation techniques and synthetic training data are used. This paper presents an accurate and efficient automated Mercosur license plate detector using a Convolutional Neural Network (CNN) trained exclusively with synthetic imagery. In order to obtain the synthetic training data, Mercosur license plates were faithfully reproduced. Digital image processing techniques were employed to reduce the domain gap and a CNN with basic image manipulation was used to embed the artificial licensed plates in to realistic contexts. The trained model was then validated on real images captured from a parking lot and a publicly available traffic monitoring video stream. The results of experiments suggest detection accuracy of about 95% and an average running time of 40 milliseconds.
Palavras-chave:
Deep learning, Applications
Referências
E. Barron A. Slomowitz "Massachusetts Should Facilitate – Not Inhibit – Law Enforcement Use of License Plate Data" Tech. Rep. 2013 [online] Available: https://netchoice.org/wp-content/uploads/MA-White-Paper.pdf.
C. S. Koper B. G. Taylor D. J. Woods "A randomized test of initial and residual deterrence from directed patrols and use of license plate readers at crime hot spots" Journal of Experimental Criminology vol. 9 no. 2 pp. 213-244 2013.
E. Ismagilova L. Hughes Y. K. Dwivedi K. R. Raman "Smart cities: Advances in research—An information systems perspective" Intern. Journal of Information Management vol. 47 pp. 88-100 2019.
S. Pellicer G. Santa A. L. Bleda R. Maestre A. J. Jara A. G. Skarmeta "A global perspective of smart cities: A survey" 2013 Seventh International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing pp. 439-444 July 2013.
S. Ijaz M. A. Shah A. Khan M. Ahmed "Smart cities: A survey on security concerns" International Journal of Advanced Computer Science and Applications vol. 7 no. 2 pp. 612-625 2016.
D. G. Costa I. Silva L. A. Guedes F. Vasques P. Portugal "Optimal sensing redundancy for multiple perspectives of targets in wireless visual sensor networks" 2015 IEEE 13th International Conference on Industrial Informatics (INDIN) pp. 185-190 2015.
M. M. Desai J. J. Patoliya "Smart toll collection system using embedded Linux environment" 2017 2nd International Conference for Convergence in Technology I2CT 2017 vol. 2017-Jan pp. 79-83 2017.
S. H. Liou Y. C. Hsieh C. Y. Chang "Design and Implementation of a Smart Parking Management System for Smart Cities" 2018 IEEE International Conference on Consumer Electronics-Taiwan ICCE-TW 2018 pp. 1-2 2018.
R. Shreyas B. V. Kumar H. B. Adithya B. Padmaja M. P. Sunil "Dynamic traffic rule violation monitoring system using automatic number plate recognition with SMS feedback" 2nd International Conference on Telecommunication and Networks TEL-NET 2017 vol. 2018-January pp. 1-5 2018.
V. R. Greati V. C. T. Ribeiro I. M. D. da Silva A. de Medeiros Martins "A Brazilian License Plate Recognition Method for Applications in Smart Cities" 2017 IEEE First Summer School on Smart Cities (S3C) pp. 43-48 2017.
S. M. Malik M. A. Iqbal Z. Hassan T. Tauqeer R. Hafiz U. Nasir "Automated over speeding detection and reporting system" 16th International Power Electronics and Motion Control Conference and Exposition PEMC 2014 pp. 1104-1109 2014.
H. Fu Y. Hu R. Huang "Non-public vehicle traffic-violation detection using mobile cameras" Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series vol. 8919 pp. 89190S 2013.
C. Patel D. Shah A. Patel "Automatic number plate recognition system (anpr): A survey" International Journal of Computer Applications vol. 69 no. 9 2013.
K. Sonavane B. Soni U. Majhi "Survey on automatic number plate recognition (anr)" International Journal of Computer Applications vol. 125 no. 6 2015.
I. M. D. Silva L. A. Guedes F. Vasques "Performance evaluation of a compression algorithm for wireless sensor networks in monitoring applications" 2008 IEEE International Conference on Emerging Technologies and Factory Automation pp. 672-678 2008.
S.-L. Chang L.-S. Chen Y.-C. Chung S.-W. Chen "Automatic license plate recognition" IEEE transactions on intelligent transportation systems vol. 5 no. 1 pp. 42-53 2004.
S. Z. Masood G. Shu A. Dehghan E. G. Ortiz "License plate detection and recognition using deeply learned convolutional neural networks" 2017.
Z. Selmi M. Ben Halima A. M. Alimi "Deep learning system for automatic license plate detection and recognition" 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) vol. 01 pp. 1132-1138 2017.
S. M. Silva C. R. Jung "Real-time brazilian license plate detection and recognition using deep convolutional neural networks" 2017 30th SIBGRAPI Conference on Graphics Patterns and Images (SIBGRAPI) pp. 55-62 2017.
C. Arth F. Limberger H. Bischof "Real-time license plate recognition on an embedded dsp-platform" 2007 IEEE Conference on Computer Vision and Pattern Recognition pp. 1-8 2007.
Resolução n° 729 2018 [online] Available: http://www.denatran.gov.br/images/Resolucoes/Resolucao7292018consolidada.pdf.
M. A. Oliveira Mercosul cria placa única para veículos 2014 [online] Available: http://g1.globo.com/mundo/noticia/2010/12/mercosul-cria-placa-unica-para-veiculos.html.
Brasil terá modelo de placas de veículos unificado com o mercosul [online] Available: http://www.brasil.gov.br/governo/2014/10/brasil-tera-modelo-de-placas-de-veiculos-unificado-com-o-mercosul.
S. Hinterstoisser V. Lepetit P. Wohlhart K. Konolige "On pre-trained image features and synthetic images for deep learning" Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) pp. 682-697 2019.
J. Tremblay A. Prakash D. Acuna M. Brophy V. Jampani C. Anil T. To E. Cameracci S. Boochoon S. Birchfield "Training deep networks with synthetic data: Bridging the reality gap by domain randomization" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops pp. 969-977 2018.
C. Gou H. Zhang K. Wang F.-Y. Wang Q. Ji "Cascade learning from adversarial synthetic images for accurate pupil detection" Pattern Recognition vol. 88 pp. 584-594 2019.
E. Olson C. Barbalata J. Zhang K. Skinner M. Johnson-Roberson "Synthetic data generation for deep learning of underwater disparity estimation" OCEANS 2018 MTS/IEEE Charleston OCEAN 2018 2019.
A. Marcu D. Costea V. Licǎreţ M. Pîrvu E. Sluşanschi M. Leordeanu "Safeuav: learning to estimate depth and safe landing areas for uavs from synthetic data" Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) pp. 43-58 2019.
H. Seibel S. Goldenstein A. Rocha "Eyes on the target: Superresolution and license-plate recognition in low-quality surveillance videos" IEEE Access vol. 5 pp. 20 020-20 035 2017.
A. N. Vicente H. Pedrini "A learning-based single-image superresolution method for very low quality license plate images" 2016 IEEE International Conference on Systems Man and Cybernetics (SMC) no. October pp. 000 515-000 520 2016 [online] Available: http://ieeexplore.ieee.org/document/7844291/.
J.-W. Hsieh S.-H. Yu Y.-S. Chen "Morphology-based license plate detection from complex scenes" in Object recognition supported by user interaction for service robots IEEE vol. 3 pp. 176-179 2002.
C. Gou K. Wang Y. Yao Z. Li "Vehicle license plate recognition based on extremal regions and restricted boltzmann machines" IEEE Transactions on Intelligent Transportation Systems vol. 17 no. 4 pp. 1096-1107 2016.
Y. Yuan W. Zou Y. Zhao X. Wang X. Hu N. Komodakis "A robust and efficient approach to license plate detection" IEEE Transactions on Image Processing vol. 26 no. 3 pp. 1102-1114 2017.
H. Li P. Wang M. You C. Shen "Reading car license plates using deep neural networks" Image and Vision Computing vol. 72 pp. 14-23 2018.
T. Björklund A. Fiandrotti M. Annarumma G. Francini E. Magli "Robust license plate recognition using neural networks trained on synthetic images" Pattern Recognition vol. 93 pp. 134-146 2019.
J. Han J. Yao J. Zhao J. Tu Y. Liu "Multi-oriented and scale-invariant license plate detection based on convolutional neural networks" Sensors vol. 19 no. 5 pp. 1175 2019.
J. Redmon S. K. Divvala R. B. Girshick A. Farhadi "You only look once: Unified real-time object detection" CoRR vol. abs/1506.02640 2015.
R. Laroca E. Severo L. A. Zanlorensi L. S. Oliveira G. R. Gonçalves W. R. Schwartz D. Menotti "A robust real-time automatic license plate recognition based on the yolo detector" 2018 International Joint Conference on Neural Networks (IJCNN) pp. 1-10 2018.
D. M. F. Izidio A. P. A. Ferreira E. N. S. Barros "An embedded automatic license plate recognition system using deep learning" 2018 VIII Brazilian Symposium on Computing Systems Engineering (SBESC) pp. 38-45 2018.
J. A. R. Serrano R. Bala F. Perronnin C. J. Saunders Y. Zhao "Methods And Systems For Improved License Plate Signature Matching By Similarity Learning On Synthetic Images" 2013.
O. Bulan V. Kozitsky P. Ramesh M. Shreve "Segmentation- and Annotation-Free License Plate Recognition with Deep Localization and Failure Identification" IEEE Transactions on Intelligent Transportation Systems vol. 18 no. 9 pp. 2351-2363 2017.
J. Wang H. Huang X. Qian J. Cao Y. Dai "Sequence recognition of Chinese license plates" Neurocomputing vol. 317 pp. 149-158 2018.
"Resolução n° 45/98" Brasília 1998 [online] Available: https://www.detran.sp.gov.br/wps/wcm/connect/edadabed-f11f-4381-825a-0ac888bd320f/resolucao045_98.pdf.
"Resolução n° 241/2007" Brasília 2007 [online] Available: https://www.denatran.gov.br/download/Resolucoes/RESOLUCAO_CONTRAN_241.pdf.
Patente e Sistema De Consultas Sobre Veículos do Mercosul 2014 [online] Available: http://www.cartillaciudadania.mercosur.int.
K. Lenc A. Vedaldi M. W. J. Xianghua Xie G. K. L. Tam "R-cnn minus r" Proceedings of the British Machine Vision Conference (BMVC) pp. 5.1-5.12 2015.
J. Redmon Darknet: Open source neural networks in c pp. 2013-2016 [online] Available: http://pjreddie.com/darknet/.
J. Redmon A. Farhadi "YOLO9000: better faster stronger" CoRR vol. abs/1612.08242 2016.
J. Redmon A. Farhadi "Yolov3: An incremental improvement" CoRR vol. abs/1804.02767 2018.
Resolução N° 510 2014 [online] Available: http://www.denatran.gov.br/download/Resolucoes/Resolucao5102014.pdf.
G. Bradski "The OpenCV Library" Dr. Dobb’s Journal of Software Tools 2000.
J. Matas C. Galambos J. Kittler "Robust detection of lines using the progressive probabilistic hough transform" Computer Vision and Image Understanding vol. 78 no. 1 pp. 119-137 2000.
C. S. Koper B. G. Taylor D. J. Woods "A randomized test of initial and residual deterrence from directed patrols and use of license plate readers at crime hot spots" Journal of Experimental Criminology vol. 9 no. 2 pp. 213-244 2013.
E. Ismagilova L. Hughes Y. K. Dwivedi K. R. Raman "Smart cities: Advances in research—An information systems perspective" Intern. Journal of Information Management vol. 47 pp. 88-100 2019.
S. Pellicer G. Santa A. L. Bleda R. Maestre A. J. Jara A. G. Skarmeta "A global perspective of smart cities: A survey" 2013 Seventh International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing pp. 439-444 July 2013.
S. Ijaz M. A. Shah A. Khan M. Ahmed "Smart cities: A survey on security concerns" International Journal of Advanced Computer Science and Applications vol. 7 no. 2 pp. 612-625 2016.
D. G. Costa I. Silva L. A. Guedes F. Vasques P. Portugal "Optimal sensing redundancy for multiple perspectives of targets in wireless visual sensor networks" 2015 IEEE 13th International Conference on Industrial Informatics (INDIN) pp. 185-190 2015.
M. M. Desai J. J. Patoliya "Smart toll collection system using embedded Linux environment" 2017 2nd International Conference for Convergence in Technology I2CT 2017 vol. 2017-Jan pp. 79-83 2017.
S. H. Liou Y. C. Hsieh C. Y. Chang "Design and Implementation of a Smart Parking Management System for Smart Cities" 2018 IEEE International Conference on Consumer Electronics-Taiwan ICCE-TW 2018 pp. 1-2 2018.
R. Shreyas B. V. Kumar H. B. Adithya B. Padmaja M. P. Sunil "Dynamic traffic rule violation monitoring system using automatic number plate recognition with SMS feedback" 2nd International Conference on Telecommunication and Networks TEL-NET 2017 vol. 2018-January pp. 1-5 2018.
V. R. Greati V. C. T. Ribeiro I. M. D. da Silva A. de Medeiros Martins "A Brazilian License Plate Recognition Method for Applications in Smart Cities" 2017 IEEE First Summer School on Smart Cities (S3C) pp. 43-48 2017.
S. M. Malik M. A. Iqbal Z. Hassan T. Tauqeer R. Hafiz U. Nasir "Automated over speeding detection and reporting system" 16th International Power Electronics and Motion Control Conference and Exposition PEMC 2014 pp. 1104-1109 2014.
H. Fu Y. Hu R. Huang "Non-public vehicle traffic-violation detection using mobile cameras" Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series vol. 8919 pp. 89190S 2013.
C. Patel D. Shah A. Patel "Automatic number plate recognition system (anpr): A survey" International Journal of Computer Applications vol. 69 no. 9 2013.
K. Sonavane B. Soni U. Majhi "Survey on automatic number plate recognition (anr)" International Journal of Computer Applications vol. 125 no. 6 2015.
I. M. D. Silva L. A. Guedes F. Vasques "Performance evaluation of a compression algorithm for wireless sensor networks in monitoring applications" 2008 IEEE International Conference on Emerging Technologies and Factory Automation pp. 672-678 2008.
S.-L. Chang L.-S. Chen Y.-C. Chung S.-W. Chen "Automatic license plate recognition" IEEE transactions on intelligent transportation systems vol. 5 no. 1 pp. 42-53 2004.
S. Z. Masood G. Shu A. Dehghan E. G. Ortiz "License plate detection and recognition using deeply learned convolutional neural networks" 2017.
Z. Selmi M. Ben Halima A. M. Alimi "Deep learning system for automatic license plate detection and recognition" 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) vol. 01 pp. 1132-1138 2017.
S. M. Silva C. R. Jung "Real-time brazilian license plate detection and recognition using deep convolutional neural networks" 2017 30th SIBGRAPI Conference on Graphics Patterns and Images (SIBGRAPI) pp. 55-62 2017.
C. Arth F. Limberger H. Bischof "Real-time license plate recognition on an embedded dsp-platform" 2007 IEEE Conference on Computer Vision and Pattern Recognition pp. 1-8 2007.
Resolução n° 729 2018 [online] Available: http://www.denatran.gov.br/images/Resolucoes/Resolucao7292018consolidada.pdf.
M. A. Oliveira Mercosul cria placa única para veículos 2014 [online] Available: http://g1.globo.com/mundo/noticia/2010/12/mercosul-cria-placa-unica-para-veiculos.html.
Brasil terá modelo de placas de veículos unificado com o mercosul [online] Available: http://www.brasil.gov.br/governo/2014/10/brasil-tera-modelo-de-placas-de-veiculos-unificado-com-o-mercosul.
S. Hinterstoisser V. Lepetit P. Wohlhart K. Konolige "On pre-trained image features and synthetic images for deep learning" Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) pp. 682-697 2019.
J. Tremblay A. Prakash D. Acuna M. Brophy V. Jampani C. Anil T. To E. Cameracci S. Boochoon S. Birchfield "Training deep networks with synthetic data: Bridging the reality gap by domain randomization" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops pp. 969-977 2018.
C. Gou H. Zhang K. Wang F.-Y. Wang Q. Ji "Cascade learning from adversarial synthetic images for accurate pupil detection" Pattern Recognition vol. 88 pp. 584-594 2019.
E. Olson C. Barbalata J. Zhang K. Skinner M. Johnson-Roberson "Synthetic data generation for deep learning of underwater disparity estimation" OCEANS 2018 MTS/IEEE Charleston OCEAN 2018 2019.
A. Marcu D. Costea V. Licǎreţ M. Pîrvu E. Sluşanschi M. Leordeanu "Safeuav: learning to estimate depth and safe landing areas for uavs from synthetic data" Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) pp. 43-58 2019.
H. Seibel S. Goldenstein A. Rocha "Eyes on the target: Superresolution and license-plate recognition in low-quality surveillance videos" IEEE Access vol. 5 pp. 20 020-20 035 2017.
A. N. Vicente H. Pedrini "A learning-based single-image superresolution method for very low quality license plate images" 2016 IEEE International Conference on Systems Man and Cybernetics (SMC) no. October pp. 000 515-000 520 2016 [online] Available: http://ieeexplore.ieee.org/document/7844291/.
J.-W. Hsieh S.-H. Yu Y.-S. Chen "Morphology-based license plate detection from complex scenes" in Object recognition supported by user interaction for service robots IEEE vol. 3 pp. 176-179 2002.
C. Gou K. Wang Y. Yao Z. Li "Vehicle license plate recognition based on extremal regions and restricted boltzmann machines" IEEE Transactions on Intelligent Transportation Systems vol. 17 no. 4 pp. 1096-1107 2016.
Y. Yuan W. Zou Y. Zhao X. Wang X. Hu N. Komodakis "A robust and efficient approach to license plate detection" IEEE Transactions on Image Processing vol. 26 no. 3 pp. 1102-1114 2017.
H. Li P. Wang M. You C. Shen "Reading car license plates using deep neural networks" Image and Vision Computing vol. 72 pp. 14-23 2018.
T. Björklund A. Fiandrotti M. Annarumma G. Francini E. Magli "Robust license plate recognition using neural networks trained on synthetic images" Pattern Recognition vol. 93 pp. 134-146 2019.
J. Han J. Yao J. Zhao J. Tu Y. Liu "Multi-oriented and scale-invariant license plate detection based on convolutional neural networks" Sensors vol. 19 no. 5 pp. 1175 2019.
J. Redmon S. K. Divvala R. B. Girshick A. Farhadi "You only look once: Unified real-time object detection" CoRR vol. abs/1506.02640 2015.
R. Laroca E. Severo L. A. Zanlorensi L. S. Oliveira G. R. Gonçalves W. R. Schwartz D. Menotti "A robust real-time automatic license plate recognition based on the yolo detector" 2018 International Joint Conference on Neural Networks (IJCNN) pp. 1-10 2018.
D. M. F. Izidio A. P. A. Ferreira E. N. S. Barros "An embedded automatic license plate recognition system using deep learning" 2018 VIII Brazilian Symposium on Computing Systems Engineering (SBESC) pp. 38-45 2018.
J. A. R. Serrano R. Bala F. Perronnin C. J. Saunders Y. Zhao "Methods And Systems For Improved License Plate Signature Matching By Similarity Learning On Synthetic Images" 2013.
O. Bulan V. Kozitsky P. Ramesh M. Shreve "Segmentation- and Annotation-Free License Plate Recognition with Deep Localization and Failure Identification" IEEE Transactions on Intelligent Transportation Systems vol. 18 no. 9 pp. 2351-2363 2017.
J. Wang H. Huang X. Qian J. Cao Y. Dai "Sequence recognition of Chinese license plates" Neurocomputing vol. 317 pp. 149-158 2018.
"Resolução n° 45/98" Brasília 1998 [online] Available: https://www.detran.sp.gov.br/wps/wcm/connect/edadabed-f11f-4381-825a-0ac888bd320f/resolucao045_98.pdf.
"Resolução n° 241/2007" Brasília 2007 [online] Available: https://www.denatran.gov.br/download/Resolucoes/RESOLUCAO_CONTRAN_241.pdf.
Patente e Sistema De Consultas Sobre Veículos do Mercosul 2014 [online] Available: http://www.cartillaciudadania.mercosur.int.
K. Lenc A. Vedaldi M. W. J. Xianghua Xie G. K. L. Tam "R-cnn minus r" Proceedings of the British Machine Vision Conference (BMVC) pp. 5.1-5.12 2015.
J. Redmon Darknet: Open source neural networks in c pp. 2013-2016 [online] Available: http://pjreddie.com/darknet/.
J. Redmon A. Farhadi "YOLO9000: better faster stronger" CoRR vol. abs/1612.08242 2016.
J. Redmon A. Farhadi "Yolov3: An incremental improvement" CoRR vol. abs/1804.02767 2018.
Resolução N° 510 2014 [online] Available: http://www.denatran.gov.br/download/Resolucoes/Resolucao5102014.pdf.
G. Bradski "The OpenCV Library" Dr. Dobb’s Journal of Software Tools 2000.
J. Matas C. Galambos J. Kittler "Robust detection of lines using the progressive probabilistic hough transform" Computer Vision and Image Understanding vol. 78 no. 1 pp. 119-137 2000.
Publicado
19/11/2019
Como Citar
RIBEIRO, Vinícius; GREATI, Vitor ; BEZERRA, Aguinaldo ; SILVANO, Gilles ; SILVA, Ivanovitch ; ENDO, Patricia ; LYNN, Theo .
Brazilian Mercosur License Plate Detection: a Deep Learning Approach Relying on Synthetic Imagery. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SISTEMAS COMPUTACIONAIS (SBESC), 9. , 2019, Natal.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 177-184.
ISSN 2237-5430.
