Automated Visual Inspection of Aircraft Exterior Using Deep Learning
Resumo
Aircraft visual inspections, or General Visual Inspections (GVIs), aim at finding damages or anomalies on the exterior and interior surfaces of the aircraft, which might compromise its operation, structure, or safety when flying. Visual inspection is part of the activities of aircraft Maintenance, Repair and Overhaul (MRO). Specialists perform quality inspections to identify problems and determine the type and importance that they will report. This process is time-consuming, subjective, and varies according to each individual. The time that an aircraft stays grounded without flight clearance means financial losses. The main goal of this work is to advance the state-of-the-art of defect detection on aircraft exterior with deep learning and computer vision. We investigate improvements to the accuracy of dent detection. Besides, we investigate new classes of identified defects, such as scratches. We also plan to demonstrate that it is possible to develop a complete system to automate the visual inspection of aircraft exterior using images of the aircraft acquired by drones. We will use deep neural networks for the detection and segmentation of defective regions. This system will aid in the elimination of subjectivity caused by human errors and shorten the time required to inspect an aircraft, bringing benefits to its safety, maintenance, and operation.
Referências
I. Savage, “Comparing the fatality risks in united states transportation across modes and over time,” Research in transportation economics, vol. 43, no. 1, pp. 9–22, 2013.
C. V. Oster Jr, J. S. Strong, and C. K. Zorn, “Analyzing aviation safety: Problems, challenges, opportunities,” Research in transportation economics, vol. 43, no. 1, pp. 148–164, 2013.
A. Barnett, “Aviation safety: a whole new world?” Transportation science, vol. 54, no. 1, pp. 84–96, 2020.
A. K. Gramopadhye and C. G. Drury, “Human factors in aviation maintenance: how we got to where we are,” 2000.
W. Chen and S. Huang, “Human reliability analysis for visual inspection in aviation maintenance by a bayesian network approach,” Transportation Research Record, vol. 2449, no. 1, pp. 105–113, 2014.
I. Jovancevic, S. Larnier, J.-J. Orteu, and T. Sentenac, “Automated exterior inspection of an aircraft with a pan-tilt-zoom camera mounted on a mobile robot,” Journal of Electronic Imaging, vol. 24, no. 6, p. 061110, 2015.
I. Jovanzeviz, I. Viana, J.-J. Orteu, T. Sentenac, and S. Larnier, “Matching cad model and image features for robot navigation and inspection of an aircraft,” in Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods, 2016, pp. 359–366.
L. T. Ostrom, C. A. Wilhelmsen, and R. L. Scott, “Use of three dimensional imaging to perform aircraft composite inspection: Proof of concept,” in 2012 5th International Conference on Human System Interactions. IEEE, 2012, pp. 53–58.
C. A. Wilhelmsen and L. T. Ostrom, “Remote aircraft composite inspection using 3d imaging,” in 2016 9th International Conference on Human System Interactions (HSI). IEEE, 2016, pp. 316–322.
I. Jovancevic, A. Arafat, J.-J. Orteu, and T. Sentenac, “Airplane tire inspection by image processing techniques,” in 2016 5th Mediterranean Conference on Embedded Computing (MECO). IEEE, 2016, pp. 176–179.
J. R. Leiva, T. Villemot, G. Dangoumeau, M.-A. Bauda, and S. Larnier, “Automatic visual detection and verification of exterior aircraft elements,” in 2017 IEEE International Workshop of Electronics, Control, Measurement, Signals and their Application to Mechatronics (ECMSM). IEEE, 2017, pp. 1–5.
J. E. See, “Visual inspection: a review of the literature,” Sandia Report SAND2012-8590, Sandia National Laboratories, Albuquerque, New Mexico, 2012.
B. Dhillon and Y. Liu, “Human error in maintenance: a review,” Journal of quality in maintenance engineering, 2006.
K. A. Latorella and P. V. Prabhu, “A review of human error in aviation maintenance and inspection,” International Journal of industrial ergonomics, vol. 26, no. 2, pp. 133–161, 2000.
J. Miranda, J. Veith, S. Larnier, A. Herbulot, and M. Devy, “Machine learning approaches for defect classification on aircraft fuselage images aquired by an uav,” in Fourteenth International Conference on Quality Control by Artificial Vision, vol. 11172. International Society for Optics and Photonics, 2019, p. 1117208.
T. Malekzadeh, M. Abdollahzadeh, H. Nejati, and N.-M. Cheung, “Aircraft fuselage defect detection using deep neural networks,” arXiv preprint arXiv:1712.09213, 2017.
Y. Li, Z. Han, H. Xu, L. Liu, X. Li, and K. Zhang, “Yolov3-lite: A lightweight crack detection network for aircraft structure based on depthwise separable convolutions,” Applied Sciences, vol. 9, no. 18, p. 3781, 2019.
S. Bouarfa, A. Dogru, R. Arizar, R. Aydogan, and J. Serafico, “Towards automated aircraft maintenance inspection. a use case of detecting aircraft dents using mask r-cnn,” in AIAA Scitech 2020 Forum, 2020, p. 0389.
A. Dogru, S. Bouarfa, R. Arizar, and R. Aydogan, “Using convolutional neural networks to automate aircraft maintenance visual inspection,” Aerospace, vol. 7, no. 12, p. 171, 2020.
K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.
A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.
J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018.
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–755.
A. Dutta, A. Gupta, and A. Zissermann, “VGG image annotator (VIA),” http://www.robots.ox.ac.uk/vgg/software/via/, 2016, version: 2.0.11, Accessed: 15 Jul. 2021.
A. Dutta and A. Zisserman, “The VIA annotation software for images, audio and video,” in Proceedings of the 27th ACM International Conference on Multimedia. New York, NY, USA: ACM, 2019. [Online]. Available: https://doi.org/10.1145/3343031.3350535
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
C. V. Oster Jr, J. S. Strong, and C. K. Zorn, “Analyzing aviation safety: Problems, challenges, opportunities,” Research in transportation economics, vol. 43, no. 1, pp. 148–164, 2013.
A. Barnett, “Aviation safety: a whole new world?” Transportation science, vol. 54, no. 1, pp. 84–96, 2020.
A. K. Gramopadhye and C. G. Drury, “Human factors in aviation maintenance: how we got to where we are,” 2000.
W. Chen and S. Huang, “Human reliability analysis for visual inspection in aviation maintenance by a bayesian network approach,” Transportation Research Record, vol. 2449, no. 1, pp. 105–113, 2014.
I. Jovancevic, S. Larnier, J.-J. Orteu, and T. Sentenac, “Automated exterior inspection of an aircraft with a pan-tilt-zoom camera mounted on a mobile robot,” Journal of Electronic Imaging, vol. 24, no. 6, p. 061110, 2015.
I. Jovanzeviz, I. Viana, J.-J. Orteu, T. Sentenac, and S. Larnier, “Matching cad model and image features for robot navigation and inspection of an aircraft,” in Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods, 2016, pp. 359–366.
L. T. Ostrom, C. A. Wilhelmsen, and R. L. Scott, “Use of three dimensional imaging to perform aircraft composite inspection: Proof of concept,” in 2012 5th International Conference on Human System Interactions. IEEE, 2012, pp. 53–58.
C. A. Wilhelmsen and L. T. Ostrom, “Remote aircraft composite inspection using 3d imaging,” in 2016 9th International Conference on Human System Interactions (HSI). IEEE, 2016, pp. 316–322.
I. Jovancevic, A. Arafat, J.-J. Orteu, and T. Sentenac, “Airplane tire inspection by image processing techniques,” in 2016 5th Mediterranean Conference on Embedded Computing (MECO). IEEE, 2016, pp. 176–179.
J. R. Leiva, T. Villemot, G. Dangoumeau, M.-A. Bauda, and S. Larnier, “Automatic visual detection and verification of exterior aircraft elements,” in 2017 IEEE International Workshop of Electronics, Control, Measurement, Signals and their Application to Mechatronics (ECMSM). IEEE, 2017, pp. 1–5.
J. E. See, “Visual inspection: a review of the literature,” Sandia Report SAND2012-8590, Sandia National Laboratories, Albuquerque, New Mexico, 2012.
B. Dhillon and Y. Liu, “Human error in maintenance: a review,” Journal of quality in maintenance engineering, 2006.
K. A. Latorella and P. V. Prabhu, “A review of human error in aviation maintenance and inspection,” International Journal of industrial ergonomics, vol. 26, no. 2, pp. 133–161, 2000.
J. Miranda, J. Veith, S. Larnier, A. Herbulot, and M. Devy, “Machine learning approaches for defect classification on aircraft fuselage images aquired by an uav,” in Fourteenth International Conference on Quality Control by Artificial Vision, vol. 11172. International Society for Optics and Photonics, 2019, p. 1117208.
T. Malekzadeh, M. Abdollahzadeh, H. Nejati, and N.-M. Cheung, “Aircraft fuselage defect detection using deep neural networks,” arXiv preprint arXiv:1712.09213, 2017.
Y. Li, Z. Han, H. Xu, L. Liu, X. Li, and K. Zhang, “Yolov3-lite: A lightweight crack detection network for aircraft structure based on depthwise separable convolutions,” Applied Sciences, vol. 9, no. 18, p. 3781, 2019.
S. Bouarfa, A. Dogru, R. Arizar, R. Aydogan, and J. Serafico, “Towards automated aircraft maintenance inspection. a use case of detecting aircraft dents using mask r-cnn,” in AIAA Scitech 2020 Forum, 2020, p. 0389.
A. Dogru, S. Bouarfa, R. Arizar, and R. Aydogan, “Using convolutional neural networks to automate aircraft maintenance visual inspection,” Aerospace, vol. 7, no. 12, p. 171, 2020.
K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.
A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.
J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018.
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–755.
A. Dutta, A. Gupta, and A. Zissermann, “VGG image annotator (VIA),” http://www.robots.ox.ac.uk/vgg/software/via/, 2016, version: 2.0.11, Accessed: 15 Jul. 2021.
A. Dutta and A. Zisserman, “The VIA annotation software for images, audio and video,” in Proceedings of the 27th ACM International Conference on Multimedia. New York, NY, USA: ACM, 2019. [Online]. Available: https://doi.org/10.1145/3343031.3350535
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
Publicado
18/10/2021
Como Citar
YASUDA, Yuri D. V.; CAPPABIANCO, Fabio A. M.; MARTINS, Luiz Eduardo G.; GRIPP, Jorge A. B..
Automated Visual Inspection of Aircraft Exterior Using Deep Learning. In: WORKSHOP DE TRABALHOS EM ANDAMENTO - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 34. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 173-176.
DOI: https://doi.org/10.5753/sibgrapi.est.2021.20034.
