PCBShot: An Assisted Image Acquisition Method for PCB Damage Detection With Mobile Devices
Resumo
Identifying damages in Printed Circuit Boards is a critical task for quality assurance and repair inspection workflows. Image processing mobile applications, with embedded deep learning, assist technicians in detecting damages in this task, increasing accuracy and agility. However, the performance of such applications is highly dependent on the ability of the user in taking adequate photos. We propose an automatic capture method named PCBShot, that assists users of mobile applications of PCB damage detection to take better photos, enhancing the detection performance. Our method uses classical image processing algorithms to detect if a target PCB is inside a virtual guideline, ensuring that the position and distance are appropriate. Then, a photo is automatically captured, the background is cropped and the image is sliced into four quadrants for resolution preservation. The damage detection is performed in the slices. We evaluate our method through a real-life mobile application used in repair centers of an electronics manufacturer, comparing the detection performance with the manual image acquisition, without further assistance. Our results show that our method largely surpasses the manual acquisition, as it allows the capture of higher-quality images due to framing assistance with image processing methods, eliminating noisy backgrounds and preserving resolution.
Palavras-chave:
pcb damage detection, image acquisition, mobile application, automatic capture
Referências
K.-F. Henning, A. Fritze, E. Gillich, U. Mönks, and V. Lohweg, “Stable image acquisition for mobile image processing applications,” in Digital Photography XI, vol. 9404. SPIE, 2015, pp. 177–186.
A. M. Da Costa, A. O. De Sà, and R. C. Machado, “Data acquisition and extraction on mobile devices-a review,” in 2022 IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4. 0&IoT). IEEE, 2022, pp. 294–299.
J. Lei, X. Gao, Z. Feng, H. Qiu, and M. Song, “Scale insensitive and focus driven mobile screen defect detection in industry,” Neurocomputing, vol. 294, pp. 72–81, 2018.
A. Picon, A. Alvarez-Gila, M. Seitz, A. Ortiz-Barredo, J. Echazarra, and A. Johannes, “Deep convolutional neural networks for mobile capture device-based crop disease classification in the wild,” Computers and Electronics in Agriculture, vol. 161, pp. 280–290, 2019.
D. Moreira, P. Alves, F. Veiga, L. Rosado, and M. J. M. Vasconcelos, “Automated mobile image acquisition of macroscopic dermatological lesions.” in HEALTHINF, 2021, pp. 122–132.
V. A. Adibhatla, H.-C. Chih, C.-C. Hsu, J. Cheng, M. F. Abbod, and J.-S. Shieh, “Defect detection in printed circuit boards using you-only-look-once convolutional neural networks,” Electronics, vol. 9, no. 9, 2020. [Online]. Available: [link]
J. P. Santiago, V. Farias, L. Sena, J. P. P. Gomes, and J. Machado, “Real-time detection of customer-induced damage in printed circuit boards using mobile devices and YOLO detectors,” Learning & Nonlinear Models, vol. 22, no. 2, pp. 17–31, 2024.
D. Alves, V. Farias, I. Chaves, R. Chao, J. P. Madeiro, J. P. Gomes, and J. Machado, “Detecting customer induced damages in motherboards with deep neural networks,” in 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022, pp. 1–8.
C. Morikawa, M. Kobayashi, M. Satoh, Y. Kuroda, T. Inomata, H. Matsuo, T. Miura, and M. Hilaga, “Image and video processing on mobile devices: a survey,” the visual Computer, vol. 37, no. 12, pp. 2931–2949, 2021.
L. Zu, Y. Zhao, J. Liu, F. Su, Y. Zhang, and P. Liu, “Detection and segmentation of mature green tomatoes based on mask r-cnn with automatic image acquisition approach,” Sensors, vol. 21, no. 23, p. 7842, 2021.
Y. Liu, P. Sun, N. Wergeles, and Y. Shang, “A survey and performance evaluation of deep learning methods for small object detection,” Expert Systems with Applications, vol. 172, p. 114602, 2021.
J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan, “Perceptual generative adversarial networks for small object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1222–1230.
Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “Sod-mtgan: Small object detection via multi-task generative adversarial network,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 206–221.
L. Cabral, V. Farias, L. Sena, I. Chaves, J. P. Pordeus, J. P. Santiago, D. Sá, J. Machado, and J. P. Madeiro, “An active learning approach for detecting customer induced damages in motherboards with deep neural networks,” Learning & Nonlinear Models, vol. 21, no. 2, pp. 29–42, 2023.
J. Kaur and W. Singh, “Tools, techniques, datasets and application areas for object detection in an image: a review,” Multimedia Tools and Applications, vol. 81, no. 27, pp. 38 297–38 351, 2022.
A. A. Ahmed and G. H. Reddy, “A mobile-based system for detecting plant leaf diseases using deep learning,” AgriEngineering, vol. 3, no. 3, pp. 478–493, 2021.
J.-W. Chen, W.-J. Lin, H.-J. Cheng, C.-L. Hung, C.-Y. Lin, and S.-P. Chen, “A smartphone-based application for scale pest detection using multiple-object detection methods,” Electronics, vol. 10, no. 4, p. 372, 2021.
T. Napier and I. Lee, “Using mobile-based augmented reality and object detection for real-time abalone growth monitoring,” Computers and Electronics in Agriculture, vol. 207, p. 107744, 2023.
C. A. Hartanto and A. Wibowo, “Development of mobile skin cancer detection using faster r-cnn and mobilenet v2 model,” in 2020 7th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE). IEEE, 2020, pp. 58–63.
V. Agarwal, V. Gupta, V. M. Vashisht, K. Sharma, and N. Sharma, “Mobile application based cataract detection system,” in 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI). IEEE, 2019, pp. 780–787.
P. S. SM, M. Shariff, D. Subramanyam, M. Varun, K. Shruthi, and A. Poornima, “Real time oral cavity detection leading to oral cancer using cnn,” in 2023 International Conference on Network, Multimedia and Information Technology (NMITCON). IEEE, 2023, pp. 1–7.
N. Rane, “Yolo and faster r-cnn object detection for smart industry 4.0 and industry 5.0: applications, challenges, and opportunities,” Available at SSRN 4624206, 2023.
A. C. Bergstrom and D. W. Messinger, “Image quality and object detection performance of convolutional neural networks,” in Pattern Recognition and Tracking XXXIV, vol. 12527. SPIE, 2023, pp. 159–177.
Y. Hao, H. Pei, Y. Lyu, Z. Yuan, J.-R. Rizzo, Y. Wang, and Y. Fang, “Understanding the impact of image quality and distance of objects to object detection performance,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 11 436–11 442.
P. Faria, T. Nogueira, A. Ferreira, C. Carlos, and L. Rosado, “Ai-powered mobile image acquisition of vineyard insect traps with automatic quality and adequacy assessment,” Agronomy, vol. 11, no. 4, p. 731, 2021.
H. M. Heres, M. Sjoerdsma, T. Schoots, M. C. Rutten, F. N. van de Vosse, and R. G. Lopata, “Image acquisition stability of fixated musculoskeletal sonography in an exercise setting: A quantitative analysis and comparison with freehand acquisition,” Journal of Medical Ultrasonics, vol. 47, pp. 47–56, 2020.
R. C. Gonzales and P. Wintz, Digital image processing. Addison-Wesley Longman Publishing Co., Inc., 1987.
E. R. Dougherty, “An introduction to morphological image processing,” in SPIE. Optical Engineering Press, 1992.
N. Otsu et al., “A threshold selection method from gray-level histograms,” Automatica, vol. 11, no. 285-296, pp. 23–27, 1975.
S. Suzuki et al., “Topological structural analysis of digitized binary images by border following,” Computer vision, graphics, and image processing, vol. 30, no. 1, pp. 32–46, 1985.
D. Yang, B. Peng, Z. Al-Huda, A. Malik, and D. Zhai, “An overview of edge and object contour detection,” Neurocomputing, vol. 488, pp. 470–493, 2022.
Z. Wang, E. Wang, and Y. Zhu, “Image segmentation evaluation: a survey of methods,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5637–5674, 2020.
K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask R-CNN,” CoRR, vol. abs/1703.06870, 2017. [Online]. Available: [link]
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
G. Bradski, “The opencv library.” Dr. Dobb’s Journal: Software Tools for the Professional Programmer, vol. 25, no. 11, pp. 120–123, 2000.
A. M. Da Costa, A. O. De Sà, and R. C. Machado, “Data acquisition and extraction on mobile devices-a review,” in 2022 IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4. 0&IoT). IEEE, 2022, pp. 294–299.
J. Lei, X. Gao, Z. Feng, H. Qiu, and M. Song, “Scale insensitive and focus driven mobile screen defect detection in industry,” Neurocomputing, vol. 294, pp. 72–81, 2018.
A. Picon, A. Alvarez-Gila, M. Seitz, A. Ortiz-Barredo, J. Echazarra, and A. Johannes, “Deep convolutional neural networks for mobile capture device-based crop disease classification in the wild,” Computers and Electronics in Agriculture, vol. 161, pp. 280–290, 2019.
D. Moreira, P. Alves, F. Veiga, L. Rosado, and M. J. M. Vasconcelos, “Automated mobile image acquisition of macroscopic dermatological lesions.” in HEALTHINF, 2021, pp. 122–132.
V. A. Adibhatla, H.-C. Chih, C.-C. Hsu, J. Cheng, M. F. Abbod, and J.-S. Shieh, “Defect detection in printed circuit boards using you-only-look-once convolutional neural networks,” Electronics, vol. 9, no. 9, 2020. [Online]. Available: [link]
J. P. Santiago, V. Farias, L. Sena, J. P. P. Gomes, and J. Machado, “Real-time detection of customer-induced damage in printed circuit boards using mobile devices and YOLO detectors,” Learning & Nonlinear Models, vol. 22, no. 2, pp. 17–31, 2024.
D. Alves, V. Farias, I. Chaves, R. Chao, J. P. Madeiro, J. P. Gomes, and J. Machado, “Detecting customer induced damages in motherboards with deep neural networks,” in 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022, pp. 1–8.
C. Morikawa, M. Kobayashi, M. Satoh, Y. Kuroda, T. Inomata, H. Matsuo, T. Miura, and M. Hilaga, “Image and video processing on mobile devices: a survey,” the visual Computer, vol. 37, no. 12, pp. 2931–2949, 2021.
L. Zu, Y. Zhao, J. Liu, F. Su, Y. Zhang, and P. Liu, “Detection and segmentation of mature green tomatoes based on mask r-cnn with automatic image acquisition approach,” Sensors, vol. 21, no. 23, p. 7842, 2021.
Y. Liu, P. Sun, N. Wergeles, and Y. Shang, “A survey and performance evaluation of deep learning methods for small object detection,” Expert Systems with Applications, vol. 172, p. 114602, 2021.
J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan, “Perceptual generative adversarial networks for small object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1222–1230.
Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “Sod-mtgan: Small object detection via multi-task generative adversarial network,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 206–221.
L. Cabral, V. Farias, L. Sena, I. Chaves, J. P. Pordeus, J. P. Santiago, D. Sá, J. Machado, and J. P. Madeiro, “An active learning approach for detecting customer induced damages in motherboards with deep neural networks,” Learning & Nonlinear Models, vol. 21, no. 2, pp. 29–42, 2023.
J. Kaur and W. Singh, “Tools, techniques, datasets and application areas for object detection in an image: a review,” Multimedia Tools and Applications, vol. 81, no. 27, pp. 38 297–38 351, 2022.
A. A. Ahmed and G. H. Reddy, “A mobile-based system for detecting plant leaf diseases using deep learning,” AgriEngineering, vol. 3, no. 3, pp. 478–493, 2021.
J.-W. Chen, W.-J. Lin, H.-J. Cheng, C.-L. Hung, C.-Y. Lin, and S.-P. Chen, “A smartphone-based application for scale pest detection using multiple-object detection methods,” Electronics, vol. 10, no. 4, p. 372, 2021.
T. Napier and I. Lee, “Using mobile-based augmented reality and object detection for real-time abalone growth monitoring,” Computers and Electronics in Agriculture, vol. 207, p. 107744, 2023.
C. A. Hartanto and A. Wibowo, “Development of mobile skin cancer detection using faster r-cnn and mobilenet v2 model,” in 2020 7th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE). IEEE, 2020, pp. 58–63.
V. Agarwal, V. Gupta, V. M. Vashisht, K. Sharma, and N. Sharma, “Mobile application based cataract detection system,” in 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI). IEEE, 2019, pp. 780–787.
P. S. SM, M. Shariff, D. Subramanyam, M. Varun, K. Shruthi, and A. Poornima, “Real time oral cavity detection leading to oral cancer using cnn,” in 2023 International Conference on Network, Multimedia and Information Technology (NMITCON). IEEE, 2023, pp. 1–7.
N. Rane, “Yolo and faster r-cnn object detection for smart industry 4.0 and industry 5.0: applications, challenges, and opportunities,” Available at SSRN 4624206, 2023.
A. C. Bergstrom and D. W. Messinger, “Image quality and object detection performance of convolutional neural networks,” in Pattern Recognition and Tracking XXXIV, vol. 12527. SPIE, 2023, pp. 159–177.
Y. Hao, H. Pei, Y. Lyu, Z. Yuan, J.-R. Rizzo, Y. Wang, and Y. Fang, “Understanding the impact of image quality and distance of objects to object detection performance,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 11 436–11 442.
P. Faria, T. Nogueira, A. Ferreira, C. Carlos, and L. Rosado, “Ai-powered mobile image acquisition of vineyard insect traps with automatic quality and adequacy assessment,” Agronomy, vol. 11, no. 4, p. 731, 2021.
H. M. Heres, M. Sjoerdsma, T. Schoots, M. C. Rutten, F. N. van de Vosse, and R. G. Lopata, “Image acquisition stability of fixated musculoskeletal sonography in an exercise setting: A quantitative analysis and comparison with freehand acquisition,” Journal of Medical Ultrasonics, vol. 47, pp. 47–56, 2020.
R. C. Gonzales and P. Wintz, Digital image processing. Addison-Wesley Longman Publishing Co., Inc., 1987.
E. R. Dougherty, “An introduction to morphological image processing,” in SPIE. Optical Engineering Press, 1992.
N. Otsu et al., “A threshold selection method from gray-level histograms,” Automatica, vol. 11, no. 285-296, pp. 23–27, 1975.
S. Suzuki et al., “Topological structural analysis of digitized binary images by border following,” Computer vision, graphics, and image processing, vol. 30, no. 1, pp. 32–46, 1985.
D. Yang, B. Peng, Z. Al-Huda, A. Malik, and D. Zhai, “An overview of edge and object contour detection,” Neurocomputing, vol. 488, pp. 470–493, 2022.
Z. Wang, E. Wang, and Y. Zhu, “Image segmentation evaluation: a survey of methods,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5637–5674, 2020.
K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask R-CNN,” CoRR, vol. abs/1703.06870, 2017. [Online]. Available: [link]
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
G. Bradski, “The opencv library.” Dr. Dobb’s Journal: Software Tools for the Professional Programmer, vol. 25, no. 11, pp. 120–123, 2000.
Publicado
30/09/2024
Como Citar
CABRAL, Lucas; SENA, Lucas; SANTIAGO, João Pedro; NETO, Joaquim Bento Cavalcante; LENON, Yuri; MACHADO, Javam.
PCBShot: An Assisted Image Acquisition Method for PCB Damage Detection With Mobile Devices. In: WORKSHOP DE APLICAÇÕES INDUSTRIAIS - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 37. , 2024, Manaus/AM.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 186-193.
DOI: https://doi.org/10.5753/sibgrapi.est.2024.31670.
