Visual Test Automation with Convolutional Neural Networks: Comparative Study of Pre-trained Models for Android Screenshot Validation

• Leonardo Tiago Sidia Institute of Science and Technology
• Elza Simoes Bisneta Sidia Institute of Science and Technology
• Mateus Fernandes da Costa Sidia Institute of Science and Technology
• Ana Paula Silva Sidia Institute of Science and Technology
• Lennon Correa Chaves Sidia Institute of Science and Technology
• Flávia Camila Oliveira Sidia Institute of Science and Technology

DOI: https://doi.org/10.5753/ise.2026.32456

Resumo

Context: In a software institute, a test team receives test requests for mobile devices and handles them using automation tools, which generate screenshots as evidence of the success or failure of the test execution, but still need to be visually checked by a human tester, a task that becomes gradually more costly as the amount of test requests is increased. Objective: This work’s goal is to propose the automation of the visual check using pre-trained Convolutional Neural Networks (CNNs) with embedding extraction and cosine similarity. Method: Four CNN architectures were employed from the models MobileNetV2, EfficientNetB0, ResNet50V2 and InceptionV3, which were all pre-trained with weights from the ImageNet dataset, then tested with screenshots of 13 different applications in a positive scenario, to check if the images are similar, and in a negative scenario, to check if the images are different. Results: For the positive scenario, InceptionV3 and ResNet50V2 achieved higher similarity rates when compared to EfficientNetB0 and MobileNetV2, although not by much; for the negative scenario, InceptionV3, MobileNetV2 and ResNet50V2 achieved variable results for identifying differences, while EfficientNetB0 achieved the best result by a wide margin. Conclusion: Among the analysed models, the EfficientNetB0 model stood out in regards to performance, achieving similarity rates more consistent with the expected results. For future studies, it is suggested to apply this approach to other contexts of test cases, or to try to improve it including a training step for the models using a dataset built from real screenshots captured by the test team.

Palavras-chave: Android Screenshot Validation, Software Testing, Test Automation, Pre-Trained Models

Referências

Abdel Ahmed. 2025. Machine Learning for Rendering Regression Detection: Enhancing Screenshot-Based Validation for Game Engine QA.

Emil Alégroth, Robert Feldt, and Lisa Ryrholm. 2015. Visual gui testing in practice: challenges, problemsand limitations. Empirical Software Engineering 20, 3 (2015), 694–744.

Lennon Chaves, Davi Gonzaga, Leonardo Tiago, Ana Silva, and Flávia Oliveira. 2025. Automatic Generation of Bug Reports Using Large Language Models: An Evaluation in a Software Institute. In Anais do XXIV Simpósio Brasileiro de Qualidade de Software (São José dos Campos/SP). SBC, Porto Alegre, RS, Brasil, 337–345. DOI: 10.5753/sbqs.2025.13599

Lennon Chaves, Flávia Oliveira, and Leonardo Tiago. 2024. Automating issue reporting in software testing: Lessons learned from using the template generator tool. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 278–282.

Lennon Chaves, Flavia Oliveira, Leonardo Tiago, and Renata Castro. 2024. Benefits and Challenges of a Physical Device Farm for Automated Software Testing: A Case Study of the STELA Tool Implementation. In Anais do IX Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 89–91. DOI: 10.5753/sast.2024.3880

Jia Deng,Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.

Ke Dong, Chengjie Zhou, Yihan Ruan, and Yuzhi Li. 2020. MobileNetV2 model for image classification. In 2020 2nd International Conference on Information Technology and Computer Application (ITCA). IEEE, 476–480.

Delvi Hastari, SalsaWinanda, Aditya Rezky Pratama, Nana Nurhaliza, and Ella Silvana Ginting. 2024. Application of convolutional neural network ResNet-50 V2 on image classification of rice plant disease. Public Research Journal of Engineering, Data Technology and Computer Science 1, 2 (2024).

Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Srinivas Sunkara, Victor Cărbune, Jason Lin, Maria Wang, Yun Zhu, and Jindong Chen. 2025. Screenqa: Large-scale question-answer pairs over mobile app screenshots. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 9427–9452.

Phil Kim. 2017. Convolutional neural network. In MATLAB deep learning: with machine learning, neural networks and artificial intelligence. Springer, 121–147.

Dongjie Liu and Jong-Hyouk Lee. 2020. CNN Based Malicious Website Detection by Invalidating Multiple Web Spams. IEEE Access 8 (2020), 97258–97266. DOI: 10.1109/ACCESS.2020.2995157

Flavia Oliveira, Leonardo Tiago, and Lennon Chaves. 2024. Reducing the Allocation of Software TestingDemand: A Study on the Pros andCons using STELATool. In Anais do IX Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 77–79. DOI: 10.5753/sast.2024.3673

Charmy H Patel, Devang Undaviya, Harsh Dave, Sheshang Degadwala, and Dhairya Vyas. 2023. EfficientNetB0 for brain stroke classification on computed tomography scan. In 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC). IEEE, 713–718.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.

Xiaoling Xia, Cui Xu, and Bing Nan. 2017. Inception-v3 for flower classification. In 2017 2nd international conference on image, vision and computing (ICIVC). IEEE, 783–787.