Automatic Systematic GUI Testing for Web Applications
Abstract
Graphical User Interface (GUI) testing plays a fundamental role in ensuring the reliability and quality of web applications by validating functionalities and exposing faults. Automated GUI testing tools are crucial for scaling this process, particularly in identifying visible failures such as crashes, error messages, and unexpected behaviors. However, their application in real-world, large-scale systems poses persistent challenges, including: (i) the automatic and non-redundant discovery of actionable elements, (ii) reliable synchronization with dynamic content, and (iii) efficient and thorough exploration of complex GUIs. This work addresses these challenges through the design and evaluation of three novel techniques: Unique Actionable Elements Search (UAES) for accurate element detection, the NetworkWait mechanism for robust synchronization, and the Iterative Deepening URL-Based Search (IDUBS) algorithm to guide scalable and effective exploration. These techniques are integrated into Cytestion, a new automated GUI testing tool for web applications. Cytestion follows a scriptless and progressive testing approach, starting from a seed test and incrementally exploring the interface by leveraging browser console errors, HTTP status codes, and GUI failure indicators as oracles. We evaluated our solution in a series of empirical studies involving four open-source and twenty industrial web applications. Results show that Cytestion significantly outperforms a state-of-the-art GUI testing tool in both fault detection effectiveness and runtime efficiency. In addition to its academic contributions, Cytestion has been adopted as part of the quality gate mechanism of an industry partner, demonstrating its practical applicability and impact.
References
Axel Bons, Beatriz Marín, Pekka Aho, and Tanja EJ Vos. 2023. Scripted and scriptless GUI testing for web applications: An industrial case. Information and Software Technology 158 (2023), 107172.
Maura Cerioli, Maurizio Leotta, and Filippo Ricca. 2020. What 5 million job advertisements tell us about testing: a preliminary empirical investigation. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 1586–1594.
Hatim Chahim, Mehmet Duran, Tanja EJ Vos, Pekka Aho, and Nelly Condori Fernandez. 2020. Scriptless testing at the GUI level in an industrial setting. In Research Challenges in Information Science: 14th International Conference, RCIS 2020, Limassol, Cyprus, September 23–25, 2020, Proceedings 14. Springer, 267–284.
Imran Akhtar Khan and Roopa Singh. 2012. Quality Assurance And Integration Testing Aspects In Web Based Applications. ArXiv abs/1207.3213 (2012). DOI: 10.5121/ijcsea.2012.2310
Thiago Santos de Moura, Everton L. G. Alves, Regina Letícia Santos Felipe, Cláudio de Souza Baptista, Ismael Raimundo da Silva Neto, and Hugo Feitosa de Figueirêdo. 2024. Addressing the Synchronization Challenge in Cypress End-to-End Tests. In Proceedings of the XXXVIII Brazilian Symposium on Software Engineering.
Thiago Santos de Moura, Everton L. G. Alves, Hugo Feitosa de Figueirêdo, and Cláudio de Souza Baptista. 2023. Cytestion: Automated GUI Testing for Web Applications. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering. 388–397.
Thiago Santos de Moura, Regina Letícia Santos Felipe, Everton L. G. Alves, Pedro Henrique S. C. Gregório, Cláudio de Souza Baptista, and Hugo Feitosa de Figueirêdo. 2024. Iterative Deepening URL-Based Search: Enhancing GUI Testing for Web Applications. In Proceedings of the XXXVIII Brazilian Symposium on Software Engineering.
Thiago Santos de Moura, Francisco Igor de Lima Mendes, Everton L. G. Alves, Ismael Raimundo da Silva Neto, and Cláudio de Souza Baptista. 2024. An Automatic Approach for Uniquely Discovering Actionable Elements for Systematic GUI Testing in Web Applications. In 2024 IEEE International Conference on Software Quality, Reliability and Security. IEEE.
Michel Nass, Emil Alégroth, and Robert Feldt. 2021. Why many challenges with GUI test automation (will) remain. Information and Software Technology 138 (2021), 106625.
Olivia Rodríguez-Valdés, Tanja EJ Vos, Pekka Aho, and Beatriz Marín. 2021. 30 years of automated GUI testing: a bibliometric analysis. In Quality of Information and Communications Technology: 14th International Conference, QUATIC 2021, Algarve, Portugal, September 8–11, 2021, Proceedings 14. Springer, 473–488.
Tanja EJ Vos, Pekka Aho, Fernando Pastor Ricos, Olivia Rodriguez-Valdes, and Ad Mulders. 2021. testar–scriptless testing through graphical user interface. Software Testing, Verification and Reliability 31, 3 (2021), e1771.
