Thiago Santos de Moura
ABSTRACT
Automated testing in web applications is becoming increasingly popular due to its cost-effectiveness, efficiency, and repeatability. Ensuring the quality of such software is vital, especially in scenarios where business requirements are uncertain and volatile. In this paper, we propose Cytestion, a tool that automatically explores a web application’s Graphical User Interface (GUI) to detect faults that cause visible failures, such as crashes, errors, and unexpected behaviors. For that, it applies a scriptless and progressive approach. To work as an oracle, Cytestion evaluates the output of each test case execution by considering the HTTP status of server requests, default and customized failure GUI messages, and console failure messages. We evaluated the effectiveness and cost of our approach in two empirical studies. The first study works with injected faults in open-source projects, while the second is a case study on eight industrial projects. Cytestion detected several faults in both studies and outperformed a state-of-the-art GUI testing tool.
- Andrea Adamoli, Dmitrijs Zaparanuks, Milan Jovic, and Matthias Hauswirth. 2011. Automated GUI performance testing. Software Quality Journal 19, 4 (2011), 801–839.Google Scholar
Digital Library
- Emil Alégroth, Robert Feldt, and Helena H Olsson. 2013. Transitioning manual system test suites to automated testing: An industrial case study. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. IEEE, IEEE, Luxembourg, Luxembourg, 56–65. https://doi.org/10.1109/ICST.2013.14Google ScholarDigital Library
- Emil Alégroth, Zebao Gao, Rafael Oliveira, and Atif Memon. 2015. Conceptualization and evaluation of component-based testing unified with visual gui testing: an empirical study. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, IEEE, Graz, Austria, 1–10. https://doi.org/10.1109/ICST.2015.7102584Google ScholarCross Ref
- Francisco Almenar, Anna I Esparcia-Alcázar, Mirella Martínez, and Urko Rueda. 2016. Automated Testing of Web Applications with TESTAR: Lessons Learned Testing the Odoo Tool. In Search Based Software Engineering: 8th International Symposium, SSBSE 2016, Raleigh, NC, USA, October 8-10, 2016, Proceedings 8. Springer, Springer, Raleigh, North Carolina, USA, 218–223. https://doi.org/10.1007/978-3-319-47106-8_15Google ScholarCross Ref
- James H Andrews, Lionel C Briand, and Yvan Labiche. 2005. Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th international conference on Software engineering. Association for Computing Machinery, St. Louis MO USA, 402–411.Google Scholar
- Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications. Association for Computing Machinery, New York, NY, USA, 641–660. https://doi.org/10.1145/2509136.2509549Google ScholarDigital Library
- Ishan Banerjee, Bao Nguyen, Vahid Garousi, and Atif Memon. 2013. Graphical user interface (GUI) testing: Systematic mapping and repository. Information and Software Technology 55, 10 (2013), 1679–1694.Google ScholarDigital Library
- Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering 41, 5 (2014), 507–525.Google ScholarDigital Library
- Sebastian Bauersfeld, Tanja EJ Vos, Nelly Condori-Fernández, Alessandra Bagnato, and Etienne Brosse. 2014. Evaluating the TESTAR tool in an industrial case study. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. Association for Computing Machinery, New York, NY, USA, 1–9.Google ScholarDigital Library
- Axel Bons, Beatriz Marín, Pekka Aho, and Tanja EJ Vos. 2023. Scripted and scriptless GUI testing for web applications: An industrial case. Information and Software Technology 158 (2023), 107172.Google ScholarDigital Library
- Emil Borjesson and Robert Feldt. 2012. Automated system testing using visual gui testing tools: A comparative study in industry. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. IEEE, 350–359.Google ScholarDigital Library
- Hatim Chahim, Mehmet Duran, Tanja EJ Vos, Pekka Aho, and Nelly Condori Fernandez. 2020. Scriptless testing at the GUI level in an industrial setting. In Research Challenges in Information Science: 14th International Conference, RCIS 2020, Limassol, Cyprus, September 23–25, 2020, Proceedings 14. Springer, 267–284.Google ScholarCross Ref
- Peter Chapman and David Evans. 2011. Automated black-box detection of side-channel vulnerabilities in web applications. In Proceedings of the 18th ACM conference on Computer and communications security. 263–274.Google ScholarDigital Library
- Kinga Dobolyi, Elizabeth Soechting, and Westley Weimer. 2011. Automating regression testing using web-based application similarities. International journal on software tools for technology transfer 13 (2011), 111–129. https://doi.org/10.1007/s10009-010-0170-xGoogle ScholarCross Ref
- Zebao Gao, Chunrong Fang, and Atif M Memon. 2015. Pushing the limits on automation in GUI regression testing. In 2015 IEEE 26th international symposium on software reliability engineering (ISSRE). IEEE, 565–575.Google ScholarDigital Library
- Boni García, Micael Gallego, Francisco Gortázar, and Mario Munoz-Organero. 2020. A survey of the selenium ecosystem. Electronics 9, 7 (2020), 1067.Google ScholarCross Ref
- Mark Grechanik, Qing Xie, and Chen Fu. 2009. Creating GUI testing tools using accessibility technologies. In 2009 International Conference on Software Testing, Verification, and Validation Workshops. IEEE, 243–250.Google ScholarDigital Library
- Mark Grechanik, Qing Xie, and Chen Fu. 2009. Maintaining and evolving GUI-directed test scripts. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 408–418.Google ScholarDigital Library
- Birgit Hofer, Bernhard Peischl, and Franz Wotawa. 2009. Gui savvy end-to-end testing with smart monkeys. In 2009 ICSE Workshop on Automation of Software Test. IEEE, 130–137.Google ScholarCross Ref
- Gunel Jahangirova. 2017. Oracle problem in software testing. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 444–447.Google ScholarDigital Library
- René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 654–665.Google ScholarDigital Library
- Rebecca Krosnick and Steve Oney. 2021. Understanding the challenges and needs of programmers writing web automation scripts. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1–9.Google ScholarCross Ref
- Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2013. Capture-replay vs. programmable web testing: An empirical assessment during test case evolution. In 2013 20th Working Conference on Reverse Engineering (WCRE). IEEE, 272–281.Google ScholarCross Ref
- Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2014. Visual vs. DOM-based web locators: An empirical study. In Web Engineering: 14th International Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings 14. Springer, 322–340.Google ScholarCross Ref
- Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2016. Approaches and tools for automated end-to-end web testing. In Advances in Computers. Vol. 101. Elsevier, 193–237.Google Scholar
- Grischa Liebel, Emil Alégroth, and Robert Feldt. 2013. State-of-practice in GUI-based system and acceptance testing: An industrial multiple-case study. In 2013 39th Euromicro Conference on Software Engineering and Advanced Applications. IEEE, 17–24.Google ScholarDigital Library
- Mirella Martínez, Anna I Esparcia-Alcázar, Tanja EJ Vos, Pekka Aho, and Joan Fons i Cors. 2018. Towards automated testing of the internet of things: Results obtained with the testar tool. In Leveraging Applications of Formal Methods, Verification and Validation. Distributed Systems: 8th International Symposium, ISoLA 2018, Limassol, Cyprus, November 5-9, 2018, Proceedings, Part III 8. Springer, 375–385.Google Scholar
- Atif Memon, Ishan Banerjee, and Adithya Nagarajan. 2003. GUI ripping: Reverse engineering of graphical user interfaces for testing. In 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings. IEEE, 260–269.Google ScholarCross Ref
- Ali Mesbah, Arie van Deursen, and Stefan Lenselink. 2012. Crawling Ajax-Based Web Applications through Dynamic Analysis of User Interface State Changes. ACM Trans. Web 6, 1, Article 3 (mar 2012), 30 pages. https://doi.org/10.1145/2109205.2109208Google ScholarDigital Library
- Fatini Mobaraya, Shahid Ali, 2019. Technical Analysis of Selenium and Cypress as Functional Automation Framework for Modern Web Application Testing. In 9th International Conference on Computer Science.Google ScholarCross Ref
- Ad Mulders, Olivia Rodriguez Valdes, Fernando Pastor Ricós, Pekka Aho, Beatriz Marín, and Tanja EJ Vos. 2022. State model inference through the GUI using run-time test generation. In International Conference on Research Challenges in Information Science. Springer, 546–563.Google ScholarCross Ref
- Michel Nass, Emil Alégroth, and Robert Feldt. 2021. Why many challenges with GUI test automation (will) remain. Information and Software Technology 138 (2021), 106625.Google ScholarCross Ref
- Bao N. Nguyen, Bryan Robbins, Ishan Banerjee, and Atif Memon. 2014. GUITAR: An Innovative Tool for Automated Testing of GUI-Driven Software. Automated Software Engg. 21, 1 (mar 2014), 65–105. https://doi.org/10.1007/s10515-013-0128-9Google ScholarDigital Library
- Dario Olianas, Maurizio Leotta, and Filippo Ricca. 2022. SleepReplacer: a novel tool-based approach for replacing thread sleeps in selenium WebDriver test code. Software Quality Journal (2022), 1–33.Google Scholar
- Rafael AP Oliveira, Upulee Kanewala, and Paulo A Nardi. 2014. Automated test oracles: State of the art, taxonomies, and trends. Advances in computers 95 (2014), 113–199.Google ScholarCross Ref
- Filippo Ricca, Maurizio Leotta, and Andrea Stocco. 2019. Three open problems in the context of e2e web testing and a vision: Neonate. In Advances in Computers. Vol. 113. Elsevier, 89–133.Google Scholar
- Fernando Pastor Ricós, Arend Slomp, Beatriz Marín, Pekka Aho, and Tanja EJ Vos. 2023. Distributed state model inference for scriptless GUI testing. Journal of Systems and Software 200 (2023), 111645.Google ScholarDigital Library
- Olivia Rodríguez-Valdés, Tanja EJ Vos, Pekka Aho, and Beatriz Marín. 2021. 30 years of automated GUI testing: A bibliometric analysis. In Quality of Information and Communications Technology: 14th International Conference, QUATIC 2021, Algarve, Portugal, September 8–11, 2021, Proceedings 14. Springer, 473–488.Google ScholarCross Ref
- Carlos Gabriel Gomes de Melo Silva. 2019. Estudo comparativo de ferramentas de testes de ponta a ponta automatizados em sistemas web. B.S. thesis. Universidade Federal do Rio Grande do Norte.Google Scholar
- Tommi Takala, Mika Katara, and Julian Harty. 2011. Experiences of system-level model-based GUI testing of an Android application. In 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 377–386.Google ScholarDigital Library
- Paolo Tonella, Filippo Ricca, and Alessandro Marchetto. 2014. Recent advances in web testing. Advances in Computers 93 (2014), 1–51.Google ScholarCross Ref
- Tanja EJ Vos, Pekka Aho, Fernando Pastor Ricos, Olivia Rodriguez-Valdes, and Ad Mulders. 2021. TESTAR–scriptless testing through graphical user interface. Software Testing, Verification and Reliability 31, 3 (2021), e1771.Google ScholarCross Ref
- Tanja EJ Vos, Peter M Kruse, Nelly Condori-Fernández, Sebastian Bauersfeld, and Joachim Wegener. 2015. Testar: Tool support for test automation at the user interface level. International Journal of Information System Modeling and Design (IJISMD) 6, 3 (2015), 46–83.Google ScholarDigital Library
- Dacong Yan, Shengqian Yang, and Atanas Rountev. 2013. Systematic testing for resource leaks in Android applications. In 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 411–420. https://doi.org/10.1109/ISSRE.2013.6698894Google ScholarCross Ref
- Xun Yuan and Atif M. Memon. 2010. Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback. IEEE Transactions on Software Engineering 36, 1 (2010), 81–95. https://doi.org/10.1109/TSE.2009.68Google ScholarDigital Library
Index Terms
- Cytestion: Automated GUI Testing for Web Applications
Recommendations
Coverage criteria for GUI testing
A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Coverage criteria for GUI testing
ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineeringA widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Towards a pattern language for model-based GUI testing
EuroPLoP '14: Proceedings of the 19th European Conference on Pattern Languages of ProgramsGraphical user interfaces (GUIs) have become popular as they appear in everydays' software. GUIs have become an ideal way of interacting with computer programs, making the software friendlier to its users. GUIs have grown, and so has the usage of UI ...
Comments