Addressing the Synchronization Challenge in Cypress End-to-End Tests
Resumo
Automated end-to-end testing plays a crucial role in modern web software projects, helping testers identify faults within complex applications and shorten development cycles. Frameworks such as Cypress are essential to provide a comprehensive testing environment with features that facilitate better access and validation of page elements. However, time-related challenges (synchronization issues) remain a significant concern in such suites. Testers need to be aware of these challenges and employ appropriate waiting mechanisms to ensure test reliability. This paper presents a catalog of waiting mechanisms for Cypress tests and a set of empirical studies that investigate the potential impact of synchronization issues and waiting mechanisms on test suites. Our studies examine the suites of an open-source and industrial project. Our findings reveal that up to 32% of a suite can break due to synchronization issues, exposing flaky tests. Subsequently, we revised the suites by applying four waiting mechanisms (Static Wait, Stable DOM Wait, Network Wait, and Explicit Wait). Network Wait and Explicit Wait emerged as the most promising strategies leading to no breakages.
Palavras-chave:
end-to-end testing, flaky tests, synchronization, cypress, empirical studies
Referências
Nauman Bin Ali, Emelie Engström, Masoumeh Taromirad, Mohammad Reza Mousavi, Nasir Mehmood Minhas, Daniel Helgesson, Sebastian Kunze, and Mahsa Varshosaz. 2019. On the search for industry-relevant regression testing research. Empirical Software Engineering 24 (2019), 2020–2055.
Andrea Arcuri and Lionel Briand. 2014. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24, 3 (2014), 219–250.
Satya Avasarala. 2014. Selenium WebDriver practical guide. PACKT publishing.
Axel Bons, Beatriz Marín, Pekka Aho, and Tanja EJ Vos. 2023. Scripted and scriptless GUI testing for web applications: An industrial case. Information and Software Technology 158 (2023), 107172.
Maura Cerioli, Maurizio Leotta, and Filippo Ricca. 2020. What 5 million job advertisements tell us about testing: a preliminary empirical investigation. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 1586–1594.
Diego Clerissi, Maurizio Leotta, Gianna Reggio, and Filippo Ricca. 2017. Towards the generation of end-to-end web test scripts from requirements specifications. In 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW). IEEE, 343–350.
Cypress.io. 2018/2023. Key Differences. [link]. Accessed: 10-10-2023.
Edward Dunn Ekelund and Emelie Engström. 2015. Efficient regression testing based on test history: An industrial evaluation. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 449–457.
Sidong Feng, Mulong Xie, and Chunyang Chen. 2023. Efficiency Matters: Speeding Up Automated Testing with GUI Rendering Inference. DOI: 10.1109/icse48619.2023.00084
Cypress framework. 2018/2023. Why Cypress? [link]. Accessed: 10-10-2023.
Boni Garcia. 2022. Hands-On Selenium WebDriver with Java. " O’Reilly Media, Inc.".
Boni García, Micael Gallego, Francisco Gortázar, and Mario Munoz-Organero. 2020. A survey of the selenium ecosystem. Electronics 9, 7 (2020), 1067.
Sarra Habchi, Guillaume Haben, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2022. A qualitative study on the sources, impacts, and mitigation strategies of flaky tests. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 244–255.
Mouna Hammoudi, Gregg Rothermel, and Andrea Stocco. 2016. WATERFALL: an incremental approach for repairing record-replay tests of web applications. DOI: 10.1145/2950290.2950294
Mouna Hammoudi, Gregg Rothermel, and Paolo Tonella. 2016. Why do Record/Replay Tests of Web Applications Break? DOI: 10.1109/icst.2016.16
Md Hossain, Hyunsook Do, and Ravi Eda. 2014. Regression testing for web applications using reusable constraint values. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 312–321.
Wing Lam, Kıvanç Muşlu, Hitesh Sajnani, and Suresh Thummalapenta. 2020. A study on the lifecycle of flaky tests. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1471–1482.
Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2016. Approaches and tools for automated end-to-end web testing. In Advances in Computers. Vol. 101. Elsevier, 193–237.
Maurizio Leotta, Boni García, Filippo Ricca, and Jim Whitehead. 2023. Challenges of end-to-end testing with selenium WebDriver and how to face them: A survey. In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 339–350.
Xinyue Liu, Zihe Song, Weike Fang, Wei Yang, and Weihang Wang. 2024. WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests. arXiv preprint arXiv:2402.09745 (2024).
Fatini Mobaraya, Shahid Ali, et al. 2019. Technical Analysis of Selenium and Cypress as Functional Automation Framework for Modern Web Application Testing. Department of Information Technology, AGI Institute, Auckland, New Zealand (2019).
Thiago Santos de Moura, Everton LG Alves, Hugo Feitosa de Figueirêdo, and Cláudio de Souza Baptista. 2023. Cytestion: Automated GUI Testing for Web Applications. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering. 388–397.
Waweru Mwaura. 2021. End-to-End Web Testing with Cypress: Explore techniques for automated frontend web testing with Cypress and JavaScript. Packt Publishing Ltd.
Takao Nakagawa, Kazuki Munakata, and Koji Yamamoto. 2019. Applying modified code entity-based regression test selection for manual end-To-end testing of commercial web applications. In 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 1–6.
Michel Nass, Emil Alégroth, and Robert Feldt. 2021. Why many challenges with GUI test automation (will) remain. Information and Software Technology 138 (2021), 106625.
Dario Olianas, Maurizio Leotta, and Filippo Ricca. 2022. SleepReplacer: A novel tool-based approach for replacing thread sleeps in selenium webdriver test code. Software Quality Journal 30, 4 (2022), 1089–1121.
Dario Olianas, Maurizio Leotta, Filippo Ricca, and Luca Villa. 2021. Reducing flakiness in End-to-End test suites: An experience report. In International Conference on the Quality of Information and Communications Technology. Springer, 3–17.
Ana CR Paiva, Nuno H Flores, João P Faria, and José MG Marques. 2018. End-toend automatic business process validation. Procedia Computer Science 130 (2018), 999–1004.
Narayanan Palani. 2021. Automated Software Testing with Cypress. CRC Press.
Owain Parry, GregoryMKapfhammer, Michael Hilton, and Phil McMinn. 2021. A survey of flaky tests. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1–74.
Yu Pei, Sarra Habchi, Renaud Rwemalika, Jeongju Sohn, and Mike Papadakis. 2022. An empirical study of async wait flakiness in front-end testing.. In BENEVOL.
Yu Pei, Jeongju Sohn, Sarra Habchi, and Mike Papadakis. 2023. TRaf: Timebased Repair for Asynchronous Wait Flaky Tests in Web Testing. arXiv preprint arXiv:2305.08592 (2023).
Kai Presler-Marshall, Eric Horton, Sarah Heckman, and Kathryn Stolee. 2019. Wait, wait. no, tell me. analyzing selenium configuration effects on test flakiness. In 2019 IEEE/ACM 14th International Workshop on Automation of Software Test (AST). IEEE, 7–13.
Sujay Raghavendra. 2021. Python Testing with Selenium: Learn to Implement Different Testing Techniques Using the Selenium WebDriver. Springer.
Filippo Ricca, Maurizio Leotta, and Andrea Stocco. 2019. Three Open Problems in the Context of E2E Web Testing and a Vision: NEONATE. , 89–133 pages. DOI: 10.1016/bs.adcom.2018.10.005
Alan Romano, Zihe Song, Sampath Grandhi,Wei Yang, andWeihangWang. 2021. An empirical analysis of UI-based flaky tests. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1585–1597.
Érica Sousa, Carla Bezerra, and Ivan Machado. 2023. Flaky Tests in UI: Understanding Causes and Applying Correction Strategies. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering. 398–406.
Andrea Stocco, Rahulkrishna Yandrapally, and Ali Mesbah. 2018. Visual web test repair. DOI: 10.1145/3236024.3236063
Tanja EJ Vos, Pekka Aho, Fernando Pastor Ricos, Olivia Rodriguez-Valdes, and Ad Mulders. 2021. testar–scriptless testing through graphical user interface. Software Testing, Verification and Reliability 31, 3 (2021), e1771.
Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering. 1–10.
Andrea Arcuri and Lionel Briand. 2014. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24, 3 (2014), 219–250.
Satya Avasarala. 2014. Selenium WebDriver practical guide. PACKT publishing.
Axel Bons, Beatriz Marín, Pekka Aho, and Tanja EJ Vos. 2023. Scripted and scriptless GUI testing for web applications: An industrial case. Information and Software Technology 158 (2023), 107172.
Maura Cerioli, Maurizio Leotta, and Filippo Ricca. 2020. What 5 million job advertisements tell us about testing: a preliminary empirical investigation. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 1586–1594.
Diego Clerissi, Maurizio Leotta, Gianna Reggio, and Filippo Ricca. 2017. Towards the generation of end-to-end web test scripts from requirements specifications. In 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW). IEEE, 343–350.
Cypress.io. 2018/2023. Key Differences. [link]. Accessed: 10-10-2023.
Edward Dunn Ekelund and Emelie Engström. 2015. Efficient regression testing based on test history: An industrial evaluation. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 449–457.
Sidong Feng, Mulong Xie, and Chunyang Chen. 2023. Efficiency Matters: Speeding Up Automated Testing with GUI Rendering Inference. DOI: 10.1109/icse48619.2023.00084
Cypress framework. 2018/2023. Why Cypress? [link]. Accessed: 10-10-2023.
Boni Garcia. 2022. Hands-On Selenium WebDriver with Java. " O’Reilly Media, Inc.".
Boni García, Micael Gallego, Francisco Gortázar, and Mario Munoz-Organero. 2020. A survey of the selenium ecosystem. Electronics 9, 7 (2020), 1067.
Sarra Habchi, Guillaume Haben, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2022. A qualitative study on the sources, impacts, and mitigation strategies of flaky tests. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 244–255.
Mouna Hammoudi, Gregg Rothermel, and Andrea Stocco. 2016. WATERFALL: an incremental approach for repairing record-replay tests of web applications. DOI: 10.1145/2950290.2950294
Mouna Hammoudi, Gregg Rothermel, and Paolo Tonella. 2016. Why do Record/Replay Tests of Web Applications Break? DOI: 10.1109/icst.2016.16
Md Hossain, Hyunsook Do, and Ravi Eda. 2014. Regression testing for web applications using reusable constraint values. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 312–321.
Wing Lam, Kıvanç Muşlu, Hitesh Sajnani, and Suresh Thummalapenta. 2020. A study on the lifecycle of flaky tests. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1471–1482.
Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2016. Approaches and tools for automated end-to-end web testing. In Advances in Computers. Vol. 101. Elsevier, 193–237.
Maurizio Leotta, Boni García, Filippo Ricca, and Jim Whitehead. 2023. Challenges of end-to-end testing with selenium WebDriver and how to face them: A survey. In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 339–350.
Xinyue Liu, Zihe Song, Weike Fang, Wei Yang, and Weihang Wang. 2024. WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests. arXiv preprint arXiv:2402.09745 (2024).
Fatini Mobaraya, Shahid Ali, et al. 2019. Technical Analysis of Selenium and Cypress as Functional Automation Framework for Modern Web Application Testing. Department of Information Technology, AGI Institute, Auckland, New Zealand (2019).
Thiago Santos de Moura, Everton LG Alves, Hugo Feitosa de Figueirêdo, and Cláudio de Souza Baptista. 2023. Cytestion: Automated GUI Testing for Web Applications. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering. 388–397.
Waweru Mwaura. 2021. End-to-End Web Testing with Cypress: Explore techniques for automated frontend web testing with Cypress and JavaScript. Packt Publishing Ltd.
Takao Nakagawa, Kazuki Munakata, and Koji Yamamoto. 2019. Applying modified code entity-based regression test selection for manual end-To-end testing of commercial web applications. In 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 1–6.
Michel Nass, Emil Alégroth, and Robert Feldt. 2021. Why many challenges with GUI test automation (will) remain. Information and Software Technology 138 (2021), 106625.
Dario Olianas, Maurizio Leotta, and Filippo Ricca. 2022. SleepReplacer: A novel tool-based approach for replacing thread sleeps in selenium webdriver test code. Software Quality Journal 30, 4 (2022), 1089–1121.
Dario Olianas, Maurizio Leotta, Filippo Ricca, and Luca Villa. 2021. Reducing flakiness in End-to-End test suites: An experience report. In International Conference on the Quality of Information and Communications Technology. Springer, 3–17.
Ana CR Paiva, Nuno H Flores, João P Faria, and José MG Marques. 2018. End-toend automatic business process validation. Procedia Computer Science 130 (2018), 999–1004.
Narayanan Palani. 2021. Automated Software Testing with Cypress. CRC Press.
Owain Parry, GregoryMKapfhammer, Michael Hilton, and Phil McMinn. 2021. A survey of flaky tests. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1–74.
Yu Pei, Sarra Habchi, Renaud Rwemalika, Jeongju Sohn, and Mike Papadakis. 2022. An empirical study of async wait flakiness in front-end testing.. In BENEVOL.
Yu Pei, Jeongju Sohn, Sarra Habchi, and Mike Papadakis. 2023. TRaf: Timebased Repair for Asynchronous Wait Flaky Tests in Web Testing. arXiv preprint arXiv:2305.08592 (2023).
Kai Presler-Marshall, Eric Horton, Sarah Heckman, and Kathryn Stolee. 2019. Wait, wait. no, tell me. analyzing selenium configuration effects on test flakiness. In 2019 IEEE/ACM 14th International Workshop on Automation of Software Test (AST). IEEE, 7–13.
Sujay Raghavendra. 2021. Python Testing with Selenium: Learn to Implement Different Testing Techniques Using the Selenium WebDriver. Springer.
Filippo Ricca, Maurizio Leotta, and Andrea Stocco. 2019. Three Open Problems in the Context of E2E Web Testing and a Vision: NEONATE. , 89–133 pages. DOI: 10.1016/bs.adcom.2018.10.005
Alan Romano, Zihe Song, Sampath Grandhi,Wei Yang, andWeihangWang. 2021. An empirical analysis of UI-based flaky tests. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1585–1597.
Érica Sousa, Carla Bezerra, and Ivan Machado. 2023. Flaky Tests in UI: Understanding Causes and Applying Correction Strategies. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering. 398–406.
Andrea Stocco, Rahulkrishna Yandrapally, and Ali Mesbah. 2018. Visual web test repair. DOI: 10.1145/3236024.3236063
Tanja EJ Vos, Pekka Aho, Fernando Pastor Ricos, Olivia Rodriguez-Valdes, and Ad Mulders. 2021. testar–scriptless testing through graphical user interface. Software Testing, Verification and Reliability 31, 3 (2021), e1771.
Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering. 1–10.
Publicado
30/09/2024
Como Citar
MOURA, Thiago Santos de; ALVES, Everton L. G.; FELIPE, Regina Letícia Santos; BAPTISTA, Cláudio de Souza; SILVA NETO, Ismael Raimundo da; FIGUEIRÊDO, Hugo Feitosa de.
Addressing the Synchronization Challenge in Cypress End-to-End Tests. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SOFTWARE (SBES), 38. , 2024, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 92-102.
DOI: https://doi.org/10.5753/sbes.2024.3298.
