Flaky Tests in UI: Understanding Causes and Applying Correction Strategies
Resumo
Flaky tests are tests with non-deterministic behavior that can produce random and inconsistent results. While researchers has investigated Flaky tests in traditional unit tests, less is known about their occurrence in user interface (UI) tests. This study proposes an empirical analysis of 24 open-source projects to identify the main causes and correction strategies for Flaky tests in UI tests. Our analysis identified 8 categories of causes and 7 categories of correction strategies, with Race Condition, Logic Issues, and Test Dependency being the main causes of flaky tests in UI tests. Addition of Wait, Correction of Logic, and Ignored Test were the most commonly applied correction categories for resolving Flaky tests in UI tests. Specifically, 89% of Flaky tests caused by a Race Condition were corrected by adding Wait, while 100% of Flaky tests caused by Test Logic and Dependency Issues were fixed by patching the test logic. Our results provide insights into the occurrence of flaky tests in UI tests and can help Test Analysts and testers develop UI test automation projects with higher quality. This study contributes to the literature on flaky tests by providing empirical evidence of their occurrence in UI tests and identifying their main causes and correction strategies.
