The Vocabulary of Flaky Tests in Swift

• João Medeiros UFPE
• Denini Silva UFRPE
• Breno Miranda UFPE

DOI: https://doi.org/10.5753/sast.2026.31085

Resumo

Flaky tests produce non-deterministic outcomes without code change, eroding CI confidence and delaying deliveries. While vocabularybased machine learning prediction has proven effective for Java and JavaScript, no study has evaluated it for Swift, a language whose testing style is dominated by UI and asynchronous code. We collect 91 flaky and 22,349 stable tests from 15 open-source Swift projects via re-execution and commit-history mining, then train five classifiers (Random Forest, Decision Tree, Naive Bayes, SVM, KNN) on TF-IDF unigram+bigram features under stratified 5-fold cross-validation. Random Forest achieves the best performance (Precision = 0.92, F1 = 0.86, AUC= 0.95) and substantially outperforms trivial baselines, among them a vocabulary-threshold rule applied to the most informative tokens, confirming a genuine discriminative signal (MCC = 0.75 vs. 0.08 for the best baseline). Information-gain analysis reveals two complementary signal types. Flakiness markers appear predominantly in unstable tests and comprise concurrency primitives (async, await), expectation-based synchronisation (expectation, fulfill), error propagation (throws), and explicit timing dependence (timeout, wait, now). Stability markers, chiefly the assertion vocabulary of plainly synchronous tests (xctassertequal), count as evidence against flakiness. Error analysis shows that the model fails when flakiness is hidden in shared infrastructure outside the test body or when async constructs are used in a deterministic context, exposing the intrinsic limit of lexical prediction. These results extend vocabulary-based flakiness detection to the Swift ecosystem and characterise both its effectiveness and its boundaries.

Palavras-chave: Flaky Tests, Unstable Tests, Machine Learning, Flakiness, Swift, iOS

Referências

Azeem Ahmad, Xin Sun, Muhammad Rashid Naeem, Yasir Javed, Mohammad Akour, and Kristian Sandahl. 2025. Understanding Flaky Tests Through Linguistic Diversity: A Cross-Language and Comparative Machine Learning Study. IEEE Access 13 (2025).

Abdulrahman Alshammari, Christopher Morris, Michael Hilton, and Jonathan Bell. 2021. FlakeFlagger: Predicting Flakiness Without Rerunning Tests. In 43rd International Conference on Software Engineering.

Keila Barbosa, Ronivaldo Ferreira, Gustavo Pinto, Marcelo d’Amorim, and Breno Miranda. 2022. Test flakiness across programming languages. IEEE Transactions on Software Engineering (2022).

Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. 2018. DeFlaker: automatically detecting flaky tests. In 40th International Conference on Software Engineering.

Alexander Berndt, Vekil Bekmyradov, Rainer Gemulla, Marcus Kessel, Thomas Bach, and Sebastian Baltes. 2026. Can We Classify Flaky Tests Using Only Test Code? An LLM-Based Empirical Study. arXiv preprint arXiv:2602.05465 (2026).

Sentry Cocoa Contributors. 2026. [link].

Moritz Eck, Fabio Palomba, Marco Castelluccio, and Alberto Bacchelli. 2019. Understanding flaky tests: the developer’s perspective. In 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.

Sakina Fatima, Taher A Ghaleb, and Lionel Briand. 2022. Flakify: A black-box, language model-based predictor for flaky tests. IEEE TSE (2022).

Martin Gruber and Gordon Fraser. 2022. A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 82–92.

Kim Herzig and Nachiappan Nagappan. 2015. Empirically detecting false test alarms using association rules. In 37th International Conference on Software Engineering - Volume 2.

Mozilla iOS Contributors. 2025. Pull Request #27270. [link].

Gregory M. Kapfhammer. 2004. Software testing. In The Computer Science Handbook. CRC Press.

Wing Lam, Kıvanç Muşlu, Hitesh Sajnani, and Suresh Thummalapenta. 2020. A study on the lifecycle of flaky tests. In 42nd International Conference on Software Engineering.

Wing Lam, Stefan Winter, Angello Astorga, Victoria Stodden, and Darko Marinov. 2020. Understanding Reproducibility and Characteristics of Flaky Tests Through Test Reruns in Java Projects. In IEEE 31st International Symposium on Software Reliability Engineering (ISSRE).

Wing Lam, Stefan Winter, Angello Astorga, Victoria Stodden, and Darko Marinov. 2020. Understanding reproducibility and characteristics of flaky tests through test reruns in java projects (Proceedings - International Symposium on Software Reliability Engineering, ISSRE). IEEE Computer Society.

Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering.

John Micco. 2017. The State of Continuous Integration Testing @Google.

Mozilla. 2026. Firefox for iOS: Open-SourceWeb Browser for iOS. [link]

Owain Parry, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. 2021. A Survey of Flaky Tests. ACM Trans. Softw. Eng. Methodol. 31, 1, Article 17 (2021).

Gustavo Pinto, Breno Miranda, Supun Dissanayake, Marcelo d’Amorim, Christoph Treude, and Antonia Bertolino. 2020. What is the Vocabulary of Flaky Tests?. In 17th International Conference on Mining Software Repositories.

Md Tajmilur Rahman and Peter C. Rigby. 2018. The impact of failing, flaky, and high failure tests on the number of crash reports associated with Firefox builds. In 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.

Shanto Rahman, Abdelrahman Baz, Sasa Misailovic, and August Shi. 2024. Quantizing large-language models for predicting flaky tests. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 93–104.

Shanto Rahman, Saikat Dutta, and August Shi. 2025. Understanding and improving flaky test classification. Proceedings of the ACM on Programming Languages 9, OOPSLA2 (2025), 1345–1371.

Rafael Rampim Soratto and Marco Aurélio Graciotto Silva. 2023. Vocabulary of Flaky Tests in Javascript. In XXII Brazilian Symposium on Software Quality.

Alan Romano, Zihe Song, Sampath Grandhi,Wei Yang, andWeihangWang. 2021. An empirical analysis of UI-based flaky tests. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1585–1597.

Denini Silva, Leopoldo Teixeira, and Marcelo d’Amorim. 2020. Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 301–311.

Swapna Thorve, Chandani Sreshtha, and Na Meng. 2018. An empirical study of flaky tests in android apps. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 534–538.

Roberto Verdecchia, Emilio Cruciani, Breno Miranda, and Antonia Bertolino. 2021. Know You Neighbor: Fast Static Prediction of Test Flakiness. IEEE Access (2021).