Timing Sensitivity in the Parsl Test Suite: A Study Based on Controlled Delay Injection
Resumo
Testing parallel and distributed software is challenging because test outcomes may depend on scheduling, asynchronous communication, timeouts, and the relative progress of threads and processes. In this paper, we investigate timing sensitivity in the test suite of Parsl, a Python library for parallel and distributed computing. We first evaluate controlled delay injection for exposing timing-sensitive tests by perturbing concurrent execution components and observing the resulting failures. We then analyze the exposed tests to determine the causes of their failures and perform a manual analysis of the complete test suite to identify additional tests whose behavior depends on timing and to characterize the mechanisms used to mitigate nondeterminism, such as condition-based polling, wall-clock constraints, and fixed sleeps. Our study provides a characterization of timing-sensitive tests in a real-world parallel runtime and examines how controlled execution delays affect test outcomes. The results showed that controlled delay injection exposed 7 tests among the 536 test cases evaluated, while manual analysis identified 21 timing-sensitive tests. These findings provide insights into common sources of timing dependency and demonstrate how controlled delay injection can be used to study timing-sensitive tests in parallel and distributed software.Referências
Arvanitou, E.-M., Ampatzoglou, A., Chatzigeorgiou, A., and Carver, J. C. (2021). Software engineering practices for scientific software development: A systematic mapping study. Journal of Systems and Software, 172:110848.
Babuji, Y., Woodard, A., Li, Z., Katz, D. S., Clifford, B., Kumar, R., Lacinski, L., Chard, R., Wozniak, J., Foster, I., Wilde, M., and Chard, K. (2019). Parsl: Pervasive parallel programming in python. In 28th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC).
Basili, V. R., Carver, J. C., Cruzes, D., Hochstein, L. M., Hollingsworth, J. K., Shull, F., and Zelkowitz, M. V. (2008). Understanding the high-performance-computing community: A software engineer’s perspective. IEEE Software, 25(4):29–36.
Deligiannis, P., McCutchen, M., Thomson, P., Chen, S., Donaldson, A. F., Erickson, J., Huang, C., Lal, A., Mudduluru, R., Qadeer, S., and Schulte, W. (2016). Uncovering bugs in distributed storage systems during testing (not in production!). In Proceedings of the 14th Usenix Conference on File and Storage Technologies, FAST’16, page 249–262, USA. USENIX Association.
Eytani, Y. and Latvala, T. (2006). Explaining intermittent concurrent bugs by minimizing scheduling noise. volume 4383, pages 183–197.
Hintjens, P. (2013). ZeroMQ: Messaging for Many Applications. O’Reilly Media.
Křena, B., Letko, Z., and Vojnar, T. (2012). Noise injection heuristics for concurrency testing. In Kotásek, Z., Bouda, J., Černá, I., Sekanina, L., Vojnar, T., and Antoš, D., editors, Mathematical and Engineering Methods in Computer Science, pages 123–135, Berlin, Heidelberg. Springer Berlin Heidelberg.
Leesatapornwongsa, T., Ren, X., and Nath, S. (2022). Flakerepro: automated and efficient reproduction of concurrency-related flaky tests. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 1509–1520, New York, NY, USA. Association for Computing Machinery.
Luo, Q., Hariri, F., Eloussi, L., and Marinov, D. (2014). An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, page 643–653, New York, NY, USA. Association for Computing Machinery.
Parsl Developers. Quickstart. [link]. Accessed: June 2026.
Parsl Project. Parsl: Parallel scripting in python. [link]. Accessed: 2026-08-06.
Rahman, S., Massey, A., Lam, W., Shi, A., and Bell, J. (2024). Automatically reproducing timing-dependent flaky-test failures. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 269–280.
Silva, D., Teixeira, L., and d’Amorim, M. (2020). Shake it! detecting flaky tests caused by concurrency with shaker. pages 301–311.
Babuji, Y., Woodard, A., Li, Z., Katz, D. S., Clifford, B., Kumar, R., Lacinski, L., Chard, R., Wozniak, J., Foster, I., Wilde, M., and Chard, K. (2019). Parsl: Pervasive parallel programming in python. In 28th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC).
Basili, V. R., Carver, J. C., Cruzes, D., Hochstein, L. M., Hollingsworth, J. K., Shull, F., and Zelkowitz, M. V. (2008). Understanding the high-performance-computing community: A software engineer’s perspective. IEEE Software, 25(4):29–36.
Deligiannis, P., McCutchen, M., Thomson, P., Chen, S., Donaldson, A. F., Erickson, J., Huang, C., Lal, A., Mudduluru, R., Qadeer, S., and Schulte, W. (2016). Uncovering bugs in distributed storage systems during testing (not in production!). In Proceedings of the 14th Usenix Conference on File and Storage Technologies, FAST’16, page 249–262, USA. USENIX Association.
Eytani, Y. and Latvala, T. (2006). Explaining intermittent concurrent bugs by minimizing scheduling noise. volume 4383, pages 183–197.
Hintjens, P. (2013). ZeroMQ: Messaging for Many Applications. O’Reilly Media.
Křena, B., Letko, Z., and Vojnar, T. (2012). Noise injection heuristics for concurrency testing. In Kotásek, Z., Bouda, J., Černá, I., Sekanina, L., Vojnar, T., and Antoš, D., editors, Mathematical and Engineering Methods in Computer Science, pages 123–135, Berlin, Heidelberg. Springer Berlin Heidelberg.
Leesatapornwongsa, T., Ren, X., and Nath, S. (2022). Flakerepro: automated and efficient reproduction of concurrency-related flaky tests. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 1509–1520, New York, NY, USA. Association for Computing Machinery.
Luo, Q., Hariri, F., Eloussi, L., and Marinov, D. (2014). An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, page 643–653, New York, NY, USA. Association for Computing Machinery.
Parsl Developers. Quickstart. [link]. Accessed: June 2026.
Parsl Project. Parsl: Parallel scripting in python. [link]. Accessed: 2026-08-06.
Rahman, S., Massey, A., Lam, W., Shi, A., and Bell, J. (2024). Automatically reproducing timing-dependent flaky-test failures. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 269–280.
Silva, D., Teixeira, L., and d’Amorim, M. (2020). Shake it! detecting flaky tests caused by concurrency with shaker. pages 301–311.
Publicado
03/11/2026
Como Citar
ALMEIDA, Arthur Prince de; GOLDMAN, Alfredo.
Timing Sensitivity in the Parsl Test Suite: A Study Based on Controlled Delay Injection. In: SIMPÓSIO EM SISTEMAS COMPUTACIONAIS DE ALTO DESEMPENHO (SSCAD), 27. , 2026, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 43-55.
DOI: https://doi.org/10.5753/sscad.2026.34359.
