Obtaining Trustworthy Test Results in Multi-threaded Systems

• Ayla Dantas UFCG
• Matheus Gaudencio UFCG
• Francisco Brasileiro UFCG
• Walfredo Cirne UFCG

DOI: https://doi.org/10.5753/sbes.2008.21322

Resumo

Testing multi-threaded systems is quite a challenge. The non-determinism of these systems makes their implementation and their test implementation far more susceptible to error. It is common to have tests of these systems that may not pass sometimes and whose failures are not caused by application faults (bugs). For instance, this can happen when test verifications (assertions) are performed at inappropriate times. Unreliable tests make developers waste their time trying to find non-existing bugs, or else make them search for bugs in the wrong place. Another problem is that developers may cease to believe that certain test failures are caused by software bugs even when this is the case. In this paper, we propose an approach to avoid test failures that are caused, not by application bugs, but by test assertions performed either too early or too late. Our basic idea is to use thread monitoring to support test development. Moreover, we put forward a testing framework that follows the proposed approach, and we also evaluate the effectiveness of this framework for testing a multi-threaded system.

Referências

Bertolino, A. (2007). Software testing research: Achievements, challenges, dreams. In FOSE ’07: 2007 Future of Software Engineering, pages 85–103, Washington, DC, USA. IEEE Computer Society.

Choi, J. and Srinivasan, H. (1998). Deterministic replay of Java multithreaded applications. Proceedings of the SIGMETRICS symposium on Parallel and distributed tools, pages 48–59.

Cirne, W., Brasileiro, F., Andrade, N., Costa, L., Andrade, A., Novaes, R., and Mowbray, M. (2006). Labs of the World, Unite!!! Journal of Grid Computing, 4(3):225–246.

Copty, S. and Ur, S. (2005). Multi-threaded Testing with AOP is Easy, and It Finds Bugs. Proc. 11th International Euro-Par Conference, LNCS, 3648:740–749.

Dantas, A., Cirne, W., and Saikoski, K. (2006). Using AOP to Bring a Project Back in Shape: The OurGrid Case. Journal of the Brazilian Computer Society (JBCS), 11:21– 36.

Dijkstra, E. (1969). Notes on structured programming.

Edelstein, O., Farchi, E., Nir, Y., Ratsaby, G., and Ur, S. (2002). Multithreaded Java program test generation. IBM Systems Journal, 41(1):111–125.

Goetz, B. and Peierls, T. (2006). Java concurrency in practice. Addison-Wesley.

Gosling, J. (2000). The Java Language Specification. Addison-Wesley Professional.

Harrold, M. (2000). Testing: a roadmap. Proceedings of the Conference on The Future of Software Engineering, pages 61–72.

Kiczales, G., Hilsdale, E., Hugunin, J., Kersten, M., Palm, J., and Griswold, W. (2001). Getting Started with AspectJ. Communications of the ACM, 44(10):59–65.

Kiczales, G., Lamping, J., Mendhekar, A., Maeda, C., Lopes, C. V., Loingtier, J.-M., and Irwin, J. (1997). Aspect–Oriented Programming. In European Conference on Object–Oriented Programming, ECOOP’97, LNCS 1241, pages 220–242, Finland. Springer–Verlag.

Lea, D. (2000). Concurrent Programming in Java (tm): Design Principles and Patterns. Addison-Wesley.

MacDonald, S., Chen, J., and Novillo, D. (2005). Choosing Among Alternative Futures. Proc. Haifa Verification Conference, LNCS, 3875:247–264.

Massol, V. (2004). JUnit in Action. Manning.

Meszaros, G. (2007). XUnit Test Patterns: Refactoring Test Code. Addison-Wesley.

Moon, S. and Chang, B. (2006). A thread monitoring system for multithreaded Java programs. ACM SIGPLAN Notices, 41(5):21–29.

Naik, M., Aiken, A., and Whaley, J. (2006). Effective static race detection for java. In PLDI ’06: Proceedings of the 2006 ACM SIGPLAN conference on Programming language design and implementation, pages 308–319, New York, NY, USA. ACM Press.

Oliveira, M. I. S. (2007). Ourbackup: Uma solucao p2p de backup baseada em redes sociais. Master’s thesis, Universidade Federal de Campina Grande.

Pugh, W. and Ayewah, N. (2007). Unit testing concurrent software. Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, pages 513–516.

Rinard, M. (2001). Analysis of multithreaded programs. Proceedings of the 8th International Symposium on Static Analysis, pages 1–19.

Rocha, A., Simão, A., Maldonado, J., and Masiero, P. (2005). Uma ferramenta baseada em aspectos para o teste funcional de programas Java. In Simpósio Brasileiro de Engenharia de Software, SBES 2005, Uberlândia-MG.

Ronsse, M. and De Bosschere, K. (1999). RecPlay: a fully integrated practical record/replay system. ACM Transactions on Computer Systems (TOCS), 17(2):133– 152.

Savage, S., Burrows, M., Nelson, G., Sobalvarro, P., and Anderson, T. (1997). Eraser: A Dynamic Data Race Detector for Multithreaded Programs. ACM Transactions on Computer Systems, 15(4):391–411.

Sen, K. (2007). Effective random testing of concurrent programs. Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, pages 323–332.

Stobie, K. (2005). Too darned big to test. Queue, 3(1):30–37.

Stoller, S. D. (2002). Testing concurrent Java programs using randomized scheduling. In Proc. Second Workshop on Runtime Verification (RV), volume 70(4) of Electronic Notes in Theoretical Computer Science. Elsevier.

Sutter, H. and Larus, J. (2005). Software and the concurrency revolution. Queue, 3(7):54–62.

United Binary, L. Harnessit test framework: Multithreaded testing considerations. At http://unittesting.com/documentation/Appendices/MultiThreadedTesting.html.