Xception fault injection and robustness testing framework: a case-study of testing RTEMS
Resumo
Xception is an automated and comprehensive fault injection and robustness testing environment that enables accurate and flexible V&V (verification & validation) and evaluation of mission and business critical computer systems and computer components, with particular emphasis to software components. In this paper we focus on the new robustness testing features of Xception and illustrate them with a concrete example of robustness testing of the Real Time Executive for Multiprocessor Systems (RTEMS) performed under a European Space Agency (ESA) contract. To the best of our knowledge, this is the first time that robustness testing results for this real time operating system are presented. The testing revealed a significant number of critical flaws in RTEMS 4.5.0 and shows the effectiveness of Xception toolset.
Referências
Arlat J., Fabre J.-C., Rodríguez M., and Salles F., “Dependability of COTS Microkernel-based Systems”, IEEE Trans. on Computers, 51 (2) February 2002, pp 138-163.
Carreira J., Madeira H., and Silva J. G., "Xception: Software Fault Injection and Monitoring in Processor Functional Units", 5th IFIP Working Conference on Dependable Computing for Critical Applications, DCCA-5, Urbana-Champaign, Illinois, USA, September 27-29, 1995.
Carreira J., Madeira H., and Silva J. G., “Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers”, IEEE Trans. on Software Engineering, 24 (2), pp.125-36, February 1998.
Koopman P. and DeVale J., “Comparing the Robustness of POSIX Operating Systems”, in Proc. 29th Int. Symposium on Fault-Tolerant Computing (FTCS-29), (Madison, WI, USA), pp.30-7, IEEE CS Press, 1999.
Koopman P., Sung J., Dingman C., Siewiorek D., Marz T., “Comparing Operating Systems using Robustness Benchmarks”, in Proceedings of the 16th International Symposium on Reliable Distributed Systems, SRDS-16, Durham, NC, USA, 1997.
Kropp, N., Koopman, P., and Siewiorek, D., "Automated Robustness Testing of Off-the_Shelf Software Components," 28th Fault Tolerant Computing Symposium, June 23-25, 1998.
Madeira H., Carreira J., and Silva J. G., “Injection of faults in complex computers", Fourth IEEE International Workshop on Evaluation Techniques for Dependable Systems, San Antonio, Texas, USA, October 2-3, 1995.
Maia R. et al, “RTEMS 4.5.0 Evaluation Report”, CSW-RAMS-2003-CTR-1306, RAMS Call-off Order 2, ESTEC/Contract No 16582/02/NL/PA, 2004 (available on specific request to rmaia@criticalsoftware.com).
Moraes, R. and Martins, E. “An Architecture-based Strategy for Interface Fault Injection”, Workshop on Architecting Dependable Systems, IEEE/IFIP International Conf. on Dependable Systems and Networks, Florence, Italy, June 28 – July 1, 2004.
Rodríguez M., Salles F., Fabre J.-C., and Arlat J., “MAFALDA: Microkernel Assessment by Fault Injection and Design Aid”, in Proc. 3rd European Dependable Computing Conf. (EDCC-3), (E. M. J. Hlavicka, A. Pataricza, Ed.), (Prague, Czech Republic), LNCS, 1667, pp.143-60, Springer, 1999.
Tsai T. and Iyer R. K., “An Approach to Benchmarking of Fault-Tolerant Commercial Systems”, Proceedings of the 26th IEEE Fault Tolerant Computing Symposium, FTCS-26, Sendai, Japan, pp. 314-323, June 1996.
Voas J. and McGraw G. “Software Fault Injection: Inoculating Programs against Errors”, John Wiley & Sons, New York, EUA, 1998.
