Uma avaliação do potencial de detecção de defeitos dos casos de teste de robustez de acordo com a análise de mutantes
Resumo
Testar se o sistema satisfaz as funcionalidades especificadas é uma etapa necessária, mas não é suficiente. Importante é também determinar como o sistema se comporta em presença de entradas inválidas ou não esperadas (inoportunas). Este é o objetivo dos testes de robustez. Caso não sejam detectadas durante a fase de testes, as falhas de robustez (robustness faults) podem ocorrer durante a operação do sistema, quando são mais difíceis de diagnosticar e corrigir, e podem levar à interrupção das operações ou ao mau funcionamento do sistema. Neste trabalho analisamos o potencial de detecção de defeitos (faults) de sequências de teste nominais e sequências de teste de robustez. As sequências nominais contêm somente entradas especificadas, enquanto as sequências de robustez contêm também entradas inoportunas. As sequências são geradas a partir de modelos de estado da UML representando o comportamento funcional do sistema. Análise de mutantes foi utilizada para avaliar o potencial de detecção de falhas das sequências geradas.
Referências
Acree Jr, A. T. (1980). On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta School of Information and Computer Science.
Aichernig, B., Brandl, H., Jobstl, E., Krenn, W., Schlick, R., and Tiran, S. (2015). Momut:: Uml model-based mutation testing for uml. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on, pages 1–8. IEEE.
Ambrosio, A. M., Mattiello-Francisco, F., Santiago Jr, V. A., Silva, W. P., and Martins, E. (2007). Designing fault injection experiments using state-based model to test a space software. In Dependable computing, pages 170–178. Springer.
Cardoso, W. F. F. (2015). Statemutest: uma ferramenta de apoio ao teste baseado em modelos de estado estendidos. Master’s thesis, Universidade Estadual de Campinas.
Cotroneo, D., Di Leo, D., Fucci, F., and Natella, R. (2013). Sabrine: State-based robustness testing of operating systems. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, pages 125–135. IEEE.
DeMillo, R. and Offutt, A. J. (1991). Constraint-based automatic test data generation. Software Engineering, IEEE Transactions on, 17(9):900–910.
Fabbri, S. C., Delamaro, M. E., Maldonado, J. C., and Masiero, P. C. (1994). Mutation analysis testing for finite state machines. In Software Reliability Engineering, 1994. Proceedings., 5th International Symposium on, pages 220–229. IEEE.
Fabbri, S. C., Maldonado, J. C., and Delamaro, M. E. (1999a). Proteum/fsm: a tool to support finite state machine validation based on mutation testing. In Computer Science Society, 1999. Proceedings. SCCC’99. XIX International Conference of the Chilean, pages 96–104. IEEE.
Fabbri, S. C. P. F., Maldonado, J. C., Sugeta, T., and Masiero, P. C. (1999b). Mutation testing applied to validate specifications based on statecharts. In Software Reliability Engineering, 1999. Proceedings. 10th International Symposium on, pages 210–219. IEEE.
Harman, M., Jia, Y., and Langdon, W. B. (2011). Strong higher order mutation-based test data generation. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 212–222. ACM.
Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. Software Engineering, IEEE Transactions on, 37(5):649–678.
Kalaji, A., Hierons, R. M., and Swift, S. (2009). Generating feasible transition paths for testing from an extended finite state machine (efsm). In Software Testing Verification and Validation, 2009. ICST’09. International Conference on, pages 230–239. IEEE.
Kim, Y. G., Hong, H. S., Bae, D.-H., and Cha, S. D. (1999). Test cases generation from uml state diagrams. In Software, IEE Proceedings-, volume 146, pages 187–192. IET.
Martins, E., Sabião, S. B., and Ambrosio, A. M. (1999). Condata: a tool for automating specification-based test case generation for communication systems. Software Quality Journal, 8(4):303–320.
Micskei, Z., Madeira, H., Avritzer, A., Majzik, I., Vieira, M., and Antunes, N. (2012). Robustness testing techniques and tools. In Resilience Assessment and Evaluation of Computing Systems, pages 323–339. Springer.
Offutt, A. J., Rothermel, G., and Zapf, C. (1993). An experimental evaluation of selective mutation. In Proceedings of the 15th international conference on Software Engineering, pages 100–107. IEEE Computer Society Press.
Papadakis, M. and Malevris, N. (2010). Automatic mutation test case generation via dynamic symbolic execution. In Software reliability engineering (ISSRE), 2010 IEEE 21st international symposium on, pages 121–130. IEEE.
Prenninger, W. and Pretschner, A. (2005). Abstractions for model-based testing. Electronic Notes in Theoretical Computer Science, 116:59–71.
Rollet, A. and Salva, S. (2009). Testing robustness of communicating systems using ioco-based approach. In Computers and Communications, 2009. ISCC 2009. IEEE Symposium on, pages 67–72. IEEE.
Rumbaugh, J., Jacobson, I., and Booch, G. (2004). Unified Modeling Language Reference Manual, The. Pearson Higher Education.
Saad-Khorchef, F., Rollet, A., and Castanet, R. (2007). A framework and a tool for robustness testing of communicating software. In Proceedings of the 2007 ACM symposium on Applied computing, pages 1461–1466. ACM.
Samuel, P., Mall, R., and Bothra, A. K. (2008). Automatic test case generation using unified modeling language (uml) state diagrams. Software, IET, 2(2):79–93.
Santiago, V., Amaral, A. S. M. d., Vijaykumar, N. L., Mattiello-Francisco, M. F., Martins, E., and Lopes, O. C. (2006). A practical approach for automated test case generation using statecharts. In Computer Software and Applications Conference, 2006. COMPSAC’06. 30th Annual International, volume 2, pages 183–188. IEEE.
Shirole, M., Suthar, A., and Kumar, R. (2011). Generation of improved test cases from uml state diagram using genetic algorithm. In Proceedings of the 4th India Software Engineering Conference, pages 125–134. ACM.
Sugeta, T. (1999). PROTEUM-RS/ST: uma ferramenta para apoiar a validação de especificações statecharts baseada na análise de mutantes. PhD thesis, Universidade de São Paulo.
Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. (2011). Probability and statistics for engineers and scientists, volume 9. Prentice Hall.
Yano, T., Martins, E., and De Sousa, F. L. (2010). Generating feasible test paths from an executable model using a multi-objective approach. In Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, pages 236–239. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011a). A model-based approach for robustness test generation. In Dependable Computing Workshops (LADCW), 2011 Fifth Latin-American Symposium on, pages 33–34. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011b). Most: a multi-objective search-based testing from efsm. In Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on, pages 164–173. IEEE.
Zhang, J., Yang, R., Chen, Z., Zhao, Z., and Xu, B. (2012). Automated efsm-based test case generation with scatter search. In Proceedings of the 7th International Workshop on Automation of Software Test, pages 76–82. IEEE Press.
Aichernig, B., Brandl, H., Jobstl, E., Krenn, W., Schlick, R., and Tiran, S. (2015). Momut:: Uml model-based mutation testing for uml. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on, pages 1–8. IEEE.
Ambrosio, A. M., Mattiello-Francisco, F., Santiago Jr, V. A., Silva, W. P., and Martins, E. (2007). Designing fault injection experiments using state-based model to test a space software. In Dependable computing, pages 170–178. Springer.
Cardoso, W. F. F. (2015). Statemutest: uma ferramenta de apoio ao teste baseado em modelos de estado estendidos. Master’s thesis, Universidade Estadual de Campinas.
Cotroneo, D., Di Leo, D., Fucci, F., and Natella, R. (2013). Sabrine: State-based robustness testing of operating systems. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, pages 125–135. IEEE.
DeMillo, R. and Offutt, A. J. (1991). Constraint-based automatic test data generation. Software Engineering, IEEE Transactions on, 17(9):900–910.
Fabbri, S. C., Delamaro, M. E., Maldonado, J. C., and Masiero, P. C. (1994). Mutation analysis testing for finite state machines. In Software Reliability Engineering, 1994. Proceedings., 5th International Symposium on, pages 220–229. IEEE.
Fabbri, S. C., Maldonado, J. C., and Delamaro, M. E. (1999a). Proteum/fsm: a tool to support finite state machine validation based on mutation testing. In Computer Science Society, 1999. Proceedings. SCCC’99. XIX International Conference of the Chilean, pages 96–104. IEEE.
Fabbri, S. C. P. F., Maldonado, J. C., Sugeta, T., and Masiero, P. C. (1999b). Mutation testing applied to validate specifications based on statecharts. In Software Reliability Engineering, 1999. Proceedings. 10th International Symposium on, pages 210–219. IEEE.
Harman, M., Jia, Y., and Langdon, W. B. (2011). Strong higher order mutation-based test data generation. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 212–222. ACM.
Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. Software Engineering, IEEE Transactions on, 37(5):649–678.
Kalaji, A., Hierons, R. M., and Swift, S. (2009). Generating feasible transition paths for testing from an extended finite state machine (efsm). In Software Testing Verification and Validation, 2009. ICST’09. International Conference on, pages 230–239. IEEE.
Kim, Y. G., Hong, H. S., Bae, D.-H., and Cha, S. D. (1999). Test cases generation from uml state diagrams. In Software, IEE Proceedings-, volume 146, pages 187–192. IET.
Martins, E., Sabião, S. B., and Ambrosio, A. M. (1999). Condata: a tool for automating specification-based test case generation for communication systems. Software Quality Journal, 8(4):303–320.
Micskei, Z., Madeira, H., Avritzer, A., Majzik, I., Vieira, M., and Antunes, N. (2012). Robustness testing techniques and tools. In Resilience Assessment and Evaluation of Computing Systems, pages 323–339. Springer.
Offutt, A. J., Rothermel, G., and Zapf, C. (1993). An experimental evaluation of selective mutation. In Proceedings of the 15th international conference on Software Engineering, pages 100–107. IEEE Computer Society Press.
Papadakis, M. and Malevris, N. (2010). Automatic mutation test case generation via dynamic symbolic execution. In Software reliability engineering (ISSRE), 2010 IEEE 21st international symposium on, pages 121–130. IEEE.
Prenninger, W. and Pretschner, A. (2005). Abstractions for model-based testing. Electronic Notes in Theoretical Computer Science, 116:59–71.
Rollet, A. and Salva, S. (2009). Testing robustness of communicating systems using ioco-based approach. In Computers and Communications, 2009. ISCC 2009. IEEE Symposium on, pages 67–72. IEEE.
Rumbaugh, J., Jacobson, I., and Booch, G. (2004). Unified Modeling Language Reference Manual, The. Pearson Higher Education.
Saad-Khorchef, F., Rollet, A., and Castanet, R. (2007). A framework and a tool for robustness testing of communicating software. In Proceedings of the 2007 ACM symposium on Applied computing, pages 1461–1466. ACM.
Samuel, P., Mall, R., and Bothra, A. K. (2008). Automatic test case generation using unified modeling language (uml) state diagrams. Software, IET, 2(2):79–93.
Santiago, V., Amaral, A. S. M. d., Vijaykumar, N. L., Mattiello-Francisco, M. F., Martins, E., and Lopes, O. C. (2006). A practical approach for automated test case generation using statecharts. In Computer Software and Applications Conference, 2006. COMPSAC’06. 30th Annual International, volume 2, pages 183–188. IEEE.
Shirole, M., Suthar, A., and Kumar, R. (2011). Generation of improved test cases from uml state diagram using genetic algorithm. In Proceedings of the 4th India Software Engineering Conference, pages 125–134. ACM.
Sugeta, T. (1999). PROTEUM-RS/ST: uma ferramenta para apoiar a validação de especificações statecharts baseada na análise de mutantes. PhD thesis, Universidade de São Paulo.
Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. (2011). Probability and statistics for engineers and scientists, volume 9. Prentice Hall.
Yano, T., Martins, E., and De Sousa, F. L. (2010). Generating feasible test paths from an executable model using a multi-objective approach. In Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, pages 236–239. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011a). A model-based approach for robustness test generation. In Dependable Computing Workshops (LADCW), 2011 Fifth Latin-American Symposium on, pages 33–34. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011b). Most: a multi-objective search-based testing from efsm. In Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on, pages 164–173. IEEE.
Zhang, J., Yang, R., Chen, Z., Zhao, Z., and Xu, B. (2012). Automated efsm-based test case generation with scatter search. In Proceedings of the 7th International Workshop on Automation of Software Test, pages 76–82. IEEE Press.
Publicado
30/05/2016
Como Citar
CARDOSO, Wallace Felipe Francisco; MARTINS, Eliane.
Uma avaliação do potencial de detecção de defeitos dos casos de teste de robustez de acordo com a análise de mutantes. In: WORKSHOP DE TESTES E TOLERÂNCIA A FALHAS (WTF), 17. , 2016, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2016
.
p. 35-48.
ISSN 2595-2684.
DOI: https://doi.org/10.5753/wtf.2016.22874.
