Evaluation of the Relevance of Warnings Reported by Defect Detection Tools Based on Static Analysis

• João Eduardo Montandon PUC Minas
• Marco Túlio Valente UFMG

Abstract

Despite the interest and the increasing number of static analysis tools, there is still no consensus on the actual gains that such tools can introduce in software development projects. In this work, we report the results of two extensive case studies involving major open source systems and with the following central goals: (a) to evaluate the relevance of the warnings reported by static analysis tools; (b) to investigate possible correlations between the warnings reported by such tools and field defects.

References

Nathaniel Ayewah, David Hovemeyer, J. David Morgenthaler, John Penix, and William Pugh. Using static analysis to find bugs. IEEE Software, 25(5), 2008.

Nathaniel Ayewah, William Pugh, J. David Morgenthaler, John Penix, and YuQian Zhou. Evaluating static analysis defect warnings on production software. In 7th Workshop on Program Analysis for Software Tools and Engineering (PASTE), pages 1–8, 2007.

Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. A few billion lines of code later: Using static analysis to find bugs in the real world. Communications of the ACM, 53(2):66–75, 2010.

Tom Copeland. PMD Applied. Centennial Books, 2005.

João Eduardo Montandon de Araujo Filho, César Francisco de Moura Couto, Marco Túlio Valente, and Silvio José de Souza. Um estudo sobre a correlação entre defeitos de campo e warnings reportados por uma ferramenta de análise estática. IX Simpósio Brasileiro de Qualidade de Software, pages 9–23, 2010.

João Eduardo Montandon de Araujo Filho, Sílvio José de Souza, and Marco Túlio Valente. Os defeitos detectados pela ferramenta de análise estática FindBugs são relevantes? IX Simpósio Brasileiro de Qualidade de Software, pages 383–390, 2010.

João Eduardo Montandon de Araujo Filho, Marco Túlio Valente, and Silvio José de Souza. A study on the relevance of the warnings reported by Java bug finding tools. IET Software, pages 1–27, 2011 (to appear).

Jeffrey S. Foster, Michael W. Hicks, and William Pugh. Improving software quality with static analysis. In 7th Workshop on Program Analysis for Software Tools and Engineering (PASTE), pages 83–84, 2007.

C. W. J. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 37(3):424–438, 1969.

David Hovemeyer and William Pugh. Finding bugs is easy. SIGPLAN Notices, 39(12):92–106, 2004.

James R. Larus, Thomas Ball, Manuvir Das, Robert DeLine, Manuel Fahndrich, Jon Pincus, Sriram K. Rajamani, and Ramanathan Venkatapathy. Righting software. IEEE Software, 21(3):92–100, 2004.

Panagiotis Louridas. Static code analysis. IEEE Software, 23(4):58–61, 2006.

Peter Sprent and Nigel C. Smeeton. Applied Nonparametric Statistical Methods. Chapman & Hall, 2007.

Stefan Wagner, Michael Aichner, Johann Wimmer, and Markus Schwalb. An evaluation of two bug pattern tools for Java. In 1st International Conference on Software Testing, Verification, and Validation (ICST), pages 248–257, 2008.

Jiang Zheng, Laurie Williams, Nachiappan Nagappan, John P. Hudepohl, and Mladen A. Vouk. On the value of static analysis for fault detection in software. IEEE Transactions on Software Engineering, 32(4), 2006.