Assessing the Credibility of Grey Literature

A Study with Brazilian Software Engineering Researchers

Authors

DOI:

https://doi.org/10.5753/jserd.2022.1897

Keywords:

Grey Literature, Evidence-Based Software Engineering, Empirical Software Engineering

Abstract

In recent years, the use and investigations about Grey Literature (GL) increased, in particular, in Software Engineering (SE) research. However, its understanding is still scarce and sometimes controversial, such as interpreting GL types and assessing their credibility. This study aimed to understand the credibility aspects that SE researchers consider in assessing GL and its types. To achieve this goal, we surveyed 53 SE researchers (who answered that they have used GL in our previous investigation), receiving a total of 34 valid responses. Our main findings show that: 1) GL source produced or cited by a renowned source is the main credibility criteria used to assess GL, 2) most of the GL types tend to have a Low to Moderate level of Control and Expertise, 3) there is a positive statistical correlation between the level of Control and Expertise for most GL types, and 4) the different respondent profiles shared similar opinions about the credibility criteria. Our investigation contributes to helping future SE researchers that intend to use GL with more credibility. Additionally, shows the need for future studies to better understand the GL types in SE research.

Downloads

Download data is not yet available.

References

Adams, J., HillierBrown, F. C., Moore, H. J., Lake, A. A., Araujo Soares, V., and Summerbell, M. W. C. (2016a). Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Systematic Reviews, 5(1):164.

Adams, R. J., Smart, P., and Huff, A. S. (2016b). Shades of grey: Guidelines for working with the grey literature in systematic reviews for management and organizational studies. International Journal of Management Reviews, 19(4):432–454.

Aniche, M. F., Treude, C., Steinmacher, I., Wiese, I., Pinto, G., Storey, M. D., and Gerosa, M. A. (2018). How modern news aggregators help development communities shape and share knowledge. In Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, pages 499–510.

Berg, V., Birkeland, J., NguyenDuc, A., Pappas, I. O., and Jaccheri, L. (2018). Software startup engineering: A systematic mapping study. Journal of Systems and Software, 144:255–274.

Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101.

Cartaxo, B., Pinto, G., Vieira, E., and Soares, S. (2016). Evidence briefings: Towards a medium to transfer knowledge from systematic reviews to practitioners. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’16, New York, NY, USA. Association for Computing Machinery.

Coelho, J., Valente, M. T., Milen, L., and Silva, L. L. (2020). Is this GitHub project maintained? measuring the level of maintenance activity of opensource projects. Information and Software Technology, 1:1–35.

Fischer, F., Böttinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., and Fahl, S. (2017). Stack overflow considered harmful? the impact of copy paste on android application security. In Proceedings of the IEEE Symposium on Security and Privacy, SP ’17, pages 121–136.

Garousi, V., Felderer, M., and Mäntylä, M. V. (2016). The need for multivocal literature reviews in software engineering: Complementing systematic literature reviews with grey literature. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, EASE ’16, pages 26:1–26:6, New York, NY, USA. ACM.

Garousi, V., Felderer, M., and Mäntylä, M. V. (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106:101–121.

Garousi, V., Felderer, M., Mäntylä, M. V., and Rainer, A. (2020). Benefitting from the Grey Literature in Software Engineering Research, pages 385–413. Springer International Publishing, Cham.

Hosseinzadeh, S., Rauti, S., Laurén, S., Mäkelä, J.M., Holvitie, J., Hyrynsalmi, S., and Leppänen, V. (2018). Diversification and obfuscation techniques for software security: A systematic literature review. Information and Software Technology, 104:72–93.

Kamei, F., Wiese, I., Pinto, G., Ribeiro, M., and Soares, S. (2020). On the use of grey literature: A survey with the brazilian software engineering research community. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering, SBES 2020, New York, NY, USA. Association for Computing Machinery.

Kitchenham, B., Brereton, P., Turner, M., Niazi, M., Linkman, S., Pretorius, R., and Budgen, D. (2009). The impact of limited search procedures for systematic literature reviews a participant observer case study. In Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement, ESEM ’09, pages 336–345, Washington, DC, USA. IEEE Computer Society.

Linåker, J., Sulaman, S., Maiani de Mello, R., and Martin, H. (2015). Guidelines for conducting surveys in software engineering. Technical report, Lund University.

Maro, S., Steghöfer, J.P., and Staron, M. (2018). Software traceability in the automotive domain: Challenges and solutions. Journal of Systems and Software, 141:85 – 110.

Neto, G. T. G., Santos, W. B., Endo, P. T., and Fagundes, R. A. A. (2019). Multivocal literature reviews in software engineering: Preliminary findings from a tertiary study. In Proceedings of the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’19, pages 1–6.

Oliveira, J. A., Viggiato, M., Pinheiro, D., and Figueiredo, E. (2021). Mining experts from source code analysis: An empirical evaluation. Journal of Software Engineering Research and Development, 9(1):1:1 – 1:16.

Paez, A. (2017). Gray literature: An important resource in systematic reviews. Journal of Evidence Based Medicine, 10(3):233–240. Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide, volume 11. Blackwell Publishing Ltd.

Pinto, G., Ferreira, C., Souza, C., Steinmacher, I., and Meirelles, P. (2019). Training software engineers using open source software: The students’ perspective. In Proceedings of IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSESEET’19, pages 147–157. Institute of Electrical and Electronics Engineers (IEEE).

Rainer, A. and Williams, A. (2018a). Technical report: Do software engineering practitioners cite research on software testing in their online articles? a structured search of grey data. Technical report.

Rainer, A. and Williams, A. (2018b). Using blog articles in software engineering research: Benefits, challenges and case–survey method. In Proceedings of the 25th Australasian Software Engineering Conference), ASWEC’18, pages 201–209.

Rainer, A. and Williams, A. (2019). Using bloglike documents to investigate software practice: Benefits, challenges, and research directions. Journal of Software: Evolution and Process, 31(11):e2197.

Raulamo Jurvanen, Päivi, Mäntylä, M., and Garousi, V. (2017). Choosing the right test automation tool: A grey literature review of practitioner sources. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE ’17, pages 21–30. ACM.

RodríguezPérez, G., Robles, G., and González Barahona, J. M. (2018). Reproducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the szz algorithm. Information and Software Technology, 99:164–176. Salam, M. and Khan, S. U. (2018). Challenges in the development of green and sustainable software for software multisourcing vendors: Findings from a systematic literature review and industrial survey. Journal of Software: Evolution and Process, 30(8):19–22.

Sharma, T. and Spinellis, D. (2018). A survey on software smells. Journal of Systems and Software, 138:158 – 173. Spencer, D. (2009). Card sorting: Designing usable categories. Rosenfeld Media.

Stray, V. and Moe, N. B. (2020). Understanding coordination in global software engineering: A mixed methods study on the use of meetings and slack. Journal of Systems and Software, 170:110717.

Tom, E., Aurum, A., and Vidgen, R. (2013). An exploration of technical debt. Journal of Systems and Software, 86(6):1498–1516. Viera, A. J. and Garrett, J. M. (2005). Understanding inter observer agreement: the kappa statistic. Family Medicine, 37(5):360–363.

Williams, A. and Rainer, A. (2017). Toward the use of blog articles as a source of evidence for software engineering research. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE’17, pages 280–285, New York, NY, USA. ACM.

Williams, A. and Rainer, A. (2019). How do empirical software engineering researchers assess the credibility of practitioner generated blog posts? In Proceedings of the 23nd International Conference on Evaluation and Assessment in Software Engineering, EASE ’19, pages 211–220. ACM.

Yasin, A., Fatima, R., Wen, L., Afzal, W., Azhar, M., and Torkar, R. (2020). On using grey literature and google scholar in systematic literature reviews in software engineering. IEEE Access, 8:36226–36243.

Zahedi, M., Rajapakse, R. N., and Babar, M. A. (2020). Mining questions asked about continuous software engineering: A case study of stack overflow. In Li, J., Jaccheri, L., Dingsøyr, T., and Chitchyan, R., editors, EASE ’20: Evaluation and Assessment in Software Engineering, Trondheim, Norway, April 1517, 2020, pages 41–50. ACM.

Zhang, H., Zhou, X., Huang, X., Huang, H., and Babar, M. A. (2020). An evidence based inquiry into the use of grey literature in software engineering. In Proceedings of the 42th International Conference on Software Engineering, ICSE ’20.

Downloads

Published

2022-06-14

How to Cite

Kamei, F. K., Wiese, I., Pinto, G., Ferreira, W., Ribeiro, M., Souza, R., & Soares, S. (2022). Assessing the Credibility of Grey Literature: A Study with Brazilian Software Engineering Researchers. Journal of Software Engineering Research and Development, 10, 9:1 – 9:20. https://doi.org/10.5753/jserd.2022.1897

Issue

Section

Research Article

Most read articles by the same author(s)