Surveying the Audience Effect in Open Peer Review of a Software Engineering Workshop

• Katia Felizardo UTFPR
• Christina Flach UFBA
• Roberto Pereira UFPR
• Edson OliveiraJr UEM

Resumo

Context: Even though Open Peer Review (OPR) is one of the pillars of Open Science, informed discussions are still necessary to understand its impact on the social behavior of reviewers and the quality of reviews. Goal: The main goal of this study is to understand the audience effect on reviewers’ behavior and reviews’ quality from the perspectives of reviewers and authors of a Software Engineering (SE) workshop. Method: We performed a survey with the workshop participants, including authors and experienced reviewers in the SE field. Results: Our results show that while the OPR model did not change the reviewers’ behavior, it reinforced reviewers to avoid hostile comments and help the authors improve their manuscript. Results also show that reviewers were comfortable with revealing their identities and reviews. Finally, both reviewers and authors agreed that although the OPR model did not increase the quality of reviews, it encouraged reviewers to think more carefully about scientific issues, contributing to the writing of evidence-supported feedback. Conclusions: Open Peer Review is innovative in interacting with reviewers and authors in Brazilian SE academic venues. OPR is also inspiring, given the potential for reinforcing positive reviewers’ behaviors. However, the thematic still requires further investigation and attention from the community.

Palavras-chave: Open Review, Review Quality, Audience Effect, Peer Review, Social Behavior