A Metaprotocol For a Family of Rapid Multivocal Reviews of Generative AI in the Software Industry
Abstract
Context: With the growing interest in solutions based on Generative Artificial Intelligence (GenAI) applied to SE, a need to identify solutions beyond conceptual proposals and prototypes arises. In this rapidly evolving landscape, exploring methodological strategies that enable the identification of GenAI-based solutions used in the industry becomes essential. Goals: In this paper, we report on the experience of developing a metaprotocol for a family of Rapid Multivocal Reviews (RMRs) to identify operational GenAIbased technologies being used in the software industry (i) across the phases of the software development lifecycle and (ii) in the planning and executing SE empirical studies. Method: A metaprotocol was developed through an interactive process combining GenAI’s creative support with software engineers’ feedback. GenAI assisted in defining the stages of the RMRs family, while discussion cycles with software engineers contributed to refining and validating it. Results: Six RMR instances have been organized by covering the perspectives of Software Requirements, Design, Test, Coding, Management, and Empirical Methods described in SWEBOK 4.0. It allowed guiding the execution of the RMRs and highlighted the challenges of organizing the metaprotocol and justifying the decision-making. Conclusion: Using a metaprotocol to support the observation of industrial experience using GenAI in software projects through RMRs will contribute to organizing a body of knowledge regarding GenAI-based solutions currently available and used in the software industry to support the SWEBOK 4.0 practices.
Keywords:
Generative Artificial Intelligence (GenAI), Software Engineering, Rapid Multivocal Review, Solutions
References
Jens Peter Andersen, Lise Degn, Rachel Fishberg, Ebbe K Graversen, Serge PJM Horbach, Evanthia Kalpazidou Schmidt, JesperWSchneider, and Mads P Sørensen. 2025. Generative Artificial Intelligence (GenAI) in the research process–A survey of researchers’ practices and perceptions. Technology in Society 81 (2025), 102813.
Callen Anthony, Beth A Bechky, and Anne-Laure Fayard. 2023. “Collaborating” with AI: Taking a system view to explore the future of work. Organization Science 34, 5 (2023), 1672–1694.
Bruno Cartaxo, Gustavo Pinto, and Sergio Soares. 2018. The role of rapid reviews in supporting decision-making in software engineering practice. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018. 24–34.
Christof Ebert and Panos Louridas. 2023. Generative AI for software practitioners. IEEE Software 40, 4 (2023), 30–38.
Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology 106 (2019), 101–121.
Aleksi Huotala, Miikka Kuutila, Paul Ralph, and Mika Mäntylä. 2024. The promise and challenges of using LLMs to accelerate the screening process of systematic reviews. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 262–271.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213.
Xuân-Lan Lam Hoai and Thierry Simonart. 2023. Comparing meta-analyses with ChatGPT in the evaluation of the effectiveness and tolerance of systemic therapies in moderate-to-severe plaque psoriasis. Journal of Clinical Medicine 12, 16 (2023), 5410.
William Leara. 2024. Guide to the SWEBOK v4.0 Has Been Released. [link] [accessed on Accessed: May 16, 2025].
MDB Modi. [n. d.]. Transforming Software Development Through Generative AI: A Systematic Analysis of Automated Development Practices. International Journal of Scientific Research in Computer Science, Engineering and Information Technology 10, 6 ( [n. d.]).
Kai Petersen. 2024. Case study identification with GPT-4 and implications for mapping studies. Information and Software Technology 171 (2024), 107452. DOI: 10.1016/j.infsof.2024.107452
Kai Petersen and Jan M. Gerken. 2025. On the road to interactive LLM-based systematic mapping studies. Information and Software Technology 178 (2025), 107611. DOI: 10.1016/j.infsof.2024.107611
Mark Petticrew and Helen Roberts. 2008. Systematic reviews in the social sciences: A practical guide. John Wiley & Sons.
Patricia de Oliveira Santos, Allan Chamon Figueiredo, Pedro Nuno Moura, Bruna Diirr, Adriana CF Alvim, and Rodrigo Pereira Dos Santos. 2024. Impacts of the Usage of Generative Artificial Intelligence on Software Development Process. In Proceedings of the 20th Brazilian Symposium on Information Systems. 1–9.
Yuya Sasaki, Hironori Washizaki, Jialong Li, Nobukazu Yoshioka, Naoyasu Ubayashi, and Yoshiaki Fukazawa. 2025. Landscape and Taxonomy of Prompt Engineering Patterns in Software Engineering. IT Professional 27 (2025), 41–49. DOI: 10.1109/MITP.2024.3525458
Orit Shaer, Angelora Cooper, Osnat Mokryn, Andrew L Kun, and Hagit Ben Shoshan. 2024. AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Article 1050, 17 pages. DOI: 10.1145/3613904.3642414
Bianca Trinkenreich, Fabio Calefato, Geir Hanssen, Kelly Blincoe, Marcos Kalinowski, Mauro Pezzè, Paolo Tell, and Margaret-Anne Storey. 2025. Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research [Manuscript under review]. (2025).
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang,Weisong Sun, Shengcheng Yu, and Zhenyu Chen. 2023. A survey on large language models for software engineering. arXiv preprint arXiv:2312.15223 (2023).
Callen Anthony, Beth A Bechky, and Anne-Laure Fayard. 2023. “Collaborating” with AI: Taking a system view to explore the future of work. Organization Science 34, 5 (2023), 1672–1694.
Bruno Cartaxo, Gustavo Pinto, and Sergio Soares. 2018. The role of rapid reviews in supporting decision-making in software engineering practice. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018. 24–34.
Christof Ebert and Panos Louridas. 2023. Generative AI for software practitioners. IEEE Software 40, 4 (2023), 30–38.
Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology 106 (2019), 101–121.
Aleksi Huotala, Miikka Kuutila, Paul Ralph, and Mika Mäntylä. 2024. The promise and challenges of using LLMs to accelerate the screening process of systematic reviews. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 262–271.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213.
Xuân-Lan Lam Hoai and Thierry Simonart. 2023. Comparing meta-analyses with ChatGPT in the evaluation of the effectiveness and tolerance of systemic therapies in moderate-to-severe plaque psoriasis. Journal of Clinical Medicine 12, 16 (2023), 5410.
William Leara. 2024. Guide to the SWEBOK v4.0 Has Been Released. [link] [accessed on Accessed: May 16, 2025].
MDB Modi. [n. d.]. Transforming Software Development Through Generative AI: A Systematic Analysis of Automated Development Practices. International Journal of Scientific Research in Computer Science, Engineering and Information Technology 10, 6 ( [n. d.]).
Kai Petersen. 2024. Case study identification with GPT-4 and implications for mapping studies. Information and Software Technology 171 (2024), 107452. DOI: 10.1016/j.infsof.2024.107452
Kai Petersen and Jan M. Gerken. 2025. On the road to interactive LLM-based systematic mapping studies. Information and Software Technology 178 (2025), 107611. DOI: 10.1016/j.infsof.2024.107611
Mark Petticrew and Helen Roberts. 2008. Systematic reviews in the social sciences: A practical guide. John Wiley & Sons.
Patricia de Oliveira Santos, Allan Chamon Figueiredo, Pedro Nuno Moura, Bruna Diirr, Adriana CF Alvim, and Rodrigo Pereira Dos Santos. 2024. Impacts of the Usage of Generative Artificial Intelligence on Software Development Process. In Proceedings of the 20th Brazilian Symposium on Information Systems. 1–9.
Yuya Sasaki, Hironori Washizaki, Jialong Li, Nobukazu Yoshioka, Naoyasu Ubayashi, and Yoshiaki Fukazawa. 2025. Landscape and Taxonomy of Prompt Engineering Patterns in Software Engineering. IT Professional 27 (2025), 41–49. DOI: 10.1109/MITP.2024.3525458
Orit Shaer, Angelora Cooper, Osnat Mokryn, Andrew L Kun, and Hagit Ben Shoshan. 2024. AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Article 1050, 17 pages. DOI: 10.1145/3613904.3642414
Bianca Trinkenreich, Fabio Calefato, Geir Hanssen, Kelly Blincoe, Marcos Kalinowski, Mauro Pezzè, Paolo Tell, and Margaret-Anne Storey. 2025. Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research [Manuscript under review]. (2025).
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang,Weisong Sun, Shengcheng Yu, and Zhenyu Chen. 2023. A survey on large language models for software engineering. arXiv preprint arXiv:2312.15223 (2023).
Published
2025-09-22
How to Cite
ROCHA, Sabrina; FEITOSA, Rodrigo; GALENO, Larissa; TRAVASSOS, Guilherme H..
A Metaprotocol For a Family of Rapid Multivocal Reviews of Generative AI in the Software Industry. In: BRAZILIAN SYMPOSIUM ON SOFTWARE ENGINEERING (SBES), 39. , 2025, Recife/PE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 804-810.
ISSN 2833-0633.
DOI: https://doi.org/10.5753/sbes.2025.11577.
