Fairness Risk Triage in Pull Requests Using an LLM-Based Reviewer: An Experiment on Prompting Strategies

• Gabriel Caitano UFMS
• Arthur Cacciatore UFMS
• Awdren Fontão UFMS

DOI: https://doi.org/10.5753/se4as.2026.32989

Resumo

While LLM-based code reviewers can identify traditional defects, their ability to triage fairness risks in Pull Requests—serving as specialized skills within broader agentic systems—remains underexplored. This paper evaluates how prompting strategies affect an LLM reviewer’s detection and characterization of algorithmic biases. Using Llama 3.3 70B, we conducted a controlled experiment across ten synthetic JavaScript scenarios, each pairing a biased implementation with a clean reference. We compared simple and advanced prompts across 200 executions (five repetitions per combination), evaluated by three LLM judges against ground truth. The simple prompt achieved higher mean recall (0.96 vs. 0.91) and precision (0.84 vs. 0.74). Bootstrap confidence intervals show this difference is not uniformly robust: only one scenario-level effect has non-overlapping intervals, and even the global difference overlaps given the limited number of repetitions. However, the advanced prompt better characterized the specific sensitive attribute and code location of flagged risks. While classifications remained mostly consistent across repetitions, judge-dependent divergences occurred in specific scenarios. Findings highlight a practical trade-off between detection sensitivity and characterization detail.

Palavras-chave: Fairness, Large Language Models, Pull Request Review, Software Engineering

Referências

Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and Machine Learning: Limitations and Opportunities. MIT Press, Cambridge, MA.

Yuriy Brun and Alexandra Meliou. 2018. Software fairness. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA, 754–759. DOI: 10.1145/3236024.3264838

Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. 2025. Automated Code Review in Practice. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 425–436. DOI: 10.1109/ICSE-SEIP66354.2025.00043

Yongkang Du, Jen tse Huang, Jieyu Zhao, and Lu Lin. 2025. FairCoder: Evaluating Social Bias of LLMs in Code Generation. arXiv:2501.05396 [cs.CL] [link]

Google. 2022. Fairness: Types of Bias – Machine Learning Crash Course. [link]. Accessed: 2026-07-06.

Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. 2025. Bias Testing andMitigation in LLM-based CodeGeneration. ACMTrans. Softw. Eng. Methodol. 35, 1, Article 5 (dec 2025), 31 pages. DOI: 10.1145/3724117

Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2025. From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future. arXiv:2408.02479 [cs.SE] [link]

Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). 647–658. DOI: 10.1109/ISSRE59848.2023. 00026

Qinghua Lu, Liming Zhu, Xiwei Xu, Jon Whittle, and Zhenchang Xing. 2022. Towards a roadmap on software engineering for responsible AI. In Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI (Pittsburgh, Pennsylvania) (CAIN ’22). Association for Computing Machinery, New York, NY, USA, 101–112. DOI: 10.1145/3522664.3528607

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A Survey on Bias and Fairness in Machine Learning. Comput. Surveys 54, 6, Article 115 (2021). DOI: 10.1145/3457607

National Institute of Standards and Technology. 2024. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. Technical Report NIST AI 600-1. U.S. Department of Commerce. DOI: 10.6028/NIST.AI.600-1

Zeeshan Rasheed, Malik Abdul Sami, Muhammad Waseem, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen, Kari Systä, and Pekka Abrahamsson. 2025. AI-powered Code Review with LLMs: Early Results. arXiv:2404.18496 [cs.SE] [link]

Martin P Robillard, Deeksha M Arya, Neil A Ernst, Jin LC Guo, Maxime Lamothe, Mathieu Nassif, Nicole Novielli, Alexander Serebrenik, Igor Steinmacher, and Klaas-Jan Stol. 2024. Communicating study design trade-offs in software engineering. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1–10.