Beyond More Context: Evaluating Static and Agent-Based Repository Context in LLM-Based Code Review
Resumo
Large Language Models (LLMs) have been explored for automated code reviewtasks, but it remains unclear whether their performance benefits from simply adding more context or from how such context is accessed. This paper investigates repository context in LLM-based automated code review by comparing three conditions: the original static ContextCRBench condition, static concatenation of additional repository context, and agent-based access to repository context through tools. We evaluate these conditions on the merge-decision task using a balanced subset of 144 pull requests and analyze predictive performance, response validity, latency, and tool-call traceability. Results show that richer benchmark-provided context achieved higher F1-score than minimal context. However, neither static concatenation nor agent-based repository access outperformed the corresponding benchmark condition, while repository-aware strategies increased latency. Although agent-based access did not improve predictive performance, it enabled inspectable repository access through tool-call logs. These findings suggest that effective context use in LLM-based code review depends less on simply providing additional information and more on its relevance, access strategy, and operational trade-offs.
Palavras-chave:
Large Language Models for Software Engineering, Automated Code Review, Pull Requests, Repository Context, AI Agents, Software Analytics, AI4SE
Referências
Y. Almeida, D. Albuquerque, E. Dantas Filho, F. Muniz, K. de Farias Santos, M. Perkusich, H. Almeida, and A. Perkusich. 2024. AICodeReview: Advancing Code Quality with AI-Enhanced Reviews. SoftwareX 26 (2024), 101677.
A. Bacchelli and C. Bird. 2013. Expectations, Outcomes, and Challenges ofModern Code Review. In ICSE. 712–721.
K. Baciejowski et al. 2023. Are Code Review Smells and Metrics Useful in Pull Request-Level Software Defect Prediction? In Developments in Information and Knowledge Management Systems for Business Applications. 27–52.
D. Badampudi, M. Unterkalmsteiner, and R. Britto. 2023. Modern Code Reviews: Survey of Literature and Practice. ACM Transactions on Software Engineering and Methodology 32, 4 (2023), 107:1–107:61.
M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens. 2014. Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?. In MSR. 202–211.
E. Dogan and E. Tüzün. 2022. Towards a Taxonomy of Code Review Smells. Information and Software Technology 142 (2022), 106737.
R. Hu, X. Wang, X.-C. Wen, Z. Zhang, B. Jiang, P. Gao, C. Peng, and C. Gao. 2025. Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice. arXiv preprint arXiv:2511.07017 (2025).
F. Khoshnoud et al. 2022. Which Bugs Are Missed in Code Reviews: An Empirical Study on SmartSHARK Dataset. CoRR abs/2205.09428 (2022).
O. Kononenko, O. Baysal, and M. W. Godfrey. 2016. Code Review Quality: How Developers See It. In ICSE. 1028–1038.
P. Lewis et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS, Vol. 33. 9459–9474.
Z. Li et al. 2022. Automating Code Review Activities by Large-Scale Pre-training. In ESEC/FSE.
L. Pascarella, D. Spadini, F. Palomba, M. Bruntink, and A. Bacchelli. 2018. Information Needs in Contemporary Code Review. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 135:1–135:27.
T. Schick et al. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In NeurIPS, Vol. 36.
T. Sun et al. 2025. BitsAI-CR: Automated Code Review via LLM in Practice. In Proceedings of the ACM International Conference on the Foundations of Software Engineering. 274–285.
C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering. Springer.
J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In NeurIPS, Vol. 37. 50528–50652.
S. Yao et al. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR.
F. Zhang et al. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In EMNLP. 2471–2484.
A. Bacchelli and C. Bird. 2013. Expectations, Outcomes, and Challenges ofModern Code Review. In ICSE. 712–721.
K. Baciejowski et al. 2023. Are Code Review Smells and Metrics Useful in Pull Request-Level Software Defect Prediction? In Developments in Information and Knowledge Management Systems for Business Applications. 27–52.
D. Badampudi, M. Unterkalmsteiner, and R. Britto. 2023. Modern Code Reviews: Survey of Literature and Practice. ACM Transactions on Software Engineering and Methodology 32, 4 (2023), 107:1–107:61.
M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens. 2014. Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?. In MSR. 202–211.
E. Dogan and E. Tüzün. 2022. Towards a Taxonomy of Code Review Smells. Information and Software Technology 142 (2022), 106737.
R. Hu, X. Wang, X.-C. Wen, Z. Zhang, B. Jiang, P. Gao, C. Peng, and C. Gao. 2025. Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice. arXiv preprint arXiv:2511.07017 (2025).
F. Khoshnoud et al. 2022. Which Bugs Are Missed in Code Reviews: An Empirical Study on SmartSHARK Dataset. CoRR abs/2205.09428 (2022).
O. Kononenko, O. Baysal, and M. W. Godfrey. 2016. Code Review Quality: How Developers See It. In ICSE. 1028–1038.
P. Lewis et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS, Vol. 33. 9459–9474.
Z. Li et al. 2022. Automating Code Review Activities by Large-Scale Pre-training. In ESEC/FSE.
L. Pascarella, D. Spadini, F. Palomba, M. Bruntink, and A. Bacchelli. 2018. Information Needs in Contemporary Code Review. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 135:1–135:27.
T. Schick et al. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In NeurIPS, Vol. 36.
T. Sun et al. 2025. BitsAI-CR: Automated Code Review via LLM in Practice. In Proceedings of the ACM International Conference on the Foundations of Software Engineering. 274–285.
C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering. Springer.
J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In NeurIPS, Vol. 37. 50528–50652.
S. Yao et al. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR.
F. Zhang et al. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In EMNLP. 2471–2484.
Publicado
11/09/2026
Como Citar
LIMA, Jonas Bernardino de; ALBUQUERQUE, Danyllo; DANTAS FILHO, Emanuel; RAMOS, Felipe; MEDEIROS, Juliana.
Beyond More Context: Evaluating Static and Agent-Based Repository Context in LLM-Based Code Review. In: WORKSHOP BRASILEIRO DE ENGENHARIA DE SOFTWARE INTELIGENTE (ISE), 5. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 19-24.
DOI: https://doi.org/10.5753/ise.2026.32612.
