Measure the Right Thing, at the Right Grain: Three Measurement Traps in Evaluating Code-Repair Agents

• João Pedro Wieland UFRJ
• Diego Castro CEFET/RJ
• Cláudia Werner UFRJ

DOI: https://doi.org/10.5753/se4as.2026.33026

Resumo

Code-repair agents get graded on whether they “fixed” a bug, but the machinery that decides that almost never gets checked. Building a tool that repairsweb-accessibility problems with local LLM agents, we hit three measurement traps, and each flipped a result we were ready to publish. Trusting the agent’s own report of success held the fix rate near 58% where an independent re-scan puts it near 50%, and hid the fact that 71% of the “bugs” were not bugs, just noise from our test harness. Counting violations that share a file as separate data points, the mistake called pseudo-replication, dressed up an ablation as a finding: the fix rate seemed to jump from 58% to 89% as we stripped the system down, and redone at the right level, the file, the effect vanished. And even a real verifier stayed foolable wherever it ran the detector’s own rules, so an audit flagged 17 of the 168 confirmed violations as false positives or pointless fixes. We tell each story with the numbers on both sides and say plainly which parts of the audit have been run. Observability and construct validity are things you design into an agent, not footnotes you add at the end, and none of this is really about accessibility: any agent judged by a signal it can push around is exposed to all three.

Palavras-chave: Agentic Systems, Evaluation, Construct Validity, Pseudo-replication, Verification, Automated Program Repair

Referências

J. Yang, C. E. Jimenez, A. Wettig, et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS, 2024.

X. Wang, B. Li, Y. Song, et al. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. ICLR, 2025.

C. E. Jimenez, J. Yang, A. Wettig, et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR, 2024.

W3C. Web Content Accessibility Guidelines (WCAG) 2.1, 2018.

Z. Qi, F. Long, S. Achour, and M. Rinard. An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems. ISSTA, 24–36, 2015.

E. K. Smith, E. T. Barr, C. Le Goues, and Y. Brun. Is the Cure Worse Than the Disease? Overfitting in Automated Program Repair. ESEC/FSE, 532–543, 2015.

Q. Zhang, C. Fang, Y. Xie, et al. A Systematic Literature Review on Large Language Models for Automated Program Repair. arXiv:2405.01466, 2024.

J. Petke, M. Martinez, M. Kechagia, A. Aleti, and F. Sarro. The Patch Overfitting Problem in Automated Program Repair. Companion Proc. FSE (IVR), 2024.

Y. Xiong, X. Liu, M. Zeng, L. Zhang, and G. Huang. Identifying Patch Correctness in Test-Based Program Repair. ICSE, 789–799, 2018.

C. S. Xia and L. Zhang. Automated Program Repair via Conversation. ISSTA, 2024.

S. H. Hurlbert. Pseudoreplication and the Design of Ecological Field Experiments. Ecological Monographs, 54(2):187–211, 1984.

P. Ralph and E. Tempero. Construct Validity in Software Engineering Research and Software Metrics. EASE, 13–23, 2018.

S. Puppala, I. Hossain, M. J. Alam, et al. Agent-Fence: Mapping Security Vulnerabilities Across Deep Research Agents. AAAI Symp., arXiv:2602.07652, 2026.

C. Zhang, Y. Park, F. Fleischer, et al. SoK: DARPA’s AI Cyber Challenge (AIxCC). arXiv:2602.07666, 2026.