Beyond Task Completion: Product and Human-Agent Process Quality in Agentic Software Engineering
Resumo
Large Language Model-based agents are increasingly used to plan, modify, test, and review software artifacts across the software development life cycle. Current evaluations, however, still emphasize task completion, benchmark scores, and final artifact correctness, leaving the human-agent process that produces those artifacts largely unobserved. This vision paper argues that agentic software engineering requires process-quality instruments capable of capturing how agentic decisions are generated, reviewed, trusted, and attributed in real development workflows.We propose four constructs for human-agent process quality: decision traceability, human review workload, trust calibration, and responsibility attribution. For each construct, we outline operational definitions, candidate metrics, possible measurement instruments, and empirical anchors from recent software engineering studies.We argue that these constructs are not independent: traceability conditions responsibility, while review workload and trust calibration may create tensions between efficiency and defect detection. By shifting the evaluation focus from task success alone to observable human-agent process quality, this paper outlines a research agenda for more robust, accountable, and governable agentic software engineering.
Referências
Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, and Baishakhi Ray. 2025. Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study. DOI: 10.48550/ARXIV.2506.08311, Version Number: 2.
Haoming Huang, Pongchai Jaisri, Shota Shimizu, Lingfeng Chen, Sota Nakashima, and Gema Rodríguez-Pérez. 2026. More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requests. DOI: 10.48550/ARXIV.2601.21276, Version Number: 1.
ISO/IEC. 2011. Systems and Software Engineering – Systems and Software Quality Requirements and Evaluation (SQuaRE) – System and Software Quality Models. 34 pages. Available at: [link].
Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2026. Large Language Model-Based Agents for Software Engineering: A Survey. ACM Transactions on Software Engineering and Methodology (3 2026). DOI: 10.1145/3796507
Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2149–2160. DOI: 10.1109/ICSE48619.2023.00181
Dao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy, Pham Phu Hoa, Tran Chi Nguyen, Nguyen Dinh Ha Duong, and Truong Bao Tran. 2026. Early-Stage Prediction of Review Effort in AI-Generated Pull Requests. DOI: 10.48550/ARXIV.2601.00753, Version Number: 2.
Hussein Mozannar, Valerie Chen, Mohammed Alsobay, Subhro Das, Sebastian Zhao, Dennis Wei, Manish Nagireddy, Prasanna Sattigeri, Ameet Talwalkar, and David Sontag. 2024. The RealHumanEval: Evaluating Large Language Models’ Abilities to Support Programmers. arXiv:2404.02806 [cs.SE] [link]
Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do Users Write More Insecure Code with AI Assistants?. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (New York, NY, USA). ACM, 2785–2799. DOI: 10.1145/3576915.3623157
Sadra Sabouri, Philipp Eibl, Xinyi Zhou, Morteza Ziyadi, Nenad Medvidovic, Lars Lindemann, and Souti Chattopadhyay. 2025. Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, Ottawa, ON, Canada, 1678–1690. DOI: 10.1109/ICSE55347.2025.00199
Trevor Stalnaker, Nathan Wintersgill, Oscar Chaparro, Laura A. Heymann, Massimiliano Di Penta, Daniel M. German, and Denys Poshyvanyk. 2026. Developer Perspectives on Licensing and Copyright Issues Arising from Generative AI for Software Development. ACM Transactions on Software Engineering and Methodology 35, 2 (Feb. 2026), 1–39. DOI: 10.1145/3743133
Wannita Takerngsaksiri, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, and Ming Wu. 2025. Human-In-The-Loop Software Development Agents. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 342–352. DOI: 10.1109/ICSE-SEIP66354.2025.00036
Valerio Terragni, Annie Vella, Partha Roop, and Kelly Blincoe. 2025. The Future of AI-Driven Software Engineering. ACM Transactions on Software Engineering and Methodology 34 (5 2025). Issue 5. DOI: 10.1145/3715003
Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. 2025. Agents in software engineering: survey, landscape, and vision. Automated Software Engineering 32 (11 2025), 70. Issue 2. DOI: 10.1007/s10515-025-00544-2
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proceedings of the ACM on Software Engineering 2 (6 2025), 801–824. Issue FSE. DOI: 10.1145/3715754
