System Prompts as Specifications: Assessing the Verifiability of Claude Code Sub-Agent Behavioral Claims
Resumo
The integration of AI-powered coding assistants into software development has become a main topic in Software Engineering. Recent tools such as Claude Code increasingly support developers by automating software engineering tasks. Claude Code introduces a sub-agent mechanism that allows developers to define specialized assistants through natural-language specifications. Each sub-agent encapsulates a particular role and is described by a prompt specifying its expected behavior, execution constraints, and available tools. Despite the growing adoption of such tools, little is known about the quality of the behavioral specifications that define these sub-agents. In particular, it remains unclear whether their prescribed behaviors can be objectively verified and whether the implemented agents actually conform to these specifications during execution. To address this gap, we analyze three Claude Code sub-agent specifications from a Requirements Engineering perspective. We characterize their structure, extract behavioral claims, operationalize the notion of verifiability, and evaluate whether the observed executions conform to the specified behaviors. An analysis of three representative sub-agents identified a common specification structure composed of nine recurring sections and extracted 60 behavioral claims, of which only 11 (18.3%) were objectively verifiable. Execution-based evaluation further showed that 87% of the applicable verifiable claims were satisfied, indicating high conformance for behaviors that can be objectively assessed while highlighting substantial limitations in the verifiability of current specifications.
Palavras-chave:
AI Agents, Behavioral Specifications, Requirements Engineering, Verifiability, Claude Code
Referências
Anthropic. 2025. How We Built Our Multi-Agent Research System. [link]. Anthropic Engineering. Acesso em: 6 jul. 2026.
Victor R Basili1 Gianluigi Caldiera and H Dieter Rombach. 1994. The goal question metric approach. Encyclopedia of software engineering (1994), 528–532.
Worawalan Chatlatanagulchai et al. 2025. On the Use of Agentic Coding Manifests: An Empirical Study of Claude Code. In Product-Focused Software Process Improvement: 26th International Conference, PROFES 2025, Salerno, Italy, December 1–3, 2025, Proceedings (Salerno, Italy). Springer-Verlag, Berlin, Heidelberg, 543–551. DOI: 10.1007/978-3-032-12089-2_40
Matthias Galster et al. 2026. Harness Engineering for Agentic AI Coding Tools: An Exploratory Study. arXiv:2602.14690 [cs.SE] [link]
Sirui Hong et al. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations. [link]
Xinyi Hou et al. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8, Article 220 (Dec. 2024), 79 pages. DOI: 10.1145/3695988
Junwei Liu et al. 2026. Large Language Model-Based Agents for Software Engineering: A Survey. ACM Trans. Softw. Eng. Methodol. (March 2026). DOI: 10.1145/3796507, Just Accepted.
Qianou Ma et al. 2025. What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use. ACM Trans. Comput.-Hum. Interact. 32, 4, Article 41 (Aug. 2025), 27 pages. DOI: 10.1145/3731756
Seyedmoein Mohsenimofidi et al. 2026. Context Engineering for AI Agents in Open-Source Software. arXiv:2510.21413 [cs.SE] [link]
Yunjia Qi et al. 2025. AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios. arXiv:2505.16944 [cs.AI] [link]
Karl Wiegers and Joy Beatty. 2013. Software requirements. Pearson Education.
Zhihan Zhang et al. 2025. IHEval: Evaluating Language Models on Following the Instruction Hierarchy. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 8374–8398. DOI: 10.18653/v1/2025.naacl-long.425
Jeffrey Zhou et al. 2023. Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911 [cs.CL]
Victor R Basili1 Gianluigi Caldiera and H Dieter Rombach. 1994. The goal question metric approach. Encyclopedia of software engineering (1994), 528–532.
Worawalan Chatlatanagulchai et al. 2025. On the Use of Agentic Coding Manifests: An Empirical Study of Claude Code. In Product-Focused Software Process Improvement: 26th International Conference, PROFES 2025, Salerno, Italy, December 1–3, 2025, Proceedings (Salerno, Italy). Springer-Verlag, Berlin, Heidelberg, 543–551. DOI: 10.1007/978-3-032-12089-2_40
Matthias Galster et al. 2026. Harness Engineering for Agentic AI Coding Tools: An Exploratory Study. arXiv:2602.14690 [cs.SE] [link]
Sirui Hong et al. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations. [link]
Xinyi Hou et al. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8, Article 220 (Dec. 2024), 79 pages. DOI: 10.1145/3695988
Junwei Liu et al. 2026. Large Language Model-Based Agents for Software Engineering: A Survey. ACM Trans. Softw. Eng. Methodol. (March 2026). DOI: 10.1145/3796507, Just Accepted.
Qianou Ma et al. 2025. What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use. ACM Trans. Comput.-Hum. Interact. 32, 4, Article 41 (Aug. 2025), 27 pages. DOI: 10.1145/3731756
Seyedmoein Mohsenimofidi et al. 2026. Context Engineering for AI Agents in Open-Source Software. arXiv:2510.21413 [cs.SE] [link]
Yunjia Qi et al. 2025. AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios. arXiv:2505.16944 [cs.AI] [link]
Karl Wiegers and Joy Beatty. 2013. Software requirements. Pearson Education.
Zhihan Zhang et al. 2025. IHEval: Evaluating Language Models on Following the Instruction Hierarchy. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 8374–8398. DOI: 10.18653/v1/2025.naacl-long.425
Jeffrey Zhou et al. 2023. Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911 [cs.CL]
Publicado
08/09/2026
Como Citar
SOUZA, Débora; LIMA, Pedro.
System Prompts as Specifications: Assessing the Verifiability of Claude Code Sub-Agent Behavioral Claims. In: WORKSHOP EM ENGENHARIA DE SOFTWARE PARA SISTEMAS AGÊNTICOS (SE4AS), 1. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 13-18.
DOI: https://doi.org/10.5753/se4as.2026.32976.
