RHEA: An Agentic Evaluation and Monitoring Framework

• Pedro Augusto Luiz Eldorado Research Institute
• Wandemberg Gibaut Eldorado Research Institute
• Gabriel Ukstin Talasso UNICAMP
• Gabriel Paz HP Inc.
• Thomas da Silva Paula HP Inc.
• Leandro Villas UNICAMP

DOI: https://doi.org/10.5753/eniac.2026.31366

Resumo

We introduce RHEA (Robust Heuristics for Evaluating Agents), a framework-agnostic harness that evaluates LLM workflows under a unified interface, collecting latency, token, and tool-use metrics. On the WorkBench email-and-calendar benchmark, evaluating five LLMs, six frameworks, and two orchestration architectures, we find that model choice has a larger impact on correctness than framework choice (varying 2x-2.5x), while frameworks differ mainly in efficiency. Sequential architectures outperform hierarchical ones in argument correctness (+6.7 pp), while hierarchical architectures reduce token consumption by 34.2%, and parameter mismatch emerges as the dominant failure mode in tool use (39.0%), remaining undetected by coarse task-success metrics.

Referências

Agno (2024). Agno: The complete stack for AI agents – framework, runtime and control plane. [link].

Chen, J., Hu, X., Liu, S., Huang, S., Tu, W.-W., He, Z., and Wen, L. (2024). Llmarena: Assessing capabilities of large language models in dynamic multi-agent environments. arXiv preprint arXiv:2402.16499.

Gu, K., Shang, R., Jiang, R., Kuang, K., Lin, R.-J., Lyu, D., Mao, Y., Pan, Y., Wu, T., Yu, J., et al. (2024). Blade: Benchmarking language model agents for data-driven science. arXiv preprint arXiv:2408.09667.

He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., and Yu, P. (2024). The emerged security and privacy of llm agent: A survey with case studies. ACM Computing Surveys.

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Huang, K.-H., Prabhakar, A., Thorat, O., Agarwal, D., Choubey, P. K., Mao, Y., Savarese, S., Xiong, C., and Wu, C.-S. (2025). Crmarena-pro: Holistic assessment of llm agents across diverse business scenarios and interactions. arXiv preprint arXiv:2505.18878.

Hugging Face (2024). smolagents: A barebones library for agents that think in code. [link].

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2023). Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.

Kumar, P. (2024). Large language models (llms): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(10):260.

LangChain AI (2024). LangGraph: Build resilient language agents as graphs. [link].

Li, X., Wang, S., Zeng, S., Wu, Y., and Yang, Y. (2024). A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9.

Li, Y., Naito, A., and Shirado, H. (2025). Assessing collective reasoning in multi-agent llms via hidden profile tasks. arXiv preprint arXiv:2505.11556.

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Microsoft (2024). AutoGen: A framework for building, orchestrating and deploying AI agents and multi-agent workflows. [link].

Mistral AI Team (2025). Mistral small 3.2 (24b): Model card and repository. Hugging Face. Accessed: 2026-05-22.

OpenAI (2024). Gpt-4o mini: Advancing cost-efficient intelligence. Accessed: 2024-11-21.

OpenAI (2025a). Introducing GPT-4.1 in the API. [link].

OpenAI (2025b). OpenAI Agents SDK: A lightweight, powerful framework for multi-agent workflows. [link].

Pydantic (2024). PydanticAI: Genai agent framework, the Pydantic way. [link].

Styles, O., Miller, S., Cerda-Mardini, P., Guha, T., Sanchez, V., and Vidgen, B. (2024). Workbench: a benchmark dataset for agents in a realistic workplace setting.

Sun, H., Zhang, S., Niu, L., Ren, L., Xu, H., Fu, H., Zhao, F., Yuan, C., and Wang, X. (2025). Collab-overcooked: Benchmarking and evaluating large language models as collaborative agents. arXiv preprint arXiv:2502.20073.

Wang, W., Zhang, D., Feng, T., Wang, B., and Tang, J. (2024). Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems. arXiv preprint arXiv:2408.15971.

Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo, Z., and Fan, Z. (2024). Qwen2 technical report.

Yu, M., Meng, F., Zhou, X., Wang, S., Mao, J., Pan, L., Chen, T., Wang, K., Li, X., Zhang, Y., et al. (2025). A survey on trustworthy llm agents: Threats and countermeasures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6216–6226.

Zhang, D., Zhoubian, S., Cai, M., Li, F., Yang, L., Wang, W., Dong, T., Hu, Z., Tang, J., and Yue, Y. (2025). Datascibench: An llm agent benchmark for data science. arXiv preprint arXiv:2502.13897.

Zhang, Y., Pan, Y., Wang, Y., and Cai, J. (2024). Pybench: Evaluating llm agent on various real-world coding tasks. arXiv preprint arXiv:2407.16732.

Zhu, K., Du, H., Hong, Z., Yang, X., Guo, S., Wang, D. Z., Wang, Z., Qian, C., Tang, R., Ji, H., et al. (2025). Multiagentbench: Evaluating the collaboration and competition of llm agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8580–8622.

Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. (2024). Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934.