A Comparative Evaluation of AI Agent Orchestration Frameworks for Regulated Environments

• Renzo Zukeram UFRGS https://orcid.org/0009-0007-7366-7489
• Vinícius Mergulhão UFRGS https://orcid.org/0009-0008-8305-1028
• Yuri de Medeiros UFRGS https://orcid.org/0009-0004-3825-0109
• Marcantoni Melo UFRGS https://orcid.org/0009-0009-3313-6573
• Ramiro Santos Júnior UFRGS https://orcid.org/0000-0003-2899-7458
• Stefano Loss UFRGS https://orcid.org/0000-0003-0093-4930
• Nélio Cacho UFRGS https://orcid.org/0000-0002-4706-5616
• Frederico Lopes UFRGS https://orcid.org/0000-0002-1108-8875

DOI: https://doi.org/10.5753/semish.2026.22198

Resumo

Organizations operating under compliance mandates increasingly rely on AI agents to automate workflows involving unstructured documents and dynamic decision-making. In such settings, agentic systems must reconcile autonomy with strict requirements for auditability, controlled variability, and integration with legacy infrastructures. We propose an Exemplar as a reusable evaluation artifact for analyzing agentic orchestration frameworks under Compliance Management and Documentary Uncertainty. The Exemplar captures the interaction between probabilistic extraction and deterministic constraint enforcement in workflows characterized by unstructured inputs, rigid schemas, and asynchronous processes. We empirically evaluate four frameworks (CrewAI, Embabel, LangChain, and n8n) across three dimensions: type safety, reasoning auditability, and legacy interoperability. The evaluation is grounded in a real-world instantiation within a state penitentiary agency, involving automated processing of employment contracts with incomplete data and asynchronous stakeholder interactions. Results show clear trade-offs between flexibility and control, highlighting the importance of hybrid approaches that combine agentic reasoning with deterministic validation. The exemplar generalizes to regulated domains such as public procurement, finance, and healthcare.

Referências

Bannett, M. (2024). Deep learning powered architectures for intelligent workflow dynamics, adaptive task scheduling, and autonomous orchestration of complex processes in n8n. MetaVision Journal of Multidisciplinary Studies (MVJMS), 1(3):1–19.

Derouiche, H., Brahmi, Z., and Mazeni, H. (2025). Agentic ai frameworks: Architectures, protocols, and design challenges. arXiv preprint arXiv:2508.10146.

Duan, Z. and Wang, J. (2024). Exploration of llm multi-agent application implementation based on langgraph+ crewai. arXiv preprint arXiv:2411.18241.

Jeong, J., Gil, D., Kim, D., and Jeong, J. (2024). Current research and future directions for off-site construction through langchain with a large language model. Buildings.

Krishnan, N. (2025). Ai agents: Evolution, architecture, and real-world applications. arXiv preprint arXiv:2503.12687.

Li, Z. and Ning, H. (2023). Autonomous gis: the next-generation ai-powered gis. International Journal of Digital Earth, 16(2):4668–4686.

Liu, X., Yu, H., Zhang, H., Xu, Y., et al. (2023). Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.

Luo, J., Zhang, W., Yuan, Y., Zhao, Y., Yang, J., Gu, Y., Wu, B., Chen, B., Qiao, Z., Long, Q., et al. (2025). Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460.

Luong, V., Dinh, S., Raghavan, S., Nguyen, W., Nguyen, Z., Le, Q., Vo, H., Maegaito, K., Nguyen, L., Nguyen, T., Ha, A. H., and Nguyen, C. (2024). Dana: Domain-aware neurosymbolic agents for consistency and accuracy.

Meimandi, K. J., Aránguiz-Dias, G., Kim, G. R., Saadeddin, L., Griffith, A., and Kochenderfer, M. J. (2025). The measurement imbalance in agentic ai evaluation undermines industry productivity claims. arXiv preprint arXiv:2506.02064.

Mequanenit, A. M., Nibret, E. A., Herrero-Martín, P., García-González, M. S., and Martinez-Bejar, R. (2025). A multi-agent deep reinforcement learning system for governmental interoperability. Applied Sciences, 15(6):3146.

Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22.

Pelluru, K. (2025). Langchain & langgraph in production: Architectures for multi-agent llm systems. Journal of Data and Digital Innovation (JDDI), 2(3):1–9.

Russell, S., Norvig, P., and Intelligence, A. (1995). A modern approach. Artificial Intelligence. Prentice-Hall, Egnlewood Cliffs, 25(27):79–80.

Salah, M., Alnoor, A., Abdelfattah, F., and Al Halbusi, H. (2025). Agentic artificial intelligence in public administration: Foundations, applications, and governance. Applications, and Governance (May 10, 2025).

Sapkota, R., Roumeliotis, K. I., and Karkee, M. (2025). Ai agents vs. agentic ai: A conceptual taxonomy, applications and challenges. Information Fusion, page 103599.

Sim, S. E., Easterbrook, S., and Holt, R. C. (2003). Using benchmarking to advance research: A challenge to software engineering. In 25th International Conference on Software Engineering, 2003. Proceedings., pages 74–83. IEEE.

Wang, L., Ma, C., Feng, X., Zhang, Z., et al. (2024). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.

Wirtz, B. W., Weyerer, J. C., and Geyer, C. (2019). Artificial intelligence and the public sector—applications and challenges. International Journal of Public Administration.

Wu, Q., Bansal, G., Zhang, J., et al. (2024). Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling.

Zhang, R., Liu, G., Liu, Y., et al. (2026). Toward edge general intelligence with agentic ai and agentification: Concepts, technologies, and future directions. IEEE Communications Surveys & Tutorials, 28:4285–4318.