Towards Agentic Data Engineering: A Contract-Driven System with Self-Evolving Knowledge

• Luan Prado Instituto Kunumi
• Leonardo Guerreiro Azevedo Instituto Kunumi https://orcid.org/0000-0002-2109-1285
• Adriano Veloso Instituto Kunumi / Universidade Federal de Minas Gerais (UFMG)

DOI: https://doi.org/10.5753/sbbd.2026.249234

Resumo

Data engineering suffers from long, non-repeatable cycles due to the lack of persistent domain knowledge. This work proposes a contract-driven agentic architecture that externalizes knowledge into executable specifications across sessions. A Data Explorer Agent explores databases while a Data Transformation Agent translates validated contracts into pipelines. Contracts serve as specification, validation, and documentation, forming a feedback loop of recorded, generalized, reusable results. The evaluation of the architecture across diverse databases shows knowledge accumulation and reuse, detection of data quality issues, and the completeness of data pipelines, confirming that knowledge externalization is key to reliable data engineering automation.

Palavras-chave: Agentic Data Engineering, Multi-Agent Architecture, Data Exploration, Data Pipeline Automation, Contract-Driven Systems, Self-Evolving Knowledge

Referências

Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16:3–9.

Anthropic (2026). Claude sonnet 4.6. Available at: [link]. URL date: June 15th, 2026.

Bezerra, E. (2025). Introduction to llm-based agents. In Topics in Data and Information Management: SBBD 2025 Short-Courses, Topics in Data and Information Management, pages 1 –30. SBC, Porto Alegre, RS.

Conway, M. E. (1968). How do committees invent? Datamation, 14(4):28–31. Available at: [link].

Dewey, J. (1938). Experience and Education. Kappa Delta Pi / Collier Books, New York.

Dijkstra, E. W. (1982). On the role of scientific thought. In Selected Writings on Computing: A Personal Perspective, pages 60–66. Springer-Verlag, New York. Original manuscript EWD447, 1974.

Feigenbaum, E. A., Buchanan, B. G., and Lederberg, J. (1969). On generality and problem solving: A case study using the DENDRAL program. Machine Intelligence, 6:165–190.

Gostev, P. (2026). Bullshitbench: A benchmark for detecting nonsense in ai model responses. Available at: [link]. URL date: June 15th, 2026.

Hong, S., Lin, Y., Liu, B., Liu, B., Wu, B., Zhang, C., Li, D., Chen, J., Zhang, J., Wang, J., et al. (2025). Data interpreter: An llm agent for data science. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19796–19821.

Inmon, W. H. (1992). Building the Data Warehouse. QED Technical Publishing Group / John Wiley & Sons, New York.

Johnson, A., Bulgarelli, L., Pollard, T., Gow, B., Moody, B., Horng, S., Celi, L. A., and Mark, R. (2024). MIMIC-IV. PhysioNet. Version 3.1. Available at: DOI: 10.13026/kpb9-mt58. URL date: June 15th, 2026.

Kimball, R. (1996). The Data Warehouse Toolkit: Practical Techniques for Building Dimensional Data Warehouses. John Wiley & Sons, New York, 1st edition.

Kleppmann, M. (2017). Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O’Reilly Media, Sebastopol, CA.

Kolb, D. A. (1984). Experiential Learning: Experience as the Source of Learning and Development. Prentice Hall, Englewood Cliffs, NJ.

Liu, X., Shen, S., Li, B., Ma, P., Jiang, R., Luo, Y., Zhang, Y., Fan, J., Li, G., and Tang, N. (2024). A survey of NL2SQL with large language models: Where are we, and where are we going? arXiv preprint arXiv:2408.05109.

Munappy, A. R., Bosch, J., and Holmström Olsson, H. (2020). Data pipeline management in practice: Challenges and opportunities. In Product-Focused Software Process Improvement (PROFES 2020), volume 12562 of Lecture Notes in Computer Science, pages 168–184, Cham. Springer.

Nonaka, I. and Takeuchi, H. (1995). The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation. Oxford University Press, New York.

Prado, L., Azevedo, L. G., and Veloso, A. (2026). Agentic Data Engineering. Available at: [link]. URL date: June 15th, 2026.

Rowley, J. (2007). The wisdom hierarchy: Representations of the DIKW hierarchy. Journal of Information Science, 33(2):163–180.

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., and Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems (NIPS 2015), volume 28, pages 2503–2511. Curran Associates.

Shortliffe, E. H. (1976). Computer-Based Medical Consultations: MYCIN. Artificial Intelligence Series. Elsevier, New York.

Walsh, W. (2026). NBA database. Version 3.1. Available at: [link]. URL date: June 15th, 2026.

Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., Sun, X., and Li, Z. (2025). MAC-SQL: A multi-agent collaborative framework for text-to-SQL. In Proceedings of the 31st International Conference on Computational Linguistics (COLING), pages 540–557, Abu Dhabi, UAE. Association for Computational Linguistics.

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).

Zhang, J., Hu, S., Lu, C., Lange, R., and Clune, J. (2025). Darwin Gödel machine: Open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954.

Zhang, J., Zhao, B., Yang, W., Foerster, J., Clune, J., Jiang, M., Devlin, S., and Shavrina, T. (2026). Hyperagents. arXiv preprint arXiv:2603.19461.

Zhou, X., Li, G., Sun, Z., Liu, Z., Chen, W., Wu, J., Liu, J., Feng, R., and Zeng, G. (2024). D-Bot: Database diagnosis system using large language models. Proceedings of the VLDB Endowment, 17(10):2514–2527.