An Optimizable Retrieval-Augmented Generation Workflow for Entity Extraction in Portuguese Fraud-Investigation Documents

• Amabyle Rabeche Heck UFSC
• Jônata Tyska Carvalho UFSC

DOI: https://doi.org/10.5753/eniac.2026.31469

Resumo

Public institutions handle large volumes of legal-administrative PDF documents whose information must be located and structured for analysis, reporting, and decision support. In fraud-investigation and public-procurement workflows, manually extracting fields such as bidding process number, bidding modality, and municipality is time-consuming and error-prone, especially when evidence is dispersed across long and heterogeneous documents. These fields can act as linking points between unstructured investigative documents and structured procurement records, supporting audit traceability and fraudrisk analysis. This paper presents an optimizable Retrieval-Augmented Generation (RAG) workflow for structured entity extraction from Portuguese legal PDFs. The workflow combines PDF text extraction, document segmentation, semantic retrieval, structured JSON generation, post-processing, and evaluation based on approximate string matching. It automatically searches over combinations of RAG parameters, including chunk size, top-k, temperature, language model, and embedding configuration, through grid search on a validation set, and compares the selected configuration against zero-shot and one-shot baselines on a held-out test set using precision, recall, F1, and approximate string matching. The results show that the optimizable RAG workflow outperforms the non-retrieval baselines, achieving an F1 score of 84.71% on the test set, with F1 gains of 21.72 p.p. over zero-shot and 12.63 p.p. over one-shot.

Palavras-chave: retrieval-augmented generation, large language models, entity/information extraction, Portuguese NLP, PDF processing, public procurement, fraud investigation

Referências

Bachmann, M. (2024). Levenshtein: Python extension for computing string edit distances and similarities. Software, version 0.26.1. Accessed: 2026-08-16.

Bhat, S. R., Rudat, M., Spiekermann, J., and Flores-Herr, N. (2025). Rethinking chunk size for long-document retrieval: A multi-dataset analysis.

Bosio, E., Djankov, S., Glaeser, E., and Shleifer, A. (2022). Public procurement in law and practice. American Economic Review, 112(4):1091–1117.

Cesista, F. L., Aguiar, R., Kim, J., and Acilo, P. (2024). Retrieval augmented structured generation: Business document information extraction as tool use. In 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR).

Chinchor, N. and Marsh, E. (1998). Appendix D: MUC-7 information extraction task definition (version 5.1). In Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29 - May 1, 1998.

de Aquino, I. V., dos Santos, M. M., Dorneles, C. F., and Carvalho, J. T. (2024). Extracting information from Brazilian legal documents with retrieval augmented generation. In Anais Estendidos do XXXIX Simpósio Brasileiro de Banco de Dados (SBBD Estendido 2024).

Dice, L. R. (1945). Measures of the amount of ecologic association between species. Ecology, 26(3):297–302.

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Hladěna, J., Šteflovič, K., Čech, P., Štekerová, K., and Žváčková, A. (2025). The effect of chunk size on the rag performance. In Silhavy, R. and Silhavy, P., editors, Software Engineering: Emerging Trends and Practices in System Development, pages 317–326, Cham. Springer Nature Switzerland.

Jaccard, P. (1901). Étude comparative de la distribution florale dans une portion des alpes et du jura. Bulletin de la Société Vaudoise des Sciences Naturelles, 37:547–579.

Jadon, A., Patil, A., and Kumar, S. (2025). Enhancing domain-specific retrieval-augmented generation: Synthetic data generation and evaluation using reasoning models. In 2025 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT).

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W.-t., Rocktaschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks.

Lin, D. (2024). Revolutionizing retrieval-augmented generation with enhanced PDF structure recognition. arXiv preprint arXiv:2401.12599.

OECD (2026). Public procurement. [link]. Accessed: 2026-06-20.

Open Contracting Partnership (2026). The open contracting data standard. [link]. Accessed: 2026-06-20.

Pereira, J., Assumpcao, A., Trecenti, J., Airosa, L., Lente, C., Cleto, J., Dobins, G., Nogueira, R., Mitchell, L., and Lotufo, R. (2024). INACIA: Integrating large language models in brazilian audit courts: Opportunities and challenges. arXiv preprint arXiv:2401.05273.

Perot, V., Kang, K., Luisier, F., Su, G., Sun, X., Boppana, R. S., Wang, Z., Wang, Z., Mu, J., Zhang, H., Lee, C.-Y., and Hua, N. (2023). LMDX: Language model-based document information extraction and localization. arXiv preprint arXiv:2309.10952.

Potin, L., Figueiredo, R., Labatut, V., and Largeron, C. (2023). Pattern mining for anomaly detection in graphs: Application to fraud in public procurement. arXiv preprint arXiv:2306.10857.