Análise de Hiperparâmetros em Sistemas de Geração Aumentada por Recuperação Aplicado ao Acervo do Ministério Público do Estado do Espírito Santo

• Heitor Coitinho Quartezani UFES/MPES
• Pedro Pelacani Berger MPES
• Diego Roberto Colombo Dias UFES

DOI: https://doi.org/10.5753/webmedia_estendido.2026.33406

Resumo

This paper proposes a methodology based on Design of Experiments to analyze and optimize the hyperparameters of a Retrieval-Augmented Generation system applied to the normative repository of the Public Prosecution Office of the State of Espírito Santo. The main objective is to identify, among critical factors such as the document segmentation strategy (chunking), the volume of retrieved context (𝑘), and the choice of the Large Language Model, those that present a statistically significant impact on the quality of the generated responses. The evaluation employs the LLM-as-a-Judge paradigm, supported by the Retrieval Augmented Generation Assessment framework. Due to the non-parametric nature of the data, the statistical analysis is conducted via the Aligned Rank Transform. The results reveal that the hyperparameters have a significant impact on response quality.

Palavras-chave: Retrieval-Augmented Generation, Design of Experiments, Hyperparameter Optimization, Large Language Models, Aligned Rank Transform, Ragas, LLM-as-a-Judge

Referências

Harrison Chase. 2023. LangChain: Building applications with LLMs through composability. [link] Acessado em: 23 jan. 2026.

Isabella V. de Aquino, Matheus M. dos Santos, Carina F. Dorneles, and Jônata T. Carvalho. 2024. Extracting information from Brazilian legal documents with retrieval augmented generation. In Companion Proceedings of the 39th Brazilian Symposium on Data Bases. Florianópolis, SC. Artigo aceito para publicação.

Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ragas: Automated evaluation of retrieval augmented generation. arXiv:2309.15217 [cs.CL]

Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. 2024. A comprehensive survey of retrieval-augmented generation (RAG): Evolution, current landscape and future directions. arXiv preprint arXiv:2404.10981 (2024).

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, et al. 2023. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1–38.

Matthew Kay, Lisa Elkin, and Jacob O. Wobbrock. 2021. ARTool: Aligned Rank Transform for Factorial Analysis. [link] R package version 0.11.1.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, et al. 2020. Retrievalaugmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, Vol. 33. 9459–9474.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Percy Liang, and Christopher D. Manning. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics (2024).

Ministério Público do Estado do Espírito Santo. 2024. Consulta de legislação compilada. [link]. Acesso em: 23 jul. 2025.

Douglas C. Montgomery. 2017. Design and Analysis of Experiments (9th ed.). John Wiley & Sons.

MPES. 2026. Consulta de Legislação Compilada. [link]. Acessado em: 23 jan. 2026.

OpenAI. 2024. New embedding models and API updates. [link]. Acessado em: 23 jan. 2026.

Kenneth Reitz. 2023. Requests: HTTP for Humans. [link] Acessado em: 23 jan. 2026.

Chroma Team. 2023. Chroma: The AI-native open-source embedding database. [link] Acessado em: 23 jan. 2026.

Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, et al. 2024. Searching for best practices in retrieval-augmented generation. arXiv:2407.01219 [cs.CL]

Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 143–146.

Yue Yu, Honglei Zhuang, Jie Zhang, Yu Meng, Alexander Ratner, Jingbo Shang, Jiawei Han, and Chao He. 2024. RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs. arXiv:2407.02485 [cs.CL]