Improving Legislative Accessibility with Retrieval-Augmented Generation

• Saint-Clair da Cunha Lima UFRN
• Daniel Araújo UFRN

Resumo

This study presents the design, development, and evaluation of a question-answering (Q&A) system based on the Retrieval-Augmented Generation (RAG) framework, aimed at improving public access to the legislative documents of Assembleia Legislativa do Estado do Rio Grande do Norte (Alern). The system integrates a vector-based document retrieval mechanism with large language models (LLMs) to enable users to submit natural language queries and receive relevant, accessible responses. It leverages embedding-based search to retrieve semantically relevant content and utilizes LLMs to generate clear answers. A comparative analysis of various embedding models – such as BGE-M3, ada-002, and GTE – was conducted to identify the most effective configuration for content retrieval. Results showed that BGE-M3 achieved the highest retrieval accuracy (96.86%) among the models tested. LLMs, including Llama 3.1 and DeepSeek-R1, were evaluated for response generation using BERTScore and human feedback. Llama 3.1 outperformed DeepSeek-R1 in coherence, grammar/spelling, and adequacy, achieving an average human evaluation score of 85.6%. Evaluation results, based on both synthetic datasets and human-curated questions, demonstrate the system’s potential to enhance legislative transparency and user engagement by facilitating accessible interaction with complex legal content.