Towards a Corpus Methodology for LLMs in the Legal Domain - Athaydes, Aline; Mota, Lucas B. Bulcão; Moraes Neto, Fernando Humberto de Almeida; Silva, Samuel Rios da; Mane, Babacar; Claro, Daniela Barreiro; Souza, Marlo; Lisboa, Andressa Beatriz Cardoso (Anais do Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL); 2025: Anais do XVI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, 2025)
The creation of high-quality Question-Answer (QA) datasets is critical for developing reliable legal AI systems, yet a significant gap exists between intrinsic textual metrics and real-world model performance. This paper introduces an end-to-end framework to bridge this gap. We first present a refined methodology for generating a legal QA dataset (V2) based on the Brazilian Consumer Protection Code (Código de Defesa do Consumidor CDC), demonstrating its superiority over a baseline corpus using metrics such as MTLD and Shannon Entropy. We then assess its practical impact by fine-tuning a Qwen3-8B model with LoRA. The model’s performance is evaluated on a novel, expert validated 76 question multiple choice benchmark. Results show that the fine-tuned model achieves perfect accuracy on the benchmark and surpasses the base model across text generation metrics including BLEU, METEOR and BERTScore. Our work offers a reproducible methodology for legal dataset construction and validation, providing empirical evidence that improvements in data quality yield tangible gains in downstream legal reasoning tasks.