Knowledge Distillation in Compact Models: An Approach Applied to Text Processing for Public Security - Barcelar, Ricardo Rodrigues; Garcia, Leonardo Arruda Vilela; Heleno, Alan Papafanurakis; Ventura, Thiago Meirelles; Oliveira, Allan Gonçalves de (Anais do Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL); 2025: Anais do XVI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, 2025)
This article aims to develop a model for summarizing police reports in the context of public security, with a focus on execution on limited hardware. The approach combined hybrid distillation (logits and intermediate representations) and supervised fine-tuning with LoRA, using a corpus of 19,286 police reports. The evaluation was conducted using automatic metrics (BERTScoreF1) and qualitative analysis by specialists. The results demonstrate that the distilled model generates clear, coherent, and semantically appropriate summaries, comparable to those of the larger model, with superior computational performance, including in hardware-constrained environments.