LLM4Gov: A Privacy-Preserving Approach to Teacher-Student Fine-Tuning of Distilled LLMs for the Public Sector

• Ricardo M. Marcacini USP
• Jorge Carlos Valverde-Rebaza USP
• Marcelo A. S. Turine UFMS
• Brucce Neves Santos USP
• Silvio Levcovitz PGFN
• Solange O. Rezende USP

DOI: https://doi.org/10.5753/lasdigov.2025.9471

Abstract

Large Language Models (LLMs) have revolutionized natural language processing, but their reliance on extensive computational resources and proprietary APIs poses significant challenges for public sector applications. Government agencies often face legal constraints, such as GDPR and LGPD, preventing the use of external LLMs when handling sensitive data. Even when compliance is met, the financial burden of deploying large-scale models remains a major barrier. To address these challenges, we present LLM4Gov, a privacy-preserving computational tool designed to fine-tune distilled LLMs for government-related tasks while minimizing infrastructure costs. LLM4Gov follows a structured teacher-student learning pipeline, where a lightweight anonymization module first removes personally identifiable information (PII) before any interaction with an external LLM. A teacher LLM then generates task-specific instructions from the anonymized dataset, and a distilled student LLM is fine-tuned using Low-Rank Adaptation (LoRA) and quantization, enabling deployment on resource-constrained environments. Our experimental results show that LLM4Gov consistently outperforms competitive distilled LLMs, achieving higher accuracy while preserving privacy and interpretability.

References

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024). A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45.

Gu, Y., Dong, L., Wei, F., and Huang, M. (2023). Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2022). Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.

Jin, R., Du, J., Huang, W., Liu, W., Luan, J., Wang, B., and Xiong, D. (2024). A comprehensive evaluation of quantization strategies for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 12186–12215.

Tian, Y., Han, Y., Chen, X., Wang, W., and Chawla, N. V. (2024). Tinyllm: Learning a small student from multiple large language models. arXiv e-prints, pages arXiv–2402.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Zhang, X., Pang, Y., Kang, Y., Chen, W., Fan, L., Jin, H., and Yang, Q. (2025). No free lunch theorem for privacy-preserving llm inference. Artificial Intelligence, page 104293.

Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. (2024). A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556–1577.