Efficiently fine-tuning language models to detect anomalies in private logs using Federated Learning

• Gabriel U. Talasso UNICAMP http://orcid.org/0009-0007-6422-9735
• Allan M. de Souza UNICAMP
• Daniel Guidoni UFOP
• Eduardo Cerqueira UFPA
• Leandro A. Villas UNICAMP

DOI: https://doi.org/10.5753/sbrc.2025.5859

Abstract

The rapid growth of distributed systems in computer networks has heightened concerns regarding vulnerabilities, failures, and malicious attacks, making anomaly detection a crucial task to ensure the reliability and security of these systems. System log analysis emerges as a promising approach to identify anomalous behaviors, but it faces significant challenges, such as lack of flexibility, computational efficiency, and adaptability for distributed scenarios with resource constraints, as well as issues related to data privacy. This work investigates the use of language models combined with training and communication efficiency techniques within the context of federated learning, aiming to enhance anomaly detection in challenging scenarios. The proposed approach enables collaborative and private training across multiple clients, preserving data privacy while optimizing efficiency in resource-constrained environments. Results demonstrate the successful development of a workflow for fine-tuning language models in anomaly detection, with detailed performance analysis achieving an F1 score greater than 98%, along with up to a 4000x reduction in the size of transmitted messages, outlining promising guidelines for future advancements in the field.

Keywords: Anomaly Detection, Security, Federated Learning, Distributed Learning, Language Models

References

Allal, L. B., Lozhkov, A., Bakouch, E., von Werra, L., and Wolf, T. (2024). Smollm - blazingly fast and remarkably powerful.

Almodovar, C., Sabrina, F., Karimi, S., and Azad, S. (2024). Logfit: Log anomaly detection using fine-tuned language models. IEEE Transactions on Network and Service Management.

Cho, Y. J., Wang, J., and Joshi, G. (2020). Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243.

Devlin, J. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Guan, W., Cao, J., Qian, S., and Gao, J. (2024). Logllm: Log-based anomaly detection using large language models. arXiv preprint arXiv:2411.08561.

Guo, H., Yuan, S., and Wu, X. (2021). Logbert: Log anomaly detection via bert. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685.

Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526.

Li, B., Ma, S., Deng, R., Choo, K.-K. R., and Yang, J. (2022). Federated anomaly detection on system logs for the internet of things: A customizable and communication-efficient approach. IEEE Transactions on Network and Service Management, 19(2):1705–1716.

Liu, Y. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282. PMLR.

Pang, G., Shen, C., Cao, L., and Hengel, A. V. D. (2021). Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1–38.

Qi, J., Huang, S., Luan, Z., Yang, S., Fung, C., Yang, H., Qian, D., Shang, J., Xiao, Z., and Wu, Z. (2023). Loggpt: Exploring chatgpt for log-based anomaly detection. In 2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pages 273–280. IEEE.

Souza, A., Bittencourt, L., Cerqueira, E., Loureiro, A., and Villas, L. (2023). Dispositivos, eu escolho vocês: Seleção de clientes adaptativa para comunicação eficiente em aprendizado federado. In Anais do XLI Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos, pages 1–14, Porto Alegre, RS, Brasil. SBC.

Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2024). A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness. arXiv preprint arXiv:2411.03350.

Xu, W., Huang, L., Fox, A., Patterson, D., and Jordan, M. I. (2009). Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, page 117–132, New York, NY, USA. Association for Computing Machinery.

Ye, R., Wang, W., Chai, J., Li, D., Li, Z., Xu, Y., Du, Y., Wang, Y., and Chen, S. (2024). Openfedllm: Training large language models on decentralized private data via federated learning.