Federated Learning with Accurate Model Training and Low Communication Cost in Heterogeneous Scenarios

• Lucas Airam C. de Souza UFRJ / INRIA Saclay
• Miguel Elias M. Campista UFRJ
• Luís Henrique M. K. Costa UFRJ

DOI: https://doi.org/10.5753/sbrc_estendido.2024.1633

Resumo

Federated learning (FL) is a distributed approach to train machine learning models without disclosing private data from participating clients to a central server. Nevertheless, FL performance depends on the data distribution, and the training struggles to converge when clients have distinct data distributions, increasing overall training time and the final model prediction error. This work proposes two strategies to reduce the impact of data heterogeneity in FL scenarios. Firstly, we propose a hierarchical client clustering system to mitigate the convergence obstacles of federated learning in non-Independent and Identically Distributed (IID) scenarios. The results show that our system has a better classification performance than FedAVG, increasing its accuracy by approximately 16% on non-IID scenarios. Furthermore, we improve our first proposal by implementing ATHENA-FL, a federated learning system that shares knowledge among different clusters. The proposed system also uses the one-versus-all model to train one binary detector for each class in the cluster. Thus, clients can compose complex models combining multiple detectors. ATHENA-FL mitigates data heterogeneity by maintaining the clustering step before training to mitigate data heterogeneity. Our results show that ATHENA-FL correctly identifies samples, achieving up to 10.9% higher accuracy than traditional training. Finally, ATHENA-FL achieves lower training communication costs than MobileNet architecture, reducing the number of transmitted bytes between 25% and 97% across evaluated scenarios.

Referências

Blondel, V. D. et al. (2008). Fast Unfolding of Communities in Large Networks. Journal of Statistical Mechanics: Theory and Experiment, pages 1–12.

Chu, D., Jaafar, W., and Yanikomeroglu, H. (2022). On the Design of Communication-Efficient Federated Learning for Health Monitoring. IEEE GLOBECOM, pages 1–6.

Duan, M. et al. (2022). Flexible Clustered Federated Learning for Client-Level Data Distribution Shift. Transactions on Parallel and Distributed Systems, 33(11):2661–2674.

Ghosh, A., Chung, J., Yin, D., and Ramchandran, K. (2020). An Efficient Framework for Clustered Federated Learning. arXiv preprint arXiv:2006.04088.

Li, H., Cai, Z., Wang, J., Tang, J., Ding, W., Lin, C.-T., and Shi, Y. (2023). FedTP: Federated Learning by Transformer Personalization. IEEE Transactions on Neural Networks and Learning Systems.

Liu, B. et al. (2021). When Machine Learning Meets Privacy: A Survey and Outlook. Computing Surveys (CSUR), 54(2):1–36.

Ma, X., Zhu, J., Lin, Z., Chen, S., and Qin, Y. (2022). A State-of-the-Art Survey on Solving Non-IID Data in Federated Learning. Future Generation Computer Systems, 135:244–258.

McMahan, B. et al. (2017). Communication-efficient Learning of Deep Networks from Decentralized Data. Artificial Intelligence and Statistics, pages 1273–1282.

Sattler, F., Müller, K.-R., and Samek, W. (2020). Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization under Privacy Constraints. IEEE Transactions on Neural Networks and Learning Systems.

Wang, H., Kaplan, Z., Niu, D., and Li, B. (2020). Optimizing Federated Learning on Non-IID Data with Reinforcement Learning. In IEEE INFOCOM, pages 1698–1707.

Zeng, D., Hu, X., Liu, S., Yu, Y., Wang, Q., and Xu, Z. (2023). Stochastic Clustered Federated Learning. arXiv preprint arXiv:2303.00897.

Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated Learning with Non-IID Data. arXiv preprint arXiv:1806.00582.

Zhong, Z. et al. (2022). FLEE: A Hierarchical Federated Learning Framework for Distributed Deep Neural Network over Cloud, Edge and End Device. ACM TIST, pages 1–24.

Zhu, Y., Markos, C., Zhao, R., Zheng, Y., and James, J. (2021). FedOVA: One-vs-All Training Method for Federated Learning with Non-IID Data. In IEEE IJCNN, pages 1–7.