Train Less, Predict More: Federated Learning plugin enables high efficiency on heterogeneous data

• Cláudio G. S. Capanema UFMG
• Joahannes B. D. da Costa UNICAMP
• Fabrício A. Silva UFV
• Leandro A. Villas UNICAMP
• Antonio A. F. Loureiro UFMG

DOI: https://doi.org/10.5753/courb.2024.3243

Abstract

Federated learning (FL) emerged as a technique where several devices (also called clients) can learn collaboratively from the orchestration of a central server, providing scalability, privacy and low communication costs. Most research on this topic presents proposals for the model training stage in federated learning, to address various problems such as statistical data heterogeneity, which often represents increased costs (e.g., computational, storage and communication). However, the FedPredict solution was recently proposed, a plugin that operates in the prediction stage of federated learning, which when added can significantly improve the performance of several traditional solutions in data heterogeneity scenarios, without requiring any modification to their original structure or additional training. In this direction, this work presents experiments on a new discovery: the more heterogeneous the data, the less training is needed when FedPredict is added, making the learning process highly efficient.

References

Arivazhagan, M. G., Aggarwal, V., Singh, A. K., and Choudhary, S. (2019). Federated learning with personalization layers. arXiv preprint arXiv:1912.00818.

Capanema, C. G., de Oliveira, G. S., Silva, F. A., Silva, T. R., and Loureiro, A. A. (2023a). Combining recurrent and graph neural networks to predict the next place’s category. Ad Hoc Networks, 138:103016.

Capanema, C. G., de Souza, A. M., Silva, F. A., Villas, L. A., and Loureiro, A. A. (2023b). Fedpredict: Combining global and local parameters in the prediction step of federated learning. In 2023 19th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT), pages 17–24. IEEE.

Capanema, C. G., de Souza, Joahannes B D da Costa, F. A., Villas, L. A., and Loureiro, A. A. L. (2024). A modular plugin for concept drift in federated learning. In 2024 20th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT). IEEE.

Capanema, C. G., Silva, F. A., Silva, T. R., and Loureiro, A. A. (2021). Poi-rgnn: Using recurrent and graph neural networks to predict the category of the next point of interest. In Proceedings of the 18th acm symposium on performance evaluation of wireless ad hoc, sensor, & ubiquitous networks, pages 49–56.

Capanema, C. G. S., Silva, F. A., and Silva, T. R. d. M. B. (2020). Mfa-rnn: Uma rede neural recorrente para predição de próximo local de visita com base em dados esparsos. In Anais do XXXVIII Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuidos, pages 127–140. SBC.

Cho, Y. J., Wang, J., and Joshi, G. (2022). Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics, pages 10351–10375. PMLR.

Fu, L., Zhang, H., Gao, G., Zhang, M., and Liu, X. (2023). Client selection in federated learning: Principles, challenges, and opportunities. IEEE Internet of Things Journal.

Hsu, T.-M. H., Qi, H., and Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.

Jang, J., Ha, H., Jung, D., and Yoon, S. (2022). Fedclassavg: Local representation learning for personalized federated learning on heterogeneous neural networks. In Proceedings of the 51st International Conference on Parallel Processing, pages 1–10.

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.

Lin, T., Kong, L., Stich, S. U., and Jaggi, M. (2020). Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363.

Maciel, F., De Souza, A. M., Bittencourt, L. F., and Villas, L. A. (2023). Resource aware client selection for federated learning in iot scenarios. In 2023 19th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT), pages 1–8. IEEE.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR.

Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečnỳ, J., Kumar, S., and McMahan, H. B. (2020). Adaptive federated optimization. arXiv preprint arXiv:2003.00295.

Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. (2011). The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In IEEE International Joint Conference on Neural Networks, pages 1453–1460.

Tan, A. Z., Yu, H., Cui, L., and Yang, Q. (2022a). Towards personalized federated learning. IEEE Trans. on Neural Networks and Learning Systems.

Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., and Zhang, C. (2022b). Fedproto: Federated prototype learning across heterogeneous clients. In AAAI Conference on Artificial Intelligence, volume 1, page 3.

Wu, C., Wu, F., Lyu, L., Huang, Y., and Xie, X. (2022). Communication-efficient federated learning via knowledge distillation. Nature communications, 13(1):2032.