A strategy to the reduction of communication overhead and overfitting in Federated Learning
Resumo
Federated learning (FL) is a framework to train machine learning models using decentralized data, especially unbalanced and non-iid. Adaptive methods can be used to accelerate convergence, reducing the number of rounds of local computation and communication to a centralized server. This paper proposes an adaptive controller to adapt the number of epochs needed that employs Poisson distribution to avoid overfitting of the aggregated model, promoting fast convergence. Our results indicate that increasing the local update of the model should be avoided, but yet some complementary mechanism is needed to model performance. We evaluate the impact of an increasing number of epochs of FedAVG and FedADAM.
Referências
Abdulrahman, S., Tout, H., Ould-Slimane, H., Mourad, A., Talhi, C., and Guizani, M. (2021). A survey on federated learning: The journey from centralized to distributed on-site learning and beyond. IEEE Internet of Things Journal, 8(7):5476–5497.
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konecny, J., McMahan, H. B., Smith, V., and Talwalkar, A. (2019). Leaf: A benchmark for federated settings.
Chen, M., Mathews, R., Ouyang, T., and Beaufays, F. (2019). Federated learning of out-of-vocabulary words.
Dutta, S., Joshi, G., Ghosh, S., Dube, P., and Nagpurkar, P. (2018). Slow and stale gradients can win the race: Error-runtime trade-offs in distributed sgd.
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage, D. (2019). Federated learning for mobile keyboard prediction.
Hsu, T.-M. H., Qi, H., and Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification.
Kairouz, P., McMahan, H. B., Avent, B., and et al. (2021). Advances and open problems in federated learning.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. (2021). Scaffold: Stochastic controlled averaging for federated learning.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020). Federated optimization in heterogeneous networks.
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data.
McMahan, H. B. and Streeter, M. (2010). Adaptive bound optimization for online convex optimization.
Melis, L., Song, C., Cristofaro, E. D., and Shmatikov, V. (2018). Exploiting unintended feature leakage in collaborative learning.
Mukherjee, A. and Rojas, B. internet of (jul-2020). things use case data, the long-term market perspecBusiness models for storage of idc report tive, https://www.idc.com/getdoc.jsp?containerid=prap46737220.
Ramaswamy, S., Mathews, R., Rao, K., and Beaufays, F. (2019). Federated learning for emoji prediction in a mobile keyboard.
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konecn´y, J., Kumar, S., and McMahan, H. B. (2020). Adaptive federated optimization.
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. (2020). Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization.
So, J., Guler, B., and Avestimehr, A. S. (2021). Turbo-aggregate: Breaking the quadratic aggregation barrier in secure federated learning.
Wahab, O. A., Mourad, A., Otrok, H., and Taleb, T. (2021). Federated machine learning: Survey, multi-level classification, desirable criteria and future directions in communication and networking systems. IEEE Communications Surveys Tutorials, 23(2):1342– 1397.
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. (2020). Federated learning with matched averaging.
Wang, J., Xu, Z., Garrett, Z., Charles, Z., Liu, L., and Joshi, G. (2021). Local adaptivity in federated learning: Convergence and consistency.
Yu, T., Bagdasaryan, E., and Shmatikov, V. (2020). Salvaging federated learning by local adaptation.
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konecny, J., McMahan, H. B., Smith, V., and Talwalkar, A. (2019). Leaf: A benchmark for federated settings.
Chen, M., Mathews, R., Ouyang, T., and Beaufays, F. (2019). Federated learning of out-of-vocabulary words.
Dutta, S., Joshi, G., Ghosh, S., Dube, P., and Nagpurkar, P. (2018). Slow and stale gradients can win the race: Error-runtime trade-offs in distributed sgd.
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage, D. (2019). Federated learning for mobile keyboard prediction.
Hsu, T.-M. H., Qi, H., and Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification.
Kairouz, P., McMahan, H. B., Avent, B., and et al. (2021). Advances and open problems in federated learning.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. (2021). Scaffold: Stochastic controlled averaging for federated learning.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020). Federated optimization in heterogeneous networks.
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data.
McMahan, H. B. and Streeter, M. (2010). Adaptive bound optimization for online convex optimization.
Melis, L., Song, C., Cristofaro, E. D., and Shmatikov, V. (2018). Exploiting unintended feature leakage in collaborative learning.
Mukherjee, A. and Rojas, B. internet of (jul-2020). things use case data, the long-term market perspecBusiness models for storage of idc report tive, https://www.idc.com/getdoc.jsp?containerid=prap46737220.
Ramaswamy, S., Mathews, R., Rao, K., and Beaufays, F. (2019). Federated learning for emoji prediction in a mobile keyboard.
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konecn´y, J., Kumar, S., and McMahan, H. B. (2020). Adaptive federated optimization.
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. (2020). Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization.
So, J., Guler, B., and Avestimehr, A. S. (2021). Turbo-aggregate: Breaking the quadratic aggregation barrier in secure federated learning.
Wahab, O. A., Mourad, A., Otrok, H., and Taleb, T. (2021). Federated machine learning: Survey, multi-level classification, desirable criteria and future directions in communication and networking systems. IEEE Communications Surveys Tutorials, 23(2):1342– 1397.
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. (2020). Federated learning with matched averaging.
Wang, J., Xu, Z., Garrett, Z., Charles, Z., Liu, L., and Joshi, G. (2021). Local adaptivity in federated learning: Convergence and consistency.
Yu, T., Bagdasaryan, E., and Shmatikov, V. (2020). Salvaging federated learning by local adaptation.
Publicado
16/08/2021
Como Citar
BARROS, Alex; ROSÁRIO, Denis; CERQUEIRA, Eduardo; FONSECA, Nelson L. S. da.
A strategy to the reduction of communication overhead and overfitting in Federated Learning. In: WORKSHOP DE GERÊNCIA E OPERAÇÃO DE REDES E SERVIÇOS (WGRS), 26. , 2021, Uberlândia.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 1-13.
ISSN 2595-2722.
DOI: https://doi.org/10.5753/wgrs.2021.17181.
