Towards Efficient Training through Critical Periods - Fukase, Vinicius Yuiti; Gama, Heitor; Bueno, Barbara; Libanio, Lucas (Anais Estendidos da Conference on Graphics, Patterns and Images (SIBGRAPI); 2025: Anais Estendidos da XXXVIII Conference on Graphics, Patterns and Images, 2025)
Critical Learning Periods encompass an important phenomenon involving deep learning, where early epochs play a decisive role in the success of many training recipes, such as data augmentation. Existing works confirm the existence of this phenomenon and provide useful insights. However, the literature lacks efforts to precisely identify when critical periods occur. In this work, we fill this gap by introducing a systematic approach for identifying critical periods during the training of deep neural networks, focusing on eliminating computationally intensive regularization techniques and effectively applying mechanisms for reducing computational costs, such as data pruning. Our method leverages generalization prediction mechanisms to pinpoint critical phases where training recipes yield maximum benefits to the predictive ability of models. By halting resourceintensive recipes beyond these periods, we significantly accelerate the learning phase and reduce training time, energy consumption, and CO 2 emissions. Experiments on standard architectures and benchmarks confirm the effectiveness of our method. Specifically, we achieve significant milestones by reducing the training time of popular architectures by up to 59.67%, leading to a 59.47% decrease in CO 2 emissions and a 60% reduction in financial costs, without compromising performance. Our work enhances understanding of training dynamics and paves the way for more sustainable and efficient deep learning practices, particularly in resource-constrained environments. In the era of the race for foundation models, we believe our method emerges as a valuable framework. Code and supplementary material available at https://github.com/baunilhamarga/critical-periods .