Curriculum learning

Curriculum learning is a training strategy in machine learning where the examples used to train the model are presented to the model in a meaningful order, from easiest to hardest. This is done in an attempt to increase the speed and effectiveness of the learning process, as well as the final performance of the model. The concept is inspired by the way humans learn, where new knowledge is often presented in a structured and scaffolded way. With curriculum learning, the idea is that by starting with simpler examples and gradually increasing the complexity of the examples, the model will be able to better learn the underlying patterns and features of the data.

Methods

A simple way to do automated curriculum learning is to ignore training examples with high noise. Min-SNR found that by attenuating the loss of Stable Diffusion examples below a SNR of 5 sped up training convergence by 3.4x.

Applications

Curriculum learning has been used across a variety of machine learning domains, including computer vision, natural language processing, and reinforcement learning. Some researchers have also investigated the use of curriculum learning in multi-task learning, where the model has to learn multiple tasks simultaneously.