1.Previous
3.Next

and token size, they found that "for compute-optimal training", the model size and the number of training tokens should ideally be scaled equally: for every doubling of model size the number of training tokens should also be doubled.

Compute growth and AI progress

Larger AI models trained on more data and using more computational resources, tend to perform better. This happens even if the algorithms themselves remain unchanged.

As early as 2018, OpenAI noted the exponential increase in

1.Previous
3.Next