OpenAI shows that language models improve in a predictable way as they get bigger

SAN FRANCISCO, JANUARY 2020 - How much better does a language model get when you make it bigger? Researchers at OpenAI have measured it, and the answer is strikingly calm: the gain follows a neat, predictable line. This lets companies work out in advance roughly what a bigger model will deliver.

A model that guesses the next word

A language model is a program that predicts the next word in a text, like the suggestions on a phone, but much smarter. It learns this by reading huge amounts of text and guessing, at every word, what comes next. A neural network, the kind of program used here, consists of many numbers that are adjusted again and again during learning. These numbers are called parameters.

To measure how good a model is, the researchers use an error measure. It tells how surprised the model is by the word that really follows. The lower the measure, the better the model predicts the text.

Three ingredients

A model can grow in three ways. It can use more parameters, which means a bigger network. It can read more text. And it can use more computing work, which means more chips and more time.

OpenAI trained many models of different sizes and looked at how the error changed. When one of the three ingredients grew and the other two were not a limit, the error fell following a fixed pattern. In plain words: every time you make one ingredient ten times bigger, the error becomes smaller by the same fixed factor. This pattern held over more than seven steps of ten times bigger, so over a difference of well over ten million times.

What else stood out

Within a wide range, the design of the network hardly mattered. A wider or deeper network made barely any difference; the total number of parameters was what counted. Bigger models also learned more from the same amount of text. The researchers also found simple formulas for when a model starts to memorise too much of its training text, and for how fast it learns.

This leads to a practical choice for anyone with a fixed budget of computing work. According to the researchers, it is then smartest to train a very large model on a not-too-large amount of text, and to stop before the model has finished learning.

Caveats

This is a measurement of the error when predicting text, not of what a model can do with it. The line does not show whether a model will translate or reason better later. Still, for a company, predictability is already worth a lot.

Looking back from 2026

Four months later OpenAI described GPT-3. With 175 billion parameters, it was more than a hundred times bigger than its predecessor. In 2022, DeepMind showed with its model Chinchilla that the split must be different: the model and the amount of text should grow roughly in proportion. Chinchilla had 70 billion parameters, was trained on about four times as much text as the 280-billion-parameter model Gopher, and performed better. So the lines from 2020 stayed correct; only the best split turned out to be different.

The predictability is still used today. In the technical report on GPT-4 from 2023, OpenAI writes that it could predict part of the performance in advance using models trained with at most a thousandth of the computing work. This is one reason why companies have started to invest in ever bigger AI data centres: anyone who knows what more computing work delivers can better decide whether it is worth it.