OpenAI presents GPT-3: a language model that learns tasks from a few examples in the prompt

SAN FRANCISCO, MAY 2020 - OpenAI has published a scientific paper about GPT-3, a language model of unprecedented size. The model has 175 billion parameters, more than ten times as many as any earlier language model of this type. The most striking thing is not its size. It is that GPT-3 picks up a new task when it is shown a few examples in the prompt, without the model being trained again.

A model that guesses the next piece of text

At its core, GPT-3 does the same as the suggestions on a phone: it predicts what comes next in a text. The model is a neural network, a computer program with a great many numbers that are adjusted during learning. These numbers are called parameters. GPT-3 has 175 billion of them, compared with 1.5 billion for its predecessor GPT-2 from 2019.

It learned from about 499 billion pieces of text. These came from web pages, books and Wikipedia. A piece, called a token in technical language, is a word or part of a word. The model looks at the last 2,048 pieces each time and chooses what is most likely to follow. It is built on the Transformer, the design from Google in 2017 that looks at all words at the same time.

Learning from a few examples

Until now, a language model had to be trained separately for each task. If you wanted translations, you gave it thousands of translated sentences and adjusted the parameters. GPT-3 does not need that. In the prompt itself, you write a few examples, for instance three English words with their translations, and then the start of a new word. The model completes the pattern. Nothing changes in the parameters; everything is in the text of the prompt.

According to the paper, GPT-3 performs well in this way at translating, answering questions and completing sentences. It can also put the letters of a word in the right order, use a newly invented word in a sentence and add numbers of three digits.

Can people tell the difference?

The researchers had the model write news articles. Eighty American participants had to guess whether such an article came from a human or from GPT-3. They were right in 52 per cent of cases, barely better than guessing.

What it cannot do yet

The model is not flawless. On some tests, learning from a few examples remains difficult. Because it learns from a large part of the internet, it can also imitate incorrect or harmful texts, and the makers themselves note concerns about risks of this kind. It is a first step, not a ready-made solution.

Looking back from 2026

OpenAI made GPT-3 available to developers on 11 June 2020, first in a test phase. In September 2020, Microsoft received the exclusive licence to the model. In 2022 came InstructGPT, a version adjusted by humans to follow instructions better, and then GPT-3.5. The chat programme ChatGPT, released in November 2022, ran on GPT-3.5.

The question raised by this milestone has remained: does scaling up keep producing new skills? Learning from examples in the prompt was not built in separately. It turned out to appear as the model grew larger. Since then, companies have built ever larger models, partly because the gains are predictable according to earlier research.