OpenAI presents GPT-4: a model that also understands images and passes exams

SAN FRANCISCO, MARCH 2023 - OpenAI has presented GPT-4, the successor to the language model behind ChatGPT. Besides text, the new model can also read images as input. It answers in text. On a practice exam for lawyers, the American bar exam, it scores among the best ten per cent of candidates, according to OpenAI.

What GPT-4 can do

A language model receives an instruction and writes the answer. With earlier versions, that instruction consisted of text only. GPT-4 also accepts images, such as a photo, a table or a sketch, and can answer questions about them. It writes text back, not images.

During a conversation, the model remembers a limited amount of text. In the standard version that is 8,000 pieces of text, where a piece is a word or part of a word. A special version for developers can handle 32,000 pieces.

OpenAI says the model reaches human level on many professional and academic tests. In many real-life situations it is still less capable than a person. Like its predecessors, GPT-4 can make up things that look convincing. This is called hallucinating.

How the model is made

GPT-4 is a Transformer, the design that looks at all the words of a text at the same time. It is made in two phases.

First, predicting. The model reads enormous amounts of text and learns to predict what comes next after each piece. In this way it learns not only words, but also facts and connections, as far as these appear in the text.

Then, fine-tuning. A second phase follows, in which the model learns to behave better as intended and to make up less. As with ChatGPT, people rate answers for this. OpenAI says this improves factual accuracy and makes the model keep to the desired rules of behaviour.

Predicting before building

Training a model like this takes a lot of computing work. That is why OpenAI built methods that behave predictably as they get bigger. The team was able to predict some of GPT-4's performance from smaller models, trained with at most a thousandth of the computing work. This builds on the earlier finding that models improve predictably with more scale.

What OpenAI does not say

The company does not say how big the model is, which texts it has read or how much computing work it took. This is different from earlier models.

Looking back from 2026

GPT-4 soon powered the paid version of ChatGPT, while the free version kept running on GPT-3.5. Image input only came for ChatGPT Plus users in September 2023. Researchers noted that an exam score of a language model is hard to interpret: it can score well on a test without being equally reliable elsewhere. Even so, GPT-4 showed how much such models can handle: on the American medical licensing exam it scored more than 20 points above the pass mark.

The step is important because language models became tools for tasks that need knowledge and reasoning, and not only for writing text. Since 2023, many newer models have appeared, including from OpenAI. But the combination of text and images, and predicting in advance what a larger model will deliver, remains a building block of how new models are made.