Google researchers: a network that looks at all words at once translates better and learns faster

MOUNTAIN VIEW, JUNE 2017 - Eight Google researchers have described a new way to let computers process text. Their system is called the Transformer. It translates from English into German and French better than all earlier methods, and it is much faster to train. The paper is titled "Attention Is All You Need".

How a computer has read text until now

Programs for translating and writing are built from neural networks. These are computer programs that adjust themselves on the basis of very many examples. The usual kind is called a recurrent network. It reads a sentence word by word, from left to right, and after each word it carries a kind of summary on to the next one.

This has two drawbacks. The first is distance: if the program wants to link a word to a word much earlier in the sentence, that information has to pass through all the words in between. Earlier research suggests that this makes the network harder to train. The second drawback is speed. Each word has to wait for the word before it, so the work cannot be spread over the many computing units of a modern graphics chip, which are designed to do many things at once.

Attention: every word looks at all the others

The Transformer does not read in a row. It looks at the whole sentence in one go. For each word, the system asks: how important is every other word for understanding this word? Each word gets a number for this. The system then counts in the other words in proportion to their numbers. A word with a high number counts for a lot, a word with a low number hardly counts. This is called attention.

Google gives an example itself. In the sentence "I arrived at the bank after I had crossed the river", "bank" can be a financial institution or the side of a river. The word "river" gets a high attention number, so the system understands in one step that the river side is meant. A second example: in French, the translation of "it" depends on the word it refers to. The system gave the right translation for an animal and for a street.

Reading and writing in two parts

For translation, the Transformer uses two parts. The first part reads the sentence in the source language and makes a description for each word that takes the rest of the sentence into account. The second part writes the translation word by word. In doing so, it looks at the words it has already written and at the descriptions from the first part.

Because all words are looked at at the same time, the work can be spread over many chips. The network learned to translate English into French in 3.5 days on eight graphics chips. According to the researchers, that is a small part of what the best earlier models cost.

What it delivers

On the standard test for English to German, the Transformer scored 28.4 points on the translation measure BLEU, more than 2 points above the best earlier result. This measure compares computer translations with translations by people. English to French scored 41.8, a new record for a single model. The system also did well on another task, the analysis of sentence structure.

Looking back from 2026

All eight authors worked at Google. They later left the company, to work for other companies or to start their own. In December 2017, they presented the paper at a major conference in the field.

The Transformer became the basis of almost all large language models. In 2018, Google released BERT, which uses the Transformer to understand text. OpenAI's GPT models use the part that writes text. The chatbot ChatGPT, at the end of 2022, built on that. The same technique is also used for images and video.

The importance lies in scaling. Because the Transformer can do many things at once during training, models could be made bigger and bigger, using more and more chips. The Transformer is therefore one of the building blocks under almost all AI news of today.