Network of artificial neurons learns from its own mistakes

LONDON, OCTOBER 1986 - The journal Nature publishes a short paper by David Rumelhart, Geoffrey Hinton and Ronald Williams on a way to let a network of artificial neurons learn from its own mistakes. The method is called back-propagation: working the error backwards through the network.

A network of small calculating units

An artificial neural network is a computer program loosely modelled on the brain. It consists of layers of simple units, the artificial neurons. Each unit receives numbers from the units in the layer before it. Each connection has a strength: the number that passes through it is multiplied by that strength. The unit adds everything up and passes the result on to the next layer. The input goes in at the front, for example the brightness values of a picture. The answer comes out at the back, for example what the picture shows.

So what such a network does depends entirely on those strengths. Nobody writes rules for it. The network itself has to find the right strengths.

Learning means adjusting

This is how it works. The network is shown an example for which the right answer is known. It gives its own answer, and the difference from the right answer is the error. Then all the strengths are adjusted a tiny bit, so that the error for this example becomes smaller. This is repeated with many examples.

Compare it with a mixing desk with hundreds of knobs, where the sound is not right. For each knob, the question is: should it go a little higher or lower, and by how much? The knobs are the connection strengths and the sound is the network's answer.

Where it gets difficult

For the connections just before the output this is manageable, because there it is directly visible how each strength affects the error. But between input and output there are hidden layers. For those units there is no right answer to check against, so nobody knows what they should do. Earlier, simpler learning rules only work for networks without such in-between layers.

Working backwards, layer by layer

The new method starts at the back, where the error can be seen. First it calculates how much each unit in the last layer contributes to that error. With that, it can work out the layer before, and so it goes back layer by layer to the input. At each step it uses what it has just calculated, so the work does not have to be done again and again. For each connection, the result is a number: if this strength becomes slightly larger, does the error grow or shrink, and by how much? That is exactly the answer needed for each knob.

Then comes the adjustment itself. Each strength moves a small step in the direction in which the error gets smaller. The method does this in small steps and with many examples, until the error is small enough.

What it achieves

The researchers write that, as a result, the hidden units come to represent important features of the task by themselves, even though nobody told them to. So the network finds its own in-between steps. That is what sets the method apart from the simpler learning rules of the past.

Looking back from 2026

The idea did not come out of nowhere. The researcher Seppo Linnainmaa described the calculation behind working the error backwards as early as 1970, and Paul Werbos applied it to networks with several layers in 1982. Rumelhart came up with the method independently around the same time. The 1986 paper made it widely known.

Since then, almost all modern AI models have been trained this way, right up to the largest language models. Large networks do need a great many examples and a lot of computing power, and those only came later. In 2012 a team in Toronto trained a network with 60 million connection strengths on graphics chips and won an image recognition competition with it. This is how a method from 1986 became the engine of today's AI.