ImageNet: 3.2 million photos, sorted by hand, to teach computers to see

PRINCETON, JUNE 2009 - At CVPR, the conference for research on computers that understand images, researchers from Princeton University present a new database: ImageNet. Fei-Fei Li's team describes a collection of 3.2 million photos in 5,247 categories, checked by hand. According to the makers, it is the largest clean photo collection available to researchers, in number of photos, in photos per category and in number of categories. The goal is much bigger still: about 50 million photos within two years.

Why examples are needed

A computer does not recognise a dog because someone explains to it what a dog looks like. It learns from examples: thousands of photos labelled 'dog'. The more examples there are, and the more varied they are, the better it learns. Until now, such collections were small or full of errors. The internet now holds billions of photos, but without a label the computer does not know what they show.

Step 1: choose the categories

ImageNet uses the structure of WordNet, a dictionary that sorts some 80,000 nouns into groups of meaning, arranged like a tree. Under 'mammal', for example, there is 'dog', under that 'working dog' and under that 'husky'. Each end point of that tree becomes a category. The structure is so detailed that it includes 147 kinds of dog, something no earlier collection has.

Step 2: collect candidates

For each category, the researchers ask image search engines on the internet for photos. That brings in a lot of junk: on average only about 10 per cent of the search results are correct. So they collect a large number, on average more than 10,000 per category, and also search with translations of the word into Chinese, Spanish, Dutch and Italian.

Step 3: let people sort them

Next, people check every photo through Amazon Mechanical Turk. This is a website where small tasks can be posted for others to do for payment. The workers get a pile of photos and the description of the category, and say whether each photo belongs. Because people make mistakes and sometimes disagree, several people look at the same photo independently. A photo only counts if a clear majority says yes. Easy categories such as 'cat' need fewer votes than 'Burmese cat'. The researchers first let at least ten people vote on a sample, and use that to work out how many votes each category needs.

The method works: in a sample of 80 categories, on average 99.7 per cent of the photos turned out to be sorted correctly.

Looking back from 2026

The sorting went on until April 2010. From 2010, the ImageNet team held a yearly contest using part of the collection: 1,000 categories and more than 1.2 million photos to learn from. In 2012, AlexNet won that contest with a neural network, making almost 11 percentage points fewer errors than the runner-up. That put the idea of deep learning on the map. The contest ended after 2017 because the task was considered solved. ImageNet itself now holds more than 14 million photos in more than 20,000 categories.

ImageNet shows that good AI needs not only clever calculation rules but also a lot of carefully sorted examples. Later steps followed that lead: first get the data right, then build the network.