Contrastive learning

In contrastive learning, an AI model learns by comparing. It is repeatedly shown two things and has to decide whether they belong together: a photo and a caption, a crystal and its description. Sometimes the pair is right, sometimes not. By doing this thousands of times, the model learns what goes with what.

You can compare it to an estate agent who is repeatedly given a floor plan with a property description, sometimes the right one and sometimes a wrong one. After a while he can tell from a floor plan alone whether a house is spacious and light.

How does it work?

The model converts both sides of a pair into a row of numbers, a kind of fingerprint. For a matching pair, it tries to place the two fingerprints close together. For a mismatched pair, it tries to place them far apart. After many repetitions, things that belong together lie close to each other, even if they are different kinds of data, such as image and text.

The advantage is that the model does not need a human-made label for every example. It is enough to know which pairs belong together.

Why it matters

This lets you teach a model to connect two worlds. Researchers at MIT used it to get a language model to understand the structure of crystals. Language models learn mainly from text, whereas the structure of a crystal is stored in files full of coordinates. The MIT model was given the 3D structure and a plain-language description of each crystal, and so learned to connect the 3D structure of a crystal with the words that go with it (link). As a result, it could find better candidates for, say, batteries.

That is an example of what the method is good at: linking data that at first sight have nothing to do with each other.