What A Time To Be Alive

Glossary

Language model

A language model is an AI model trained on vast amounts of text, so it can understand, write and translate text. The chatbots you know run on a language model.

A language model is an AI model that has learned from vast amounts of text. That lets it answer questions, write and summarise texts, translate and produce programming code. The well-known chatbots run on a language model. With a large language model, or LLM, both the model itself and the amount of text it was trained on are very large.

How does it work?

At its core, a language model learns to predict the next piece of text. It reads a passage, guesses what comes next and checks how far off it was. This happens billions of times. That is how it picks up patterns in language, but also in the knowledge contained in those texts. The text is chopped into small pieces, the tokens, and the model works with those pieces.

What the model has learned is stored in its parameters: billions of numbers. An answer usually emerges piece by piece, with each new piece depending on what is already there. Some models work differently. Mercury 3, from the American company Inception, first makes a rough version of the whole text and sharpens it in a few rounds. According to the company, that makes it four to eight times as fast as comparable models (link).

Why it matters

Language models are the engine behind almost all AI news. They form the basis of AI agents that carry out tasks on their own. They are also used in science, for example to design proteins or devise experiments.

Training such a model takes a lot of computing power, and a great deal of work goes into reducing that. Microsoft AI has finished the first training phase of its base model MAI-Base-2. According to the company, the model reaches the same level as a strong Chinese model with 4 to 12 times less computation (link). Researchers measure how good a language model is with fixed tests, the benchmarks.

Articles about Language model