What A Time To Be Alive

Glossary

Parameter

A parameter is an adjustable number in an AI model. During training, billions of these numbers are nudged a little at a time; together they hold everything the model has learnt.

An AI model has to store what it learnt during training somewhere. It does not do that as separate facts or sentences, but as numbers: the parameters, also called weights. Each number determines how strongly a signal passes through the network. During training the numbers are adjusted slightly millions of times, until the model does its job well.

The number of parameters says something about the size of a model. More parameters usually means a model can store more and handle more complex tasks, but also that it needs more memory and computing power. The largest models have hundreds of billions to more than a trillion parameters, while a small model that runs on a phone has a few tens of millions.

Not every parameter has to take part every time. In a model built as a mixture of experts, only part of the network is used at each step, so a large model can still work quickly and efficiently.

Why it matters

The Chinese company Meituan released a model with 1.6 trillion parameters, of which only about 48 billion are used for each piece of text. At the other end are text-to-speech models with 31 million parameters, small enough to run on a phone. In between is a growing group of people who want to use large open models on their own computer. Unsloth helps them by storing the parameters of a large model less precisely, so that the model needs much less memory.

Articles about Parameter