Quantisation

An AI model consists of billions of numbers, the parameters. Everything the model has learned is in them. Normally each number is stored very precisely, with many digits after the decimal point. Quantisation means rounding those numbers, so that each number takes up less space. The model becomes much smaller as a result.

You can compare it to a photo that you shrink in order to email it. The file becomes a lot smaller and to the eye the photo looks almost the same. Only when you zoom in do you see what has been lost.

How does it work?

The more strongly you round, the smaller the model and the bigger the difference from the original. A model that used to take up hundreds of gigabytes can thus fit in less than half the space. One trick is to round sensitive parts less aggressively than the rest, so that the loss stays limited.

You measure how much is lost by giving the shrunken model and the original the same tasks and comparing the answers. A score on a few test texts says something, but not everything, about how the model performs on real tasks.

Why it matters

This brings large models within reach of ordinary hardware. DeepSeek's open model is well over 500 gigabytes in size and normally only runs in a data centre. The company Unsloth made a heavily shrunken version of it, which runs on a powerful laptop or mini computer with 128 gigabytes of memory (link). That costs some accuracy, and the answer comes more slowly than from a chatbot in a data centre.

This means researchers, small companies and hospitals can run powerful models themselves, without sending their data elsewhere. That is the core of local AI.