What A Time To Be Alive

Glossary

Benchmark

A benchmark is a fixed test for comparing AI models. Every model takes the same tasks, and the score shows what it can and cannot yet do.

A benchmark is a yardstick for AI. Researchers devise a set of tasks, have each model attempt them and count the correct answers. Because everyone gets the same test, models can be compared fairly. It is like a final exam: all students answer the same questions, and the grades are comparable.

How does it work?

A benchmark has three parts: the tasks, a way of checking the answer and a score. Sometimes the task is a multiple-choice question. Sometimes it is an entire working process. With the GDPx test from the research firm SemiAnalysis, for example, a model has to work for hours in a simulated workplace, with real programs and thousands of files. The best model scored 12 per cent there (link).

A good benchmark has to stay hard enough. If the best models score close to the maximum, the test can no longer show differences between new models. That is why new, tougher tests keep appearing. It is also important that the tasks are not online, so that a model does not already know them.

Why it matters

Benchmarks are how you can track progress. Almost every new model comes with scores, which quickly give an impression of what the model is strong at. A benchmark does measure just one thing, though. Anyone who wants to know whether a model suits their own work looks at which tasks the test really measures. Sometimes the scores also come from the company itself, or from a test that others have not yet repeated.

An example of a test that measures something new is TasteVal. On it, an AI model came up with better set-ups for AI research than the best of 24 human experts, in less than half the computing time (link). A result like that shows how quickly models are improving at a kind of work through which AI can improve itself.

Articles about Benchmark