NederlandsBenchmark
A benchmark is a fixed test for comparing AI models. Every model takes the same tasks, and the score shows what it can and cannot yet do.
A benchmark is a yardstick for AI. Researchers devise a set of tasks, have each model attempt them and count the correct answers. Because everyone gets the same test, models can be compared fairly. It is like a final exam: all students answer the same questions, and the grades are comparable.
How does it work?
A benchmark has three parts: the tasks, a way of checking the answer and a score. Sometimes the task is a multiple-choice question. Sometimes it is an entire working process. With the GDPx test from the research firm SemiAnalysis, for example, a model has to work for hours in a simulated workplace, with real programs and thousands of files. The best model scored 12 per cent there (link).
A good benchmark has to stay hard enough. If the best models score close to the maximum, the test can no longer show differences between new models. That is why new, tougher tests keep appearing. It is also important that the tasks are not online, so that a model does not already know them.
Why it matters
Benchmarks are how you can track progress. Almost every new model comes with scores, which quickly give an impression of what the model is strong at. A benchmark does measure just one thing, though. Anyone who wants to know whether a model suits their own work looks at which tasks the test really measures. Sometimes the scores also come from the company itself, or from a test that others have not yet repeated.
An example of a test that measures something new is TasteVal. On it, an AI model came up with better set-ups for AI research than the best of 24 human experts, in less than half the computing time (link). A result like that shows how quickly models are improving at a kind of work through which AI can improve itself.
Articles about Benchmark
-
GPT-6 Cyber found 81 serious flaws in Firefox and built 948 test attacks on Linux
OpenAI's security model GPT-6 Cyber found 81 serious flaws in Firefox, the company says, and built 948 working attacks on Linux flaws in a test environment. Mozilla has already released many fixes, and repairs have also begun for Linux.
-
New AI memory condenses a whole stack of documents into 16 tokens for each question
Researchers from Stanford and other institutions built Memorilla, a memory that condenses a whole collection of documents into 16 tokens for each question. In their own tests, a language model usually scores better with it than with all the text, and answers much faster. Exact names are sometimes lost.
-
Thinking Machines Lab releases Inkling 2 Small, open version to follow after safety tests
The AI company of former OpenAI executive Mira Murati has started its second generation of models with Inkling 2 Small. The model is available through the company's own platform, Tinker. The company wants to release it next week, once the external safety tests are finished.
-
NVIDIA releases fast planner for robot arms: route in 4.4 milliseconds on average
NVIDIA researchers have released cuRoboV2-Fast, software that works out how a robot arm moves without hitting anything. In their own tests, this took 4.4 milliseconds on average. On a real robot, a work cycle became more than a third shorter than with its predecessor.
-
OpenAI launches GPT-6 Cyber: better at finding security flaws, less likely to step out of line
OpenAI has released a new cybersecurity model that, according to the company, finds vulnerabilities far better and breaks into systems without being asked less often. At the same time, the company is rolling out GPT-6 Sol and Luna to all ChatGPT users.
-
Baidu releases large AI model ERNIE 5.5 Flash Base, but not for Europe
The Chinese tech company Baidu has released a large AI model that developers can train further themselves. It is built to be fast and economical, and Baidu says it scores well in tests. The licence does not apply in the EU, the United Kingdom and South Korea.
-
Small open AI model reads text aloud on your phone, without internet
The AI company Gradium has released Phonon as open models: small text-to-speech models that read text aloud with a natural voice, entirely on a phone, a laptop or in the browser. On a public Hugging Face leaderboard, Phonon is among the most intelligible text-to-speech models.
-
Unsloth shrinks a large open AI model from Alibaba and, by its own tests, makes it more accurate than other versions
Unsloth has released new, smaller versions of Qwen3.8-Flash-Next, a large open AI model from Alibaba. By its own tests, they score 15 per cent higher on average than other shrunken versions. This lets more people run the model on their own computers.
-
Tencent releases Hy4: an open AI model that understands images and operates a computer
China's Tencent has released Hy4, a large language model that understands images for the first time and can operate a computer on its own. According to Tencent, it makes things up far less often than the previous version. The model files will be freely available in the coming days.
-
Xpeng's small open AI model locates objects better than GPT-6 Astra
Carmaker Xpeng and several universities have released GroundingPI, an open AI model that helps robots and cars see exactly where things are. On 34 tests it scores slightly higher on average than OpenAI's much larger GPT-6 Astra, according to its makers.
-
OpenAI uses its AI for hardware design: from 3D models to circuit boards
OpenAI shows how ChatGPT and Codex help engineers design physical products. Two employees had the GPT-6 Astra model design a working prototype in less than a week. Getting it made in a factory still took several rounds of checks by humans.
-
Anthropic builds FlashProve: cryptographic proof of which AI model produced an answer
Anthropic has released FlashProve, an open-source research prototype that uses zero-knowledge cryptography to show which model generated an answer, without revealing the model weights.
-
Chinese company releases free AI model for chemistry, trained on its own lab work
The Chinese company Deep Principle has released Hydrogen-1.0, an AI model for chemistry and materials research that anyone can download. It also learned from failed experiments in the company's own lab. The results have so far been measured by the company itself.
-
New test with real professional work: best AI model scores 12 per cent
The research firm SemiAnalysis let AI models work for hours in simulated workplaces, from chip design to stock trading. Claude Opus 5.5 came first with a score of 12 per cent. The test shows where AI is already strong and what the next step is.
-
Meituan releases LongCat-2.5: a multimodal agent model with 1.6 trillion parameters, trained on Chinese chips
On 2 November the Chinese company Meituan released LongCat-2.5, a large AI model that handles text and images and can carry out long programming tasks on its own. According to Meituan, it was trained entirely on Chinese chips and comes close to Claude Opus 5 in tests.
