NederlandsAI model designs experiments better than 24 experienced AI researchers
On a new test by P-Zero Research, Claude Opus 5.5 designs better AI experiments than the best of 24 human experts. The model needs less than half the computing time to do so and costs about one thirtieth as much.
Anyone who wants to improve an AI model runs experiments: you change something, train the model again and see whether it works better. Computing power is expensive, so choosing well matters. Which experiment do you try first, and what do you learn from the result? Researchers call this feel for good experiments "research taste".
The small research firm P-Zero Research has now measured that taste with a new test, TasteVal. The paper appeared in early October 2026. The best result came from Claude Opus 5.5 by Anthropic. According to the researchers, that model designs better experiments on this test than experienced human AI researchers.
How the test works
TasteVal consists of eight new research tasks that are not available anywhere online, so a model cannot already know the answers. The tasks are about things such as choosing good training data, training and fine-tuning language models, and making a model more resistant to misleading questions.
The model being tested is not allowed to write code itself. It works as a researcher: it comes up with experiments and draws conclusions from the results. A fixed programming assistant, the older model Opus 4.8, carries out all the work. This way the test measures research insight and not programming work. Each attempt gets one powerful AI chip (an Nvidia H100) and may let it compute for at most 40 hours, within a total of 120 hours.
Then 24 human experts did the same tasks. They had experience at OpenAI, Google DeepMind, Nvidia, Microsoft Research and the University of Oxford, among others. For each task, the best human attempt sets the bar.
Less computing time, much lower costs
The researchers express the result in computing time. Opus 5.5 reached the same score as the best expert with about 43 per cent of the computing time that expert needed. That is 2.3 times as efficient. The uncertainty is still large: according to the researchers, the real value lies somewhere between 1.15 and 4.4 times. An attempt by the model cost 282 dollars on average (about 250 euros), against more than 9,100 dollars (about 8,100 euros) for a human expert. That is about one thirtieth.
In total, the team tested twenty models from 2023 to 2026. This shows how fast things are moving. Since December 2025, the lead in computing time has doubled roughly every three months. Before that, it took about fourteen months. If you look only at the bare score, without counting computing time, you do not see this acceleration: that score doubled roughly every fifteen months over the whole period.
What the test does not measure
The researchers name the limits themselves. The tasks are quick and cheap to check, and real research is often messier. The test does not measure choosing a good research question: the question is always already fixed. On another test by Anthropic, TASTE, models have to choose which of two research proposals is the most promising. There, the best model scored about 60 per cent in August, against an estimated 77 per cent for humans. On that point, human researchers still have a clear lead.
Other research also shows that AI cannot yet run large research projects on its own. In a test by Epoch AI, AI agents got 3,000 hours of computing time to reinvent a recent training method by themselves. The best result reached a small part of what the human researchers had achieved.
Jack Clark, co-founder of Anthropic, wrote on 12 October in his personal newsletter Import AI that TasteVal changes his expectation. He now thinks it more likely that AI will be able to develop a stronger successor by itself as early as 2027, and not only in 2028. Clark works at the company behind the winning model; the test itself comes from the independent P-Zero Research.
What does this mean for the road to the singularity?
If AI gets better at doing AI research itself, progress can speed itself up. TasteVal is the first to put numbers on an important part of that work: choosing experiments wisely and reading the results well. In this, a model is now at the level of the best human experts, at a fraction of the cost. The next steps are clear: a second research group that repeats the result, and tests for what is not yet measured, such as choosing the right question and planning longer research projects.
What is the technological singularity? Read the explanation →
