NederlandsA different order makes AI servers start answering more than three times faster
Engineers from Amazon and Mistral have made the open software vLLM, which many companies use to run AI models, work together more cleverly. With long questions, the first word of the answer appeared more than three times faster in their test. The new method is free to use.
When you ask an AI chatbot a question, you first wait a moment. Then the answer appears word by word. That first wait often feels the longest, especially when you send a long document along with it. Behind the scenes, a lot happens during that time.
A language model works in two steps. First it reads the whole question in one go. While doing so, it makes a kind of notes about the text, so that it does not have to read it again and again. Then it writes the answer, one word piece at a time.
Those two steps ask different things of the computer. Reading is mainly heavy calculation, writing mainly fast back-and-forth with the memory. Large AI services therefore increasingly put them on separate servers: one reads, the other writes. The notes then have to travel from one server to the other.
Waiting for each other
The open software vLLM, which companies around the world use to run AI models on their own servers, did this in a fixed order until now. Only when the reading server was finished did the writing server start freeing up space in its memory and fetching the notes. All those steps came one after another, while the user waited.
Sunita Nadampalli of Amazon Web Services and Nicolò Lucchesi of the French AI company Mistral, who also helps maintain vLLM, turned that around. In the new method, the writing server already prepares a place in its memory while the reading server is still busy. As soon as the reading is done, the reading server sends the notes straight to that place. A separate helper program in the background takes care of this, so the calculations simply carry on. Compare it with a restaurant where the waiter already lays the table while the cook is still cooking, instead of only once the food is ready.
More than three times faster
They tested it with the open language model Qwen3-32B on two Amazon servers, each with eight Nvidia AI chips. At eight questions per second, each with a text of about 16,000 word pieces, the old method took about 1.7 seconds before the first word appeared. With the new method that was about 0.5 seconds: more than three times faster. The writing itself also went more smoothly, at about 7 thousandths of a second per word piece instead of just over 10. That is a gain of about 30 per cent. With short questions and little traffic, it makes almost no difference.
There is one trade-off. Because the writing server is chosen before the reading is done, the system is less able to switch at the last moment to the server that is least busy. For some services the old method therefore remains the better choice. The new one is in vLLM from version 0.29.0 and, like the rest of the software, is free to use. The measurement was made in a single set-up; how much gain it brings in other situations will become clear from use in practice.
What does this mean for the road to the singularity?
AI does not only get better through smarter models, but also through smarter work around them. This improvement gets more out of the same chips: no new hardware, just a better order. That makes long tasks in particular faster and cheaper, and those are becoming more important now that AI agents read through whole documents, folders and programs before they get to work. Strikingly, the idea comes from ordinary life: preparing while you have to wait anyway, as in a kitchen or a warehouse. AI models that have knowledge of logistics, planning and computers at the same time could soon find such tricks themselves in the software they run on. Then AI helps itself become faster, and because vLLM is open software, everyone benefits.
What is the technological singularity? Read the explanation →
