NederlandsNew AI memory condenses a whole stack of documents into 16 tokens for each question
Researchers from Stanford and other institutions built Memorilla, a memory that condenses a whole collection of documents into 16 tokens for each question. In their own tests, a language model usually scores better with it than with all the text, and answers much faster. Exact names are sometimes lost.
A language model, the type of AI behind chatbots, processes text in tokens: small pieces, often a word or part of a word. If you want it to answer questions about a whole stack of documents, or about months of earlier conversations, all that text has to go into the question. The longer the text, the more the model has to calculate, and beyond a certain length it no longer fits.
Researchers from Stanford, the University of Toronto, the University of Pennsylvania and MIT presented a different approach in early November. Their system, called Memorilla, condenses the whole collection into 16 tokens for each question. In their own tests, a language model scored better with it than with all the text on seven of the eight tests. With 512 documents, it answered 27 times as fast.
Reading everything or searching first
The usual solution is called RAG. A search system first picks the few pieces of text that seem to fit the question best and gives only those to the language model. This works well when the answer is written down word for word somewhere. It is harder when the answer is spread over pieces that seem to have little to do with the question at first sight. The search system judges each piece on its own, so such pieces are left behind.
How Memorilla works
Memorilla does not pick separate pieces. A language model first converts every piece of text internally into a row of numbers. Memorilla makes use of that idea. Each document is converted once into numbers that capture its meaning, and these are stored. When a document is added, only that document has to be converted.
For each question, a separate AI component, trained specially for this, reads through all the stored numbers. It puts what matters for this question into 16 memory tokens: not words, but rows of numbers that the model receives in the same place as ordinary text. Think of an assistant who goes through the whole archive for each question and writes a new, short note in a language that only the model can read. However large the collection is, there are always 16.
The language model itself does not change. It only receives those 16 memory tokens before the question.
Better and faster in the tests
The researchers tested Memorilla on eight tests, using the open language model Qwen3-8B. On average, the system scored higher than all the other methods, including the existing memory systems Mem0 and MemoRAG. On five tests it was the best of all.
In the FactKG test, in which a language model judges whether statements are true, it got 84.3 per cent right, against 80.4 per cent with all the text. The model had to process 108 times less text from the documents. In the TriviaQA quiz questions it was 57.0 against 52.6 per cent, with 743 times less text.
The speed also differs greatly, because the language model has far less to calculate. With 512 documents, about 128,000 tokens in total, an answer took 0.74 seconds on one Nvidia H100 chip, against 19.9 seconds with all the text. The documents are converted beforehand, once.
The team also tried it with an AI agent: a language model that takes steps by itself, here in adventures that are described only in words, with up to 65 rooms. The model already received its whole game history as ordinary text, and 8 memory tokens were added to that. It played each new game three times. On average it won 24.2 per cent of those attempts, against 18.6 per cent without memory tokens. It won at least once in 37.5 per cent of the games, against 25 per cent.
What still has to happen
The figures come from the researchers' own tests, and the researchers made one of the eight tests themselves. The code and the trained versions are public, so that others can repeat the tests.
A short note cannot contain everything. Memorilla sometimes loses precise details, such as exact names. This is why RAG, which passes on the text word for word, does better with "who" or "which" questions, and Memorilla with "why" and "how" questions. The researchers suggest combining the two.
The medical test PubMedQA involved very short texts, about 324 tokens in total. There, the language model with all the text scored higher: 67 against 59 per cent. On two other tests MemoRAG scored higher, but that system used 50 to 690 times as much text.
Memorilla has to be trained separately for each language model, because it works directly with the internal numbers of that model. This is only possible with open models that anyone can download. It has so far been tested with models from the Qwen family. The next step is to try how it works with other models.
What does this mean for the road to the singularity?
A language model does not remember anything from earlier conversations by itself. Anyone who wants AI systems that keep building up knowledge needs a memory that grows without the input getting longer all the time. Memorilla is a step in that direction: what the model has to read stays equally short, however many documents are added.
Follow-up research has to show whether this also holds for other models and much larger collections. If it does, AI assistants come closer that have their whole history at hand and can draw on it quickly.
What is the technological singularity? Read the explanation →
