Block builds a speech model that learned to talk just by listening to conversations

Anyone who talks to a voice assistant notices it straight away: you say something, there is a silence, and then the answer comes. Interrupt the assistant and it often loses track. Block, the American payments company of Twitter founder Jack Dorsey, wants to do this differently. On 23 October it showed a new AI model called Dessa, which learned to talk the way a child does: by listening.

Learning without a single written word

Most AI assistants are built around a language model that learned from huge amounts of text. Speech is first turned into text, or the model thinks in text and then turns its answer back into a voice. Along the way, much of what makes a conversation human is lost: the tone, a hesitation, a laugh, the moment someone chooses to speak.

Dessa skips that intermediate step. Only speech goes in and only speech comes out. According to Block, the model was trained from scratch on recordings of conversations between people, without text and without an existing language model as a starting point. The company says Dessa learned the language from hundreds of times fewer words than an ordinary language model needs. Block does not give an exact number.

Listening and talking at the same time

A second difference lies in how the conversation flows. Most voice assistants work like a walkie-talkie: first one side talks, then the other. Dessa works more like an ordinary phone call, in which both sides can talk and listen at the same time. This means the model can say "yeah" or "mm-hm" in between, and it can respond when someone interrupts it. This is called full duplex.

Block shared recordings of conversations with Dessa. They sound strikingly human, with the occasional awkward or odd moment, as between two people who have only just met. Ragavan Thurairatnam, who leads the project, called the launch on X a breakthrough for speech that he compares with the arrival of GPT-3 for text. Thurairatnam co-founded an earlier Canadian AI start-up that was also called Dessa. His team at Block reports directly to Dorsey.

Small enough for a laptop

Because the model does not also have to hold all the knowledge that comes from text, it can stay a lot smaller. According to Block, Dessa runs faster than normal speaking speed on a MacBook Pro with an M4 Max chip or better, so the conversation does not have to go through a tech company's servers. For the training, Block used a single Nvidia server rack with 72 graphics chips. The company does not say how long the training took or what it cost.

Speech AI that runs on your own computer has been in the news more often in recent days. On 28 October, Nvidia released a free speech model that transcribes and translates recordings in 40 languages, also without the cloud. That model does the opposite of Dessa: it listens to a recording and turns it into text, but does not talk back, and it needs a powerful graphics card. Dessa, by contrast, is made to hold a live conversation, on a laptop.

Safety researchers first

Dessa is a research version. There are no public test results yet, no technical report and no version to download. The recordings Block released were chosen by the company itself. Independent tests will have to show how well Dessa does in ordinary conversations, and whether it can reason or use other programs.

Block is first giving the model to a small group of researchers in the field of AI safety. There is a good reason for that: a voice that sounds this human also makes it easier to deceive people over the phone. Anyone who wants to try Dessa can join a waiting list at Block.

The French company Mistral also recently released new speech models. They still use the classic intermediate step: first speech to text, then text back to speech. Dessa tries to let the whole conversation happen in one model, without that conversion.

What does this mean for the road to the singularity?

Talking is the most natural way for people to deal with each other and with machines. Dessa shows that an AI model can learn to talk from speech alone, with far fewer words than a language model needs, and that such a model fits on a laptop. That makes AI cheaper and more independent: a conversation partner that does not have to run in the cloud and that you can simply interrupt. The next step is for such a model not only to sound natural, but also to think and get things done for you. If that works, a conversation with a computer will become as ordinary as a phone call with a colleague.

What is the technological singularity? Read the explanation →