The State of Voice Agents
Every 20-30-year-old who grew up on Tony Stark has this dream: an assistant so good that it just seamlessly does everything we want. Friday, or Jarvis before her. We have been shown multiple attempts at this experience, but none of them come remotely close to what a voice "assistant" needs to deliver.
I just watched Garry Tan's case for personal AGI, and it makes so much sense. A voice assistant needs to evolve with the user, get to know their preferences, adapt to requests specific to them, and make changes on the fly. And this only happens when people start using voice assistants as their own personal assistants rather than just "Siri, can you set an alarm for 1:01."
So why hasn't this happened? I have been having more and more conversations with Claude, and that is way better than talking to Siri for sure, but it still breaks a lot of the time. I have used ChatGPT too, and it's definitely better at conversing than Claude. OpenAI is going after everyday life, and you can clearly see it in their voice mode. It has a personality instead of being bland. They clearly believe the way to capture consumers is a personality that's liked by everyone.
Helpful is not the same as helping
But personality isn't the whole problem. There is another issue: AI trying to be helpful rather than actually helping you. Right now there is very little consumers can do to improve their lives using AI. Sure, they can get information and answers, and that helps in a great way. But the tiny inconveniences still exist. I can say something, and current models do a good job of getting me to a certain point, but not all the way, and that is frustrating. Because to get there all the way, I need to look at everything the agent has done, verify it, check where it went wrong or stopped, get the complete context, and then continue from there. Why not do the whole thing myself, right?
They are incredibly good, don't get me wrong. But for these agents to be available to the masses, we need to solve two things:
- Ease of use: take all the complexity of the agent (context management, memory, flows, authentication, everything) and mask it.
- Cost of verification: reduce it to as close to nothing as possible.
OpenAI and LLMs replaced Google search because they give the user the exact information they need, rather than the user doing the work of first searching and then visiting sites to see which one has the best and exact information they were looking for. And it makes a lot of sense. LLMs solved convenience for consumers.
What already works
Wispr Flow is the best example I have of getting both right. I use it a lot every day, and all it does is speech to text. It's easy to use, and it's accurate enough that I rarely check what it wrote. Even if I wanted to, it's a simple glance. That's verification at almost zero cost.
Apart from that, I can't name any other voice products I've seen people use on a daily basis or that have made a huge impact on people's lives. Voice OS is another standout. I have tried it but haven't been a daily user, and it's still early for them. For what it's worth, it's incredibly good and does a lot of things. But this begs another question for me: Wispr Flow does one thing correctly, while Voice OS is trying to replace your entire desktop. Is doing one thing the reason I trust it?
Everyone needs something different
Which brings me to the question I can't stop thinking about. It's very, very personal. Some people want an assistant while building. Some just need it to listen to their commands and execute. Some need a partner. Some need more of a personality. Some want to form a connection with their voice agent. At the end of the day, everyone needs something different, and a generalized voice agent can never be something everyone wants. So what is the solution?
I think it's this: start narrow, where verification is nearly free, and let the assistant learn the person from there. Earn trust on small things, then widen. That's how you get from "set an alarm for 1:01" to something like Friday. Not one giant leap, but an assistant that gets a little more personal every day you use it.
We're not there yet. But it's definitely something to look forward to, and definitely something to work towards.
Thanks to Paul Graham, whose talk at YC made the point about LLMs replacing search far better than I can.