You won’t build this in three minutes. But you can have something functional, talking back to you, in an afternoon. That’s worth knowing.
The gap between “AI exists” and “I have AI working the way I want it to” is narrower than it was six months ago. Not trivial, but narrow. If you’ve been waiting for the tools to mature, or for someone to make it stupidly simple, this is close enough.
Why bother building your own
The off-the-shelf assistants do a lot of things. They don’t do your things the way you’d actually want them done. A purpose-built voice assistant—one that knows your priorities, handles your workflows, speaks in your register- is something you can neither buy nor (yet) rent. You have to assemble it.
It’s also not vapourware anymore. The components are stable. Integration takes real work, but it has a clear endpoint.
What you’re actually building
A voice interface to Claude that:
- Hears you speak and converts that to text (speech-to-text)
- Sends your words to Claude for processing
- Gets a response back
- Reads it to you in a consistent voice (text-to-speech)
- Runs locally on your machine or in the cloud
It’s nothing like the monolithic AI characters in sci-fi films. It’s four services glued together with some code. Each one does one thing well, and together they create something that feels like talking to an actual person who happens to be very organised and never forgets.
The toolkit
Claude Code (free with a Claude subscription, or as a paid CLI tool)
This is your orchestrator. It’s where you’ll paste a prompt, answer questions about what your assistant should actually do, and watch it generate the code that stitches everything together. You need the desktop version or the paid CLI. If you’re serious about this, spend the €20/month. It’s not the most expensive part of what you’re about to do.
ElevenLabs (€5–€29/month depending on usage)
This gives your assistant a voice. Browse their voice library, pick one that doesn’t make you want to abandon the project, copy its voice ID, and feed it into your code. The cheaper tier gets you 10,000 characters per month. That’s enough for testing and light use. If you want something that talks constantly, move to the €29 tier.
DeepGram (€12/month, or pay-as-you-go starting around $0.0043 per minute)
This converts your speech into text. You need an API key. It’s genuinely good at understanding spoken language, doesn’t hallucinate, and won’t misinterpret accents or background noise worse than any alternative.
Claude API (pay-as-you-go; roughly €0.30 per 1 million input tokens, €1.20 per 1 million output tokens)
Your assistant’s actual brain. Claude via API is cheaper than Claude via the website, but you need to think about token usage—especially if your assistant runs continuously. Budget €10–30/month for moderate use; more if it runs eight hours a day.
A local folder on your machine where the code lives and where your assistant runs.
Total monthly: roughly €40–60 if you start lean, less if you’re careful.
The actual process
Step one: Set up Claude Code
Get the desktop version. Open it. Create a new folder for all your files.
Step two: Get the foundation prompt
Frameworks out there handle the scaffolding for you. Instead of building from scratch, use this solid prompt that describes an AI agent’s structure; copy it, paste it into Claude Code, and hit “plan mode.”
Claude Code will ask you a series of questions. What should your assistant be named? What’s its primary goal? What matters to it? (This is where you shape its personality, not afterwards.) What time should it work during? What tools does it need access to? You answer these questions and the system generates a personality profile for your assistant that actually shapes how it behaves.
Step three: Get your voice
Go to ElevenLabs, browse voices (it takes longer to pick a voice than it should; don’t overthink it), copy the voice ID, and paste it into Claude Code when prompted.
Step four: Get your speech recognition
Create a DeepGram account, generate an API key, and copy it into Claude Code.
Step five: Let Claude Code build
This takes a few minutes. It generates Python code that ties these services together, handles authentication, manages the conversation loop, and makes sure the audio in/out works.
Step six: Test it locally
Once it’s done, Claude Code will ask you to run a localhost. It spins up a web server on your machine. Open it in a browser. You should see a simple interface. Hit the microphone button, speak, and listen for a response. If it responds, you’ve built it.
The invisible work
What makes this actually usable is the prompt engineering. The initial questions Claude Code asks you—about your assistant’s goals, personality, constraints—get compiled into a detailed system prompt that Claude reads on every exchange. That prompt is doing the heavy lifting. If you describe your assistant as “a person who prioritises clarity and brevity,” it will. If you say “always explain your reasoning,” it will.
Voice and personality are linked. A voice like Morgan Freeman’s works if your assistant is thoughtful and measured. A bright, energetic voice on something designed to be dry and critical is jarring. Spend actual time on this choice; you’ll hear it constantly.
What it actually feels like
You’re not talking to a chatbot. The voice, the response time (usually under two seconds), and the fact that it’s personalised to your actual goals create something that feels different. Your brain accepts it as a tool you’re collaborating with, not something you’re using.
It’s also still genuinely limited. It won’t manage your calendar unless you write code to plug in your calendar API. It won’t take actions in the real world unless you explicitly wire it to. What it will do is listen, think, and respond in a way that’s coherent and genuinely helpful. For some workflows, that’s enough. For others, it’s a starting point.
The hard part
The hard part isn’t the setup. It’s deciding what you actually want it to do. Because once you have a voice interface to Claude, you have to be clear about your instructions, your constraints, and what matters. Most people avoid this clarity even when they’re just writing prompts in ChatGPT. With something you’re building and running yourself, you can’t skip it.
That clarity—working out exactly what you need, how you’d phrase it to a person, what you’d want them to optimise for—is worth the afternoon it takes to set this up.
The voice part is almost a side effect.
