How It Works
Ears, a brain, and a voice
Every voice assistant, from Siri to the real Jarvis, is three parts stitched together. You're going to pick the best tool for each one and let Claude Code do the stitching.
Ears: Fish Audio speech-to-text
You hold the spacebar and talk. Your words get turned into text.
Brain: Claude
Claude reads what you said, plus a personality file you control, and writes a short spoken answer.
Voice: Fish Audio S2.1 Pro
Fish Audio's best voice model reads the answer out loud in whatever voice you picked, with real emotion (it can actually chuckle or whisper).
Before You Start
What you need
A free Fish Audio account
Sign up here. The S2.1 Pro voice API has a free tier right now, so the voice side costs you nothing to build and test.
Claude Code
Comes with any paid Claude plan. It's the thing that writes and runs the code for you. You just answer its questions.
An Anthropic API key (Jarvis's brain)
Get one at console.anthropic.com. Short spoken answers cost pennies, so a normal day of chatting with Jarvis is cheaper than a coffee. Claude Code will walk you through pasting it in.
Step 1
Pick your Jarvis's voice
Go to Fish Audio and click Create Voice. You've got three ways to go:
Sound like you
Click Instant Voice Clone, then record yourself or upload a video you already have. 15 seconds works, 30 to 60 is better. Quiet room, one speaker, talk normally.
Sound like Jarvis
Search the voice library for "British butler" or "AI assistant" and save the calm, posh one you like. Pair it with the formal "sir" personality in the prompt and it gets very Tony Stark, very fast.
Sound like anyone else
Browse the 2M+ voices in the library. Narrators, characters, accents. Save whichever one you'd actually want talking to you every morning.
Grab the Voice ID before you leave
Go to My Voices, open your voice, and copy its ID (a long string of letters and numbers). The prompt needs it.
Step 2
Grab your Fish Audio API key
From the Fish Audio home screen, click Developer, then API Keys, then Create API Key. Copy it.
Treat it like a password. Don't post it, don't screenshot it, and if you're filming your screen, blur it. Anyone with that key can spend your credits.
Step 3
Let Claude build it
Make a new empty folder on your computer called jarvis, open Claude Code inside it, and paste in Prompt 1 with your key and Voice ID filled in.
Claude interviews you first (name, personality, what it should help with, what the dashboard looks like), then builds the whole thing and tells you exactly how to start it. When you hear "Good evening, [name] is online," you're done. Hold spacebar and talk.
Once it's running, Prompts 2 and 3 are upgrades. Paste them into the same Claude Code session, one at a time.
Prompt 1: The Jarvis Dashboard
I want you to build me my own Jarvis: a voice AI assistant I talk to out loud, inside a dashboard that runs on my computer.
What I have:
- Fish Audio API key: [PASTE YOUR KEY] (from fish.audio > Developer > API Keys)
- Fish Audio Voice ID: [PASTE YOUR VOICE ID] (from My Voices > your voice. Looks like 933563129e564b19a115bedd57b7406a)
Before you write any code, interview me. Ask one question at a time and wait for my answer:
1. What should my assistant be called, and how should it talk to me? (Formal like Jarvis calling me "sir," a casual friend, a hype man, something else?)
2. What do I want help with day to day?
3. What should it know about me? (My work, projects, goals, anything useful.)
4. What should the dashboard look like? Give me 3 options: an Iron Man HUD, clean and minimal, or a retro terminal.
Then build it:
- Make it a small Node.js app in this folder: a local server (server.js) and one page (public/index.html). I start it with "npm start" and open http://localhost:3000.
- Voice (how it talks): Fish Audio text-to-speech. POST https://api.fish.audio/v1/tts with the headers "Authorization: Bearer <my key>" and "model: s2.1-pro-free", and a body of { text, reference_id: <my Voice ID>, format: "mp3", latency: "balanced" }.
- Ears (how it hears me): record my mic in the browser while I hold the talk button, send the audio to the server, and transcribe it with Fish Audio speech-to-text (POST https://api.fish.audio/v1/asr). If that gives you trouble, check docs.fish.audio, and fall back to the browser's built-in speech recognition so it still works.
- Brain (how it thinks): Claude through the Anthropic API, model claude-sonnet-5. Ask me for my Anthropic API key and tell me exactly where to get it (console.anthropic.com > API Keys).
- Personality: save everything I told you in the interview to personality.md and build Jarvis's instructions from it every time the server starts, so I can edit it later. Replies are 1 to 3 sentences, written to be spoken out loud, never bullet points or markdown. It can use Fish Audio emotion tags like [chuckle] or [whisper] once in a while.
- Dashboard: a big talk button plus hold-spacebar-to-talk, a live transcript of our conversation, a status line (Standby, Listening, Thinking, Speaking), and a circular audio visualizer that moves with Jarvis's actual voice (use the Web Audio API analyser on the audio that's playing).
- Memory: remember the whole conversation while the dashboard is open.
Rules:
- Put FISH_API_KEY, FISH_VOICE_ID, and ANTHROPIC_API_KEY in a .env file, and add .env to .gitignore. Never print my keys back to me and never put them in the browser code. Every API call goes through the server.
- I'm not a coder. Keep it simple and walk me through installing and starting it step by step.
- When it's done, test it: on startup, have Jarvis say "Good evening. [Name] is online." out loud. Prompt 2: Give Jarvis a memory
Upgrade my Jarvis so it actually knows my stuff. 1. Ask me which folder on my computer has the notes, docs, or files I want Jarvis to know (an Obsidian vault, a Documents folder, a folder of PDFs). 2. When I ask Jarvis a question, have the server search that folder first (file names and contents) and give Claude the most relevant pieces along with my question. If an answer comes from a note, have Jarvis say which one. 3. Add a "remember this" command. When I say "Jarvis, remember that..." it saves the fact with today's date to memory.md in this project, and memory.md gets included in its instructions from then on. 4. Only ever read the folder I pick. Never edit, move, or delete my files. 5. Test it: ask me for one thing Jarvis should know, then have me ask Jarvis about it out loud.
Prompt 3: The morning briefing
Add a morning briefing to my Jarvis. 1. Add a "Brief me" button to the dashboard, and make it run when I say "Jarvis, brief me." 2. The briefing covers, in under 60 seconds out loud: - A greeting with my name and today's date - Today's weather for my city (ask me my city, then use Open-Meteo, which is free and needs no signup) - My top 3 priorities, read from priorities.md, a file I can edit anytime - One short line of motivation in my assistant's personality 3. Create priorities.md with an example filled in so I know the format. 4. Test it by running the briefing out loud once.
If Something Breaks
The 4 things that usually go wrong
It can't hear you
Your browser needs mic permission. Click the icon left of the address bar and allow the microphone, then refresh.
It answers in text but stays silent
Almost always a wrong Voice ID. Copy it again from My Voices and tell Claude Code "fix my Voice ID."
It talks in markdown ("asterisk asterisk...")
Tell Claude Code "Jarvis is reading formatting out loud, strip all markdown before sending text to Fish Audio."
Anything else
Copy the error, paste it into Claude Code, and say "fix this." That's the whole move. You never have to understand the code.
The Honest Part
Two rules before you go wild
Only clone voices you have permission to use
Your own voice, or someone who said yes. That's why this guide says Jarvis-style from the library, not a clone of a real actor.
Free is for personal use
Building a Jarvis for yourself is exactly what the free tier is for. If you turn this into something for clients or a business, upgrade to a paid Fish Audio plan for commercial rights.
The Whole Thing in 4 Steps
Quick recap
Pick a voice in Fish Audio. Clone yourself, go Jarvis-style, or grab one from the library. Copy the Voice ID.
Create a Fish Audio API key. Developer, API Keys, Create. Keep it private.
Paste Prompt 1 into Claude Code. Answer the interview, follow the start steps.
Upgrade it. Prompt 2 for memory, Prompt 3 for the morning briefing.
Work with Me
Want me on your problem for an hour?
Book a 1:1 call and we build the fix live — the workflow you want automated, the tool you can’t crack, whatever’s stuck. Direct, hands-on, no pitch waiting at the end.