Hey Jarvis — How It Works

Architecture · Voice Pipeline

How Hey Jarvis thinks

From wake word to spoken response — on your hardware

⚠

This is a visualization of the system built in the book. The architecture shown here is the foundation — your implementation's look, layout, and extensions are yours to design. The HUD overlay, color scheme, and visual style shown are one interpretation. The book gets you to a working system; where you take it from there is entirely on you.

Jarvis V3 live · OVERWATCH dashboard · real system · no cloud

2026-08-12 00:42:25 INFO services.stt STTService started (model=base.en)

2026-08-12 00:42:23 INFO services.tts Kokoro TTS loaded (singleton)

2026-08-12 00:42:25 INFO services.ptt PTTListener started

2026-08-12 00:42:25 INFO __main__ Voice pipeline started

2026-08-12 00:42:29 INFO __main__ Jarvis V3 startup complete

tap any node to expand

Stage 1 · Local

Wake word detection

OpenWakeWord · custom model

Always-on listener running at near-zero CPU. Your custom wake word triggers recording — nothing is recorded until you say it. Fully offline, no cloud dependency at this layer.

Stage 2 · Local

Voice capture

PyAudio · silence detection

Records until silence is detected, then closes the audio buffer. Handles background noise, mic sensitivity, and AEC so voice barge-in works during playback.

Stage 3 · Local

Speech to text

faster-whisper · base.en

Runs Whisper locally on CPU. Your audio never leaves your machine at this stage. Transcription happens in under a second on modest hardware. The model file lives in your venv — no internet call.

Stage 4 · API call

Reasoning layer

Claude API · intent routing

The one outbound hop. Your transcribed text is sent to Claude, which interprets intent, routes to the right plugin, and returns a response. Behind a VPN this connection is encrypted in transit. Anthropic does not sell API data and does not use it to train models by default.

Stage 5 · Local

Text to speech

Kokoro TTS · Kokoro-82M

Runs the Kokoro model locally. No cloud TTS service — voice synthesis happens on your machine with no audio data leaving. Response latency is typically under 300ms on modern hardware.

Stage 6 · Output

HUD overlay + speaker

Plasma overlay · Windows

Response is spoken through your audio output and displayed on the always-on HUD overlay without stealing window focus. Session context and memory live on your disk — not in any cloud.

Stays on your machine

Voice recordings

Audio transcripts

Conversation memory

Plugin data and logs

Screen captures

Trained wake word model

One outbound hop

Transcribed text to Claude API

Response text from Claude API

Anthropic does not sell API data. No ad network is in this path. Behind a VPN, this connection is encrypted in transit.

Unsolicited reader review