How to Set Up Voice Mode in Hermes Agent (STT and TTS Guide)

Hermes Agent can listen to spoken input and talk back, both in the CLI and through messaging gateways like Telegram or Discord. Voice mode ties together two separate systems: Speech-to-Text (STT) for transcribing what you say, and Text-to-Speech (TTS) for generating spoken replies. Setting both up correctly means understanding which provider to pick, how the settings interact, and a few defaults worth changing before daily use.

If Hermes isn’t installed yet, see the Hermes Agent Windows Install guide first.

Enabling voice mode in the CLI

Voice mode inside the CLI is controlled through the voice: block in config.yaml, and it’s separate from the STT and TTS provider settings covered further down.

voice:
  record_key: "ctrl+b"
  max_recording_seconds: 120
  auto_tts: false
  beep_enabled: true
  beep_volume: 0.3
  silence_threshold: 200
  silence_duration: 3.0

record_key is the push-to-talk key used to start and stop recording inside the CLI; it defaults to ctrl+b. max_recording_seconds caps how long a single recording can run, and silence_duration controls how many seconds of silence trigger an automatic stop instead of waiting for the key to be pressed again. silence_threshold sets the RMS volume level below which audio counts as silence for that auto-stop detection — raise it on a noisy microphone that never quite reads as silent, or lower it if recordings are cutting off too early during natural pauses in speech.

Once configured, type /voice on inside the CLI to turn on microphone mode. Spoken replies aren’t on by default — toggle them separately with /voice tts, or set auto_tts: true in the config so every reply is spoken automatically without needing to toggle it each session.

beep_enabled and beep_volume control the short audio cue that plays when a recording starts and stops. On quiet systems or with sensitive microphones, raising beep_volume (up to 1.0) makes it easier to tell when Hermes is actually listening.

Setting up Speech-to-Text (STT)

STT handles transcribing voice input, whether that’s a push-to-talk recording in the CLI or an inbound voice message on a messaging gateway. The cloud_trim_* settings below only apply to cloud providers and are explained further down; the setting to focus on first is provider.

stt:
  enabled: true
  echo_transcripts: true
  provider: "local"
  language: "en"
  cloud_trim_silence: true
  cloud_trim_threshold_db: -40
  cloud_trim_keep_ms: 300

provider accepts local, groq, openai, mistral, xai, elevenlabs, deepinfra, and a few others. The right choice depends on whether transcription should run entirely on the machine or through a cloud API.

Local transcription with faster-whisper

Setting stt.provider: "local" runs transcription through faster-whisper on the machine itself, with no audio ever leaving the device. It needs to be installed separately:

pip install faster-whisper

The local provider has its own model size setting under stt.local.model, with options from tiny up to large-v3 — smaller models transcribe faster but less accurately, and larger models are slower but more reliable, especially on accented or noisy audio.

Local transcription includes silence-hallucination hardening by default: a Silero VAD (voice activity detection) filter keeps silence and background noise from ever reaching Whisper, and any segment the model itself flags as low-confidence and probably-not-speech gets dropped automatically. Set stt.local.vad: false if transcribing non-speech audio like music or ambient sound, where the default filtering would strip out what’s actually needed.

The model stays loaded in memory between voice messages for faster response times. On a machine also running a local LLM on the same GPU, set stt.local.unload_after_idle_seconds (for example, 300 for five minutes) so the STT model releases GPU memory during idle periods — the next voice message reloads it automatically.

The trade-off for keeping everything local is speed and accuracy, which cloud providers usually handle better.

Cloud transcription providers

Cloud providers (groq, openai, mistral, xai, elevenlabs, deepinfra) trade local privacy for typically faster and more accurate transcription, especially on longer or lower-quality audio.

When ffmpeg is installed, cloud providers get a pre-upload silence trim by default: long pauses in a voice note are collapsed before the file uploads, using cloud_trim_keep_ms to preserve a bit of each pause so pacing still sounds natural. This means faster uploads, lower per-minute billing on providers that charge by audio length, and fewer silence-related transcription errors. Clips shorter than 12 seconds skip the trim entirely, since there’s rarely enough silence in a short clip to make it worthwhile. If ffmpeg is missing, the trim would save less than about 10%, or the trim itself fails, Hermes uploads the original file untouched rather than risk losing audio. Set stt.cloud_trim_silence: false to always upload the original file, which matters when transcribing music or ambient audio where trimming pauses isn’t appropriate.

Getting the transcription language right

Whisper-based auto-detection frequently misidentifies short or accented clips, which shows up as a voice note transcribed in the wrong language entirely. The default is stt.language: "en". Non-English speakers should set this to their language code once — for example "es" for Spanish, "zh" for Chinese, or "uk" for Ukrainian — rather than relying on auto-detection. Setting it to an empty string ("") restores auto-detection for genuinely multilingual use.

Language resolution follows a fallback order that’s the same across every STT provider: a per-provider language setting (stt.<provider>.language) takes priority, then the global stt.language, then the HERMES_LOCAL_STT_LANGUAGE environment variable, and finally the provider’s own auto-detection as a last resort.

Keeping transcripts private

By default, Hermes posts the raw transcript of a voice message back to the chat alongside a 🎙️ icon, through echo_transcripts: true. Set stt.echo_transcripts: false when the gateway should transcribe voice notes for the agent to act on, but shouldn’t post the transcript itself back into the conversation — a common requirement for customer-facing bots on WhatsApp or similar platforms.

Setting up Text-to-Speech (TTS)

TTS controls both the text_to_speech tool and spoken replies in voice mode, whether triggered from /voice tts in the CLI or through a messaging gateway.

tts:
  provider: "edge"
  speed: 1.0
  edge:
    voice: "en-US-AriaNeural"
    speed: 1.0

provider supports edge, elevenlabs, openai, minimax, mistral, gemini, xai, neutts, kittentts, piper, and deepinfra.

Choosing a TTS provider

edge is the default and requires no API key — it has access to 322 voices across 74 languages, which makes it the easiest starting point for most setups. elevenlabs and openai are common choices when a more natural or specific voice is needed, both requiring their own API keys. ElevenLabs is configured with a voice_id and model_id rather than a named voice — voice IDs come from the ElevenLabs dashboard, where custom and cloned voices can also be picked up the same way.

OpenAI’s TTS uses gpt-4o-mini-tts with a small fixed set of voices (alloy, echo, fable, onyx, nova, shimmer). Gemini offers 30 prebuilt voices under gemini-2.5-flash-preview-tts or gemini-3.1-flash-tts-preview, and also supports an optional persona_prompt_file for giving the model written voice direction beyond just picking a preset. Mistral (voxtral-mini-tts-2603) and xAI both take a voice_id, with xAI additionally exposing sample_rate and bit_rate for controlling output audio quality. Minimax needs only a speed setting, with voice selection handled on Minimax’s side.

For fully offline TTS, neutts runs locally through a GGUF model and accepts a reference audio clip and reference text (ref_audio, ref_text) for voice cloning, running on CPU by default. kittentts and piper are two additional local, no-API-key options for lighter-weight offline speech synthesis when neutts‘s voice cloning isn’t needed.

Controlling speech speed

Speed follows a fallback hierarchy: a provider-specific speed setting (for example tts.edge.speed) takes priority, then the global tts.speed, and finally a default of 1.0 if neither is set. Set the global tts.speed once to apply a uniform speed across every provider, or override it per-provider for finer control — useful when switching between providers that speak at noticeably different default paces.

Frequently Asked Questions