Audio

Audio Pipeline

flowchart LR
    Mic["Microphone"] -->|PyAudio| Wakeword["Wakeword"]
    Wakeword -->|wake| WakeListener["Wake Listener"]
    WakeListener --> Capture["Capture"]
    Capture --> WAV["input.wav"]
    Capture -->|captured| STTListener["STT Listener"]
    STTListener --> STT["STT"]
    STT --> Transcript["Transcript"]
    Transcript --> Pipeline["Conversation"]
    Pipeline --> Response["Response"]

Wakeword Detection

• WakewordManager runs in a background task, but it owns the microphone only while runtime_state.mode is idle.
• In idle mode it opens a PyAudio stream, reads chunks, and calls wakeword_engine.process_audio(chunk).
• In non-idle modes such as listening, transcribing, thinking, speaking, or cooldown, it closes the wakeword stream and does not read or drain microphone audio.
• On detection it emits wake_word_detected with AUDIO priority, resets the wakeword engine, closes the wakeword stream, and sleeps for detection_cooldown.
• This mode-based microphone ownership avoids a PyAudio backlog and avoids wakeword draining during STT capture.

Audio Capture

• Triggered by wakeword_listener.on_wake_word_detected().
• The wake/listen handshake still speaks sim? before capture.
• STT capture owns the microphone while the runtime is listening.
• Records microphone input to a temporary WAV file.
• Uses RMS silence detection: audioop.rms(data, 2) < silence_threshold.
• Stops on silence_timeout or max_record_seconds.
• Emits audio_captured with file path.

Speech-to-Text

• STTEngine is called by stt_listener.on_audio_captured().
• It opens the WAV file with wave.open().
• It creates a Vosk KaldiRecognizer and feeds 4000-byte frames.
• It returns FinalResult()["text"].strip().
• It emits transcript_ready with normalized text.

Audio Configuration

audio:
  sample_rate: 16000
  channels: 1
  chunk_size: 2048
  silence_threshold: 500
  silence_timeout: 1.5
  max_record_seconds: 30

Wakeword Optimization

The wakeword detector is the most CPU-intensive component during idle periods. Cosmo uses energy-based gating, idle sleep, optional silence grace chunks, and exclusive microphone ownership by runtime mode.

wakeword:
  energy_threshold: 350
  idle_sleep: 0.03
  silence_grace_chunks: 6
  detection_cooldown: 2.0
  require_final_result: false

• Only process Vosk AcceptWaveform() when RMS is greater than energy_threshold.
• Skip Vosk processing during silent periods to reduce CPU.
• Longer idle_sleep lowers idle CPU but increases wakeword latency.
• Larger chunk_size lowers loop frequency but increases wakeword latency.
• The previous approach could leave the wakeword microphone stream open while it was not being read, which allowed buffered audio to accumulate after the first interaction.
• Draining or clearing the microphone buffer between sim? and capture can cut the beginning of the user's phrase, so the current runtime relies on exclusive microphone ownership by mode instead.

TTS

TTS is provider-based "plug and play", with Piper synthesis and experimental Espeak support as available implementations.

tts:
  engine: piper
  language: pt
  locale: pt_BR
  model: faber/medium/pt_BR-faber-medium
  voice: pt-br
  speed: 145
  pitch: 35
  volume: 120