Voice

Someone sends your agent a voice note and the agent can only apologize: the trigger hands the model a line saying an audio file arrived that it cannot listen to (Attachments covers that honesty). The voice block turns the apology into a conversation.

There are two ways to have that conversation, and they share one config block. Voice notes are turn-based: a clip arrives, the transcript runs through the normal loop, and with tts the reply comes back as a clip. Live voice is a real conversation: the agent sits in a Discord voice channel and you talk to it out loud.

voice:
  stt:                            # voice notes in (telegram, discord)
    provider: openai              # or elevenlabs
    # model: gpt-4o-mini-transcribe (default)
    # api_key_env: OPENAI_API_KEY (default)
  tts:                            # omit for text replies to voice notes
    provider: elevenlabs
    # model: eleven_multilingual_v2 (default)
    # voice: 21m00Tcm4TlvDq8ikWAM (default: Rachel)
  live:                           # live conversation in a discord voice channel
    provider: openai
    # model: gpt-realtime-2.1 (default)
    # voice: marin (default)
    # idle_seconds: 60 (default)

Each part is optional, and you can run any combination. The block sits at the top level because the engines are shared: every voice-capable trigger in the file speaks and listens through the same ones. Mixing providers is fine - transcribe with OpenAI and speak with ElevenLabs, or the other way around.

Voice notes

The trigger downloads the audio, sends it to the transcription API and emits the transcript as the event's input. From there it is an ordinary run: same purpose, same tools, same memory, same audit trail. When the run finishes and tts is configured, the trigger asks the speech API for Ogg Opus and posts it as a real voice note, the kind that renders with a play button and a waveform.

Some replies stay text on purpose. A reply longer than 4000 characters reads better than it listens, so it posts as text. A reply routed elsewhere via reply_chat or reply_channel stays text too, because it quotes the message it answers. And when the speech API errors, the trigger logs the failure and posts the text instead, so the reply still arrives. allow_silence applies before any of this: a __NO_REPLY__ answer posts nothing, voice or text.

The default follows the message: a voice note comes back as a voice note, a typed message comes back as text. The agent can override that for a single reply by leading with a marker. __VOICE__ has the reply spoken, even in answer to a typed message; __TEXT__ sends it as text, even in answer to a voice note. This is what "reply with a voice note" or "just text me back" turns into once the agent decides to honor it. The trigger strips the marker before the reply goes out, the way __NO_REPLY__ works, so you tell the agent about the markers in its purpose. Forcing voice runs through the same gates as any other spoken reply: it lands as text when tts is off, when the reply is too long to speak, or when it's routed to another chat.

One gate to know about: require_mention. A voice note has no text to carry an @-mention, so in a mention-gated group voice notes drop before the model is called. Private chats and DMs always address the bot, and that's where voice conversations naturally live.

Live voice in a Discord voice channel

Add voice.live and point a discord trigger at a voice channel, and the agent joins it at startup and stays:

voice:
  live:
    provider: openai

triggers:
  - type: discord
    voice_channels: ["standup"]   # names or ids

Talk to it and it talks back, with the interruptions and half-second pauses of an actual conversation. You can cut it off mid-sentence and it stops.

The way that works is worth understanding, because it shapes what the agent can do. A realtime speech-to-speech model holds the conversation: it hears you, decides when you have finished a thought, and speaks. It is fast and it is good at talking, and it knows nothing about your systems. So it has exactly one tool, ask_agent, and when you ask for something real it hands the request to your agent - the same loop, the same tools, the same permissions, the same audit trail - and speaks the answer when the run comes back. The voice model is the mouth and ears. Your agent is still the one doing the work, and it is still the only thing that can touch anything.

That split is also the safety story. Nothing the voice model says to itself can call a tool. Every consequential action goes through a normal run, which means the permission engine sees it and the audit trail records it, exactly as it would for a message typed into a text channel. Spoken conversations get their own conversation key (discord-voice:<guild>), so what you say out loud stays out of your text channels' history.

A live session bills by the audio minute, including the minutes nobody is talking. That is what idle_seconds is for: the session closes after a minute of silence and reopens on the next word, so a bot sitting in an empty channel costs nothing. Set it higher if the reopening pause bothers you, and know what you are paying for.

Setup is two things beyond the Discord setup you already did. Re-run af discord-invite agent.yaml and open the URL - the invite now asks for the permissions to connect to a voice channel and speak in it. Then give the agent an OPENAI_API_KEY, even if its model: block runs on something else entirely; the realtime session is a separate connection with a separate key.

Keys and the sandbox

Every API key resolves at startup from the env var the config names, and a missing key stops the agent right there with the var's name in the error. Under hermetic mode the engines' hosts join the derived allowlist on their own, so there is nothing to declare in permissions.net.

Live voice is the exception, and it is a real one. Voice media travels over UDP to a media server Discord picks per session, and Deno's sandbox cannot hold a permission for an address it does not know in advance. An agent with voice_channels therefore runs outside hermetic mode, with the container as its egress boundary. af validate says so plainly. If hermetic mode is load-bearing for you, keep live voice on an agent that does not need it.

What this doesn't cover

Slack is absent from this whole page, because Slack gives bots no voice surface at all - no voice messages to read or post, no huddle audio to join. Telegram stops at voice notes; live calls there need a user account, which a bot is not.

Two smaller things we decided against. The waveform Discord draws on the agent's voice replies is a placeholder shape, since drawing the real envelope would mean decoding the Opus we just encoded. And live voice does not yet know who is speaking: it hears the channel as one voice, so from_users does not apply to it.

There is one risk worth naming. Discord is moving voice to end-to-end encryption (DAVE), and this bridge negotiates the older transport-encrypted path. Voice servers still accept it. When they stop, live voice needs the new protocol, and that is a change we will have to make rather than a knob you can turn.