Voice

People can talk to UGENT instead of typing, and hear it answer:

• Voice messages. A recording sent as a message. The daemon transcribes it before routing, so rules, the model and the conversation all read the same words. A model that hears audio also gets the recording itself.
• Dictation. A recording turned into text in the composer, which the person reads, edits and sends. No turn runs, and nothing is kept.
• Read aloud. A finished answer, spoken a few sentences at a time. The answer to a voice message can be read without a click.
• Live calls. A spoken conversation with a voice model that answers in real time and hands real work to the agent.

Each runs on an LLM instance you configure for it. Before a client offers any of them, the daemon reports whether it will work, and the web workspace shows a control only when the report says so. What people see is described there; this page is how you set it up.

Document Author: Uni Zhu

Quick Start

[llm.voice]
stt_instance = "openai-audio"   # transcribes voice messages and dictation
tts_instance = "gemini"         # reads answers aloud

[llm.instances.openai-audio]
type = "openai"
api_key = "$OPENAI_API_KEY"
default_model = "gpt-6-luna"

[llm.instances.openai-audio.media.stt]
enabled = true
model = "gpt-transcribe"

[llm.instances.gemini]
type = "google"
api_key = "$GOOGLE_API_KEY"
default_model = "gemini-3.8-flash"

[llm.instances.gemini.media.tts]
enabled = true
model = "gemini-3.8-flash-tts"
voice = "Kore"

That gives voice messages, dictation and read aloud. Calls need an OpenAI instance with [media.live] (see Live Calls). Then read the status report to confirm what works.

To offer	You need
Voice messages	An instance with [media.stt] enabled, or a default or selectable model that hears audio
Dictation	An instance with [media.stt] enabled
Read aloud	An instance with [media.tts] enabled
Live calls	A type = "openai" instance with [media.live] enabled

Voice is on by default, but with nothing configured to serve it the report says no_engine, and clients show no controls.

[llm.voice]

[llm.voice]
enabled = true
max_input_seconds = 300
memory_write = true
native_audio = "auto"                # "off": every model gets the transcript alone
autoplay_replies = true
# stt_instance = "openai-audio"
# tts_instance = "gemini"
# live_instances = ["openai-rt"]
# reply_voice = "Kore"               # unset: the speaking instance's own voice
# input_classification = "internal"  # read only when the tenant enforces a scheme

Key	Default	Meaning
enabled	true	false turns voice off: voice messages are answered with a refusal, and dictation, read aloud and live calls are reported unavailable. An audio file attached to a typed message is still transcribed.
stt_instance	unset	The instance that transcribes.
max_input_seconds	300	The longest voice message or dictation, in seconds. A longer one is refused before any provider hears it.
memory_write	true	false keeps every voice turn, and every turn a call hands over, out of memory. See Memory.
native_audio	"auto"	"auto": a model that hears audio gets the recording beside its transcript. "off": every model gets the transcript alone.
input_classification	unset	The level a recording is treated as. Read only when the tenant enforces a classification scheme; see The Classification Gate.
tts_instance	unset	The instance that reads answers aloud.
reply_voice	unset	The voice answers are read in, by the engine's own name (Kore, alloy). Unset: the speaking instance's [media.tts] voice.
autoplay_replies	true	Whether a client reads the answer to a voice message aloud by itself. false leaves it to the reader's click.
live_instances	[]	The instances a call may use, in the order they are tried.

Which instance does the work. Transcription, read aloud and calls each use the first instance that enables its block ([media.stt], [media.tts] or [media.live]), passes the classification gate, and has a working client, in this order:

1. the instance stt_instance or tts_instance names, or each of live_instances in turn;
2. the default instance;
3. every other instance, by name.

For an audio file attached in the TUI, the turn's own instance stands in for the default.

Naming an instance never turns an engine on: one whose block is off is passed over, and so is one whose provider cannot do the job. The work goes through that instance's own client, so a key held by reference or behind the secret broker is used exactly as it is for chat.

[llm.voice.live]

[llm.voice.live]
max_call_seconds = 900
idle_seconds = 60
delegate = "auto"     # "off": a conversation with the voice model alone
history_turns = 8     # 0: no brief

Key	Default	Meaning
max_call_seconds	900	The longest a call lasts, counted from when it is ready. Held to 30–3600: the Realtime API ends a session at an hour.
idle_seconds	60	How long a call may hear nothing from the page before it ends. Held to 10–600.
delegate	"auto"	"auto": the voice model hands requests to the agent. "off": it has no tools, and says it cannot look anything up.
history_turns	8	How many of the conversation's latest exchanges brief a call as it starts. 0 briefs it with none.

A value outside its range is held to the nearer end, so max_call_seconds = 0 gives the shortest call, not an unlimited one.

Per-Instance Media Blocks

Each engine is a block under an LLM instance: [llm.instances.<name>.media.stt], .media.tts and .media.live.

A misspelt key is ignored silently

Unknown keys in these blocks and in [llm.voice] are dropped without a warning. After a change, check the status report rather than assuming the change took.

[media.stt] — transcription

Key	Default	Meaning
enabled	false	Whether this instance transcribes.
model	per provider	OpenAI: gpt-transcribe. Gemini: the instance's default_model, which hears the recording and writes it down. On an OpenAI-compatible server, name the model it serves.
auto_transcribe_plugin_attachments	true	Whether it transcribes recordings that arrive with messages: voice messages, and audio attached on any channel. false: dictation only.
auto_transcribe_local_files	true	Whether it transcribes audio files attached in the TUI.

An older [llm.instances.<name>.audio_transcription] block is read as [media.stt] when the instance has none.

[media.tts] — read aloud

Key	Default	Meaning
enabled	false	Whether this instance reads answers aloud.
model	per provider	Gemini: gemini-3.8-flash-tts. OpenAI: gpt-4o-mini-tts.
voice	per provider	Gemini: Kore. OpenAI: alloy. [llm.voice] reply_voice takes precedence.

Read aloud always asks the engine for WAV.

[media.live] — calls

Key	Default	Meaning
enabled	false	Whether this instance serves live calls.
model	gpt-realtime-2.1	The Realtime model.
voice	marin	The voice a call speaks in, unless the page asks for another.

The TUI's /live reads this block too, with differences described below.

Which Providers Do What

Provider (type)	Transcribes	Reads aloud	Serves calls
openai	Yes, through /audio/transcriptions (not Azure OpenAI)	Yes, through /audio/speech (not Azure OpenAI)	Yes, the Realtime API (not Azure OpenAI)
openai-compatible	Yes, through the server's own /audio/transcriptions (not Azure OpenAI)	Yes, through its /audio/speech; a local speech server works (not Azure OpenAI)	No
google	Yes: the chat model hears the recording	Yes, Gemini TTS	No
anthropic, ollama	No	No	No

• Calls are native OpenAI only: an instance of type = "openai" whose base_url, if set, is not an Azure endpoint. An instance of any other provider with [media.live] enabled is passed over.
• Azure OpenAI is passed over for transcription and reading aloud too, even when stt_instance or tts_instance names it: Azure serves its audio models through deployments of their own, which UGENT does not address. Startup warns for each such block that is on.
• OpenAI's older speech models are going away. The gpt-4o transcribe models and whisper-1 shut down on 2027-02-26, which is why gpt-transcribe is the default. OpenAI's text-to-speech models are retired on 2027-01-06, so for read aloud prefer Gemini on a deployment that will outlive them.
• Gemini speech comes back as WAV, or as bare 24 kHz samples that are given a WAV header. It speaks in one voice; multi-speaker speech is not used.

Models That Hear the Recording

With native_audio = "auto", a model gets the recording itself, beside its transcript, only on positive evidence. All of these must hold:

• The catalog says it is a chat model that takes audio. A speech model never counts: an id with a transcribe, asr, stt, tts, whisper, realtime or live token is excluded.
• The route carries an audio part. Gemini on a google instance always does. An OpenAI or OpenAI-compatible model does only on Chat Completions: under api_dialect = "auto", native OpenAI sends its audio chat models (gpt-audio-1.5 and other ids with an audio token) there and everything else to the Responses API, which takes no audio. Anthropic and Ollama never do.
• The policy lets audio through: [llm.multimodal.interactive] audio_enabled, on by default (see Multimodal Input).
• The gate, under an enforced classification scheme, clears the model's instance for recordings.

A local model behind LM Studio or another uncatalogued OpenAI-compatible server never qualifies, and gets transcripts. A recording also goes only in a format the route takes: OpenAI's input_audio, native or compatible, takes WAV and MP3; Gemini also takes AIFF, AAC, OGG and FLAC. Otherwise its transcript stands for it.

The default model and the selectable models that qualify are listed in status.media.voice.input.native_models.

OpenAI's gpt-audio models

They hear a voice message natively, with the agent's tools, but they refuse any request that carries no audio, so a typed message in a conversation on one fails. With the agent's long instructions their answers are also less reliable than a text model's. For agent work, prefer a text model with gpt-transcribe; offer a gpt-audio model for voice only.

The Classification Gate

When the context engine enforces a classification scheme for the daemon's tenant, a recording is treated as [llm.voice] input_classification. It reaches only an instance whose max_classification ranks at or above that level, in the order the context engine's own scheme lists its levels, never alphabetically. That applies to:

• the instance that transcribes it;
• the instance of a model that would hear it;
• the instance that reads an answer aloud, since an answer is built from what the conversation holds;
• the instance that serves a call, since its brief is the conversation's own text.

[llm.voice]
input_classification = "internal"

[llm.instances.openai-audio]
max_classification = "internal"       # at the level: may transcribe

[llm.instances.gemini]
max_classification = "confidential"   # above it: may read answers aloud

[llm.instances.budget]
max_classification = "public"         # below it: never sent a recording

An instance the gate refuses is passed over, and the next in order is tried. The gate governs where the recording goes: once transcribed, its words travel as the message's text, as typed words do.

It fails closed. Voice is reported unavailable, with reason: "classification" and a detail saying why, when:

• the context engine cannot say whether a scheme is enforced;
• input_classification is unset, or is not a level of the scheme;
• no instance that could do the job is cleared for the level. An instance with no max_classification, or one naming a level the scheme does not define, is never cleared.

A tenant that enforces no scheme, or a deployment with no context engine configured, is not gated at all. A context engine in single-tenant mode answers the same route from its own [access] settings, and reports a scheme it compiled as enforced whatever [access] enabled says, because it applies that scheme either way. Its SERVICE_TOKENS entry for the daemon must list config:read, which nothing else on such an engine reads: without it the engine answers 403, and voice stays off, reported as classification.

The daemon asks the engine GET /v1/tenant/access/classification with the key it uses for retrieval, which therefore needs the config:read scope. An answer is reused for five minutes. A failure to get one is reused for thirty seconds, so a short outage closes voice briefly rather than for minutes.

Routing and Rules

A voice message's text is the typed text, if any, followed by the transcript. That is what routing detectors and rules read. A rule can also match how a message was made, as the input_mode field: voice for a voice message, live for a request a call handed over, and absent on a typed message.

[rules.match]
all_of = [
  { field = "input_mode", equals = "voice" },
  { field = "text", contains_any = ["transfer", "payment"] },
]

The whole rule, which ignores spoken payment instructions, is in Matching Voice Messages. source_type keeps its meaning, the kind of conversation, and is never the input mode.

Memory

With memory_write = false, a voice turn writes nothing to memory, whatever its client asks: no facts extracted after it, no compaction summary, and no memory_control tool. It also loses cron_management, since a scheduled job would not carry the restriction. The same holds for every turn a call hands over, which carries no recording but was spoken. A job a rule starts from a voice message is held to the same, and the rule action extract_memory saves nothing while memory_write = false, since a conversation does not record which of its messages were spoken. Recall is unaffected.

status.media.voice.input.memory_write reports the setting. The web workspace asks for no memory write on a voice turn unless it reads true there.

What the Daemon Reports

GET /v1/ugent/status on the web channel, with the operator key, carries status.media.voice:

curl -s -H "Authorization: Bearer $OPERATOR_KEY" "$WEB_CHANNEL_URL/v1/ugent/status" \
  | jq .status.media.voice

For a deployment whose default model is gemini-3.8-flash:

{
  "input": {
    "available": true,
    "transcription": true,
    "dictation": true,
    "native_models": ["gemini-3.8-flash", "ugent-default"],
    "max_seconds": 300,
    "max_bytes": 9600044,
    "formats": ["audio/wav"],
    "memory_write": true
  },
  "output": { "available": true, "autoplay": true, "formats": ["audio/wav"] },
  "live": { "available": true, "provider": "openai", "model": "gpt-realtime-2.1", "max_seconds": 900 }
}

The web workspace relays the same report at /api/ugent/status to anyone with chat:read, with two changes: input.max_seconds is the smaller of the daemon's limit and the web workspace's own VOICE_MAX_SECONDS (300 by default; a recording also stops at 10 MiB, about five and a half minutes), so raise both for longer recordings, and detail is kept only for someone with workspace:manage.

A client offers a control only when the report says it will work:

Control	Offered when
Microphone, for model m	input.available, and input.transcription is true or input.native_models contains m
Dictation	input.dictation
Read aloud	output.available
A voice message's answer read unasked	output.autoplay as well
Call	live.available

ugent-default is the id a client sends for the deployment's default model. dictation can be true while transcription is false: the transcribing instance has auto_transcribe_plugin_attachments = false.

When available is false, reason says why, and detail, when present, names the instance or the setting:

reason	Meaning
disabled	[llm.voice] enabled = false.
classification	The gate admits no engine.
no_engine	Input: nothing transcribes, and no default or selectable model hears audio. Output: no instance reads aloud. Live: no native OpenAI instance enables [media.live].

The web workspace shows neither reason nor detail to the people using it; to them, a control is simply absent. Read the report.

When a Voice Message Is Refused

The sender gets one of these answers in place of the model's. None is a model call, and none is kept in the session. Which instance or level was involved goes to the daemon's log, not into the answer.

Situation	Answer
Voice turned off	Voice messages are turned off here. Please type your message instead.
Longer than max_input_seconds	This voice message is 6:12 long, and the longest accepted is 5:00. Please record a shorter one.
No speech, and nothing typed	No speech was heard in this voice message. Please try recording it again.
Not 16-bit mono WAV at 16 kHz	This voice message's recording is not one this deployment takes: it must be a WAV file of 16-bit mono audio at 16 kHz.
Larger than 25 MiB	This voice message is too large to send. Please record a shorter one.
The recording could not be read	This voice message's recording could not be read. Please try recording it again.
Nothing could use it	This voice message could not be understood: nothing here transcribes recordings, and the selected model cannot hear audio. Please type your message, or choose a model that hears audio.
The gate keeps it from every provider	This voice message was not sent to any model: no speech service is approved for this workspace's data classification. Please type your message instead.
The engine failed	This voice message could not be transcribed just now. Please try again, or type your message.

A turn that fails after it has started, for example because the provider refused the request, ends with a short error naming the turn. The provider's own words can quote what it was sent, so they go to the log under that turn id.

Only the web channel takes voice messages; the web workspace is its client. A voice note from another channel, or an audio file attached to a typed message, is transcribed the same way before routing when an engine transcribes attachments, and a platform's own recognition text is used as it is. None of the refusals above applies to it, though the classification gate still decides which instance may transcribe or hear it, and when nothing can use it the model is told it could not be heard.

What Is Kept

What	Where	For how long
A voice message's recording	The web channel's plugin media	One day by default (UGENT_PLUGIN_MEDIA_TTL_SECS, in seconds)
The web workspace's playback copy	VOICE_DIR (default ~/ugent-voice)	VOICE_RETENTION_DAYS (default 30; 0 keeps none)
What was said	The conversation's session: the transcript, and a note of each recording (file id, type, length, transcript). Never the audio.	As long as the session
The transcript, again	The message's receipt, so a page that closed early can still show it after a reload	30 days after the turn settles
A dictation	Nowhere: the recording is removed once it has been transcribed or refused	—
Spoken segments	Plugin media, in ugent-speech/	One day
A call	A note in the conversation's session, at most 4,000 characters with the latest lines kept, and the web workspace's own record in the thread	As long as the conversation

A later turn never sends an earlier recording again: an earlier voice message reads as its transcript, and an attachment whose file has expired reads as a note.

The web channel logs a call's id, its length, how it ended and counts, never the audio, the credential or what was said. Keep [llm.debug] log_raw_data off on voice instances: it would log what people said.

Live Calls

A call is a spoken conversation with a voice model, OpenAI's Realtime API today. The voice model is the agent's voice, not a second agent. It has one tool, ask_agent, and hands anything about the person's own work, files or data to the agent, which runs it as an ordinary turn of the conversation. It then gives a short spoken account of the answer, and is told never to invent a result.

[llm.voice]
live_instances = ["openai-rt"]

[llm.voice.live]
max_call_seconds = 900
idle_seconds = 60

[llm.instances.openai-rt]
type = "openai"
api_key = "$OPENAI_API_KEY"
default_model = "gpt-6-luna"

[llm.instances.openai-rt.media.live]
enabled = true
model = "gpt-realtime-2.1"
voice = "marin"

Who holds the key. The browser talks only to the web workspace, which relays to the web channel (GET /v1/ugent/live, a WebSocket). The web channel passes audio between the browser and the provider but never holds a provider key: for each call the daemon builds the session and mints a credential from the instance's own key, good for opening one connection within a minute. The web channel then connects to the provider's Realtime endpoint itself, at the instance's base_url (by default wss://api.openai.com/v1/realtime), so it needs network access there.

What the voice model knows. A call starts briefed with the conversation's latest history_turns exchanges, each message clipped to 600 characters and the whole to 6,000. The model is told to speak briefly, to say what lists, links and code amount to rather than read them out, and to reply in the language the person speaks. The person's side is transcribed with gpt-transcribe, for the captions and the record.

Handing work on. A request the model hands over runs as an ordinary turn, marked input_mode = "live", which a rule can match. One at a time: asked for a second while one runs, the call is told the agent is still working.

Limits. One call per person at a time. A call ends at max_call_seconds, or after idle_seconds with nothing from the page, both counted from when it is ready. The idle wait does not run while the agent works on a request, and a muted page is warned before it runs out. Microphone audio is taken no faster than it is spoken: what runs more than three seconds ahead is dropped, with one non-fatal audio_too_fast error, so max_call_seconds also bounds the audio you pay for.

After a call. What was said is kept in the conversation's session as a note in the agent's voice, so the next turn knows what was said and asked. The daemon accepts a call's record only for a call it granted, once, from the same person on the same conversation. A call that ends while a request it handed over is still running is kept in that turn's session.

The TUI's /live

In the TUI, /live holds a realtime session with an OpenAI, Azure OpenAI or Gemini instance, no browser involved. Its OpenAI session is built and read by the same code a web call uses. It is not a call, though: it has no ask_agent tool and no time limits. It uses [media.live] model, or else the instance's default_model rather than gpt-realtime-2.1, and the name must contain realtime or live. It ignores enabled, an OpenAI session's voice defaults to alloy rather than marin, it does not read [llm.voice], and the classification gate does not apply to it. See Slash Commands.

Troubleshooting

• No microphone in the web workspace. Read status.media.voice.input; reason and detail say what is missing. With no_engine, enable [media.stt] on a non-Azure instance, or make a model that hears audio the default or selectable. With classification, detail names the instance or the setting the gate refused.
• detail says the context engine could not say whether a scheme is enforced. The engine is unreachable, or refused the daemon's key: HTTP 403 means the key lacks config:read (on a single-tenant engine, add it to the daemon's SERVICE_TOKENS entry), and HTTP 404 means the engine predates this route or, on a multi-tenant engine, the key is not a tenant key, which keeps voice off for as long as that lasts. Voice stays off until the engine answers, and the daemon asks again after thirty seconds.
• Dictation works but voice messages do not. transcription is false: the transcribing instance has auto_transcribe_plugin_attachments = false.
• "No speech was heard" for every recording. The microphone is recording silence. In automated Chrome, a fake microphone that reads a file under /tmp records digital silence unless Chrome runs with --disable-features=AudioServiceSandbox.
• The transcript never appears beside the voice message, but the answer does. The client is not reading event: ugent.user_message on the chat stream. A client that reads only message events never sees it.
• No read-aloud button. Read status.media.voice.output. It is usually no_engine: no non-Azure instance enables [media.tts].
• Read aloud answers 404 for an answer on the screen. Only a finished turn's answer, as its receipt kept it, is read. The turn has not finished, ran before the page recorded turn ids, or its receipt is past its 30 days.
• A spoken answer will not play in Safari. Something between the browser and the web channel drops Range or answers it with 200. Safari plays audio only from a server that answers 206.
• No call button. Read status.media.voice.live. no_engine means no native OpenAI instance enables [media.live].
• A call ends at once as unavailable, or because of the provider. The provider refused the credential or the connection; the daemon's log names the instance. An OS sandbox that denies network to the plugin class stops the web channel reaching the provider at all.
• The call hears itself. Use headphones, or a browser whose echo cancellation covers its own playback. Push-to-talk avoids it entirely.

See Also

• The Web Workspace — what people see
• Plugins — the HTTP side, for client developers
• LLM Providers — instances and max_classification
• Data Classification — schemes and levels
• Rule Engine — matching input_mode
• Slash Commands — the TUI's /live