LLM Providers

UGENT supports multiple LLM providers simultaneously. Each provider is configured as a named instance with its own model, API key, and transport settings.

Basic Setup

[llm]
default_instance = "opus"

[llm.instances.opus]
type = "anthropic"
api_key = "$ANTHROPIC_API_KEY"
default_model = "claude-opus-5-5"
context_window = 1000000
max_tokens = 128000

The default_instance determines which provider UGENT uses unless overridden.

Supported Providers

Type	Provider	Example models
openai	OpenAI	GPT-6 Sol, GPT-6 Luna, GPT-6.1 Sol, GPT-6 Astra
anthropic	Anthropic, Amazon Bedrock	Claude Opus 5.5, Sonnet 5.5, Haiku 5.5
google	Google AI, Vertex AI	Gemini 3.8 Flash, Gemini 3.1 Pro (preview)
dashscope	Alibaba Cloud	Qwen 3.8 Max, Qwen 3.8 Flash
openrouter	OpenRouter	Any model via unified API
ollama	Ollama (local)	Any local model
openai-compatible	Any OpenAI-compatible endpoint	Gateways, Azure OpenAI, DeepSeek, vLLM, LM Studio
bytedance	ByteDance Seedance (media only)	Seedance video, Seed Audio
decisions	TypeSafe Jev, SubQ Snap, OpenAI Decisions, local runtimes	Typed probability answers via the decide tool

openai_compatible is accepted as a spelling of openai-compatible.

Model Catalog

UGENT carries a catalog of models: each one's context window, output limit, prices, the input it accepts (images, audio, documents) and the parameters it refuses. It decides output caps, when to compress, what a turn costs, which attachments reach a model, and whether temperature is sent.

The catalog is built into the binary. UGENT_MODELS_FILE points prices, context windows, output limits and the temperature rule at your own copy, and an edited file applies to those without a restart. Which input a model accepts (images, audio, documents, and so whether it hears a voice message), the startup self-check and the setup wizard always read the built-in copy. [tokens] models_file names a file that is read first for context-window budgets only. A data/models_v2.json in the directory UGENT starts in is never read, so where you launch it from changes nothing.

Gateway model ids

A gateway often names a catalogued model with wrappers around it. UGENT finds the entry by peeling off standard wrappers only: a path segment (openai/gpt-6-sol), a Bedrock region profile and vendor namespace (us.anthropic.claude-haiku-5-5), a version suffix (-v1:0) and a snapshot date (@20250929, -2026-07-09). So us.openai.gpt-6-sol is priced and sized as gpt-6-sol, with no setting.

Nothing is guessed beyond that: us.openai.gpt-6-sol-custom, or a deployment alias of your own, reaches no entry. Map such a name with catalog_model:

[llm.instances.work]
type = "openai-compatible"
base_url = "https://gateway.example.com/v1"
api_key_ref = "@gateway_key"
default_model = "team-default-model"   # your gateway's own name
catalog_model = "gpt-6-luna"           # the entry it bills and sizes as

catalog_model applies while the instance runs its default_model. The startup self-check warns about an instance whose default model reaches no entry and sets no catalog_model; such an instance reports no cost.

Costs are estimates

Costs are computed from the catalog's public list prices, not from your provider's invoice, so they are estimates. The TUI marks them with a tilde (~$0.050) in its status bar, token panel and monitoring cards; the web workspace labels them "Estimated model cost". A negotiated rate or a gateway's markup makes your bill differ, and where a provider charges more above a prompt length, the estimate uses the base rate.

Token Limits

[llm.instances.opus]
type = "anthropic"
api_key = "$ANTHROPIC_API_KEY"
default_model = "claude-opus-5-5"
max_tokens = 128000           # max output tokens per response
context_window = 1000000      # total context window (input + output)

max_tokens (Output Limit)

The maximum number of tokens the model can generate in a single response. Keep it at or below the model's output limit: a provider refuses a larger value.

0, the default, is auto mode. UGENT works the value out from the model's catalog entry, never above its output limit; native OpenAI and OpenRouter leave the field out and let the API decide. For a model the catalog does not know, auto mode falls back to 16384 tokens on OpenAI-wire providers, or 65536 with reasoning on so that reasoning and the visible answer both fit, and to 128000 on Anthropic.

max_output_tokens caps output below whatever max_tokens allows, for cost control or to keep a verbose reasoning model short.

context_window (Total Context)

The total token budget (input + output) the model accepts. UGENT uses this to calculate compression thresholds and projection budgets. If omitted, UGENT uses max_tokens when that is not 0, and otherwise the context window from the catalog. Because max_tokens is an output limit, set context_window whenever you set max_tokens, or compression starts long before the window is full.

When the conversation approaches context_window, UGENT triggers automatic context compression to summarize older messages and keep the conversation going.

Temperature

temperature (default 0.7) is sent unless something says the model refuses it. Any one of three is enough:

• the instance sets disable_temperature = true;
• the model's catalog entry marks it unsupported, as it does for GPT-5 and later, Claude from Opus 4.7 on and Gemini 3.5 and later;
• the model-generation rules, which also cover models the catalog does not know: GPT-5 and every later GPT behind any gateway wrapper, every Claude from 4.7 on, and Gemini 3.

So a temperature line is inert on the current Claude, GPT and Gemini models. A refusal always wins, because sending a refused field fails the request while leaving out an accepted one only loses a setting. For a gateway whose models refuse the field and that the catalog does not know, switch it off:

[llm.instances.gateway]
type = "openai-compatible"
base_url = "https://gateway.example.com/v1"
api_key_ref = "@gateway_key"
default_model = "house-model-large"
disable_temperature = true    # never send temperature from this instance

max_classification

The highest data classification level this instance is approved to receive, as a level id from your context engine's scheme.

[llm.instances.onprem]
type = "openai-compatible"
base_url = "https://llm.internal.example.com/v1"
api_key = "@llm_onprem_key"
default_model = "qwen3.8-flash"
max_classification = "confidential"   # nothing above this reaches it

Sent with every pre-LLM retrieval, so the engine leaves anything above it out of what it returns. It can only narrow: the engine intersects it with the asking actor's clearance, so an instance cannot use this to receive more than the person asking may read.

Resolved per turn, from the instance the turn actually routed to. With per-user model selection a turn can land on a different provider than the default, and the ceiling follows it.

An instance that sets none declares no ceiling, which normally reads as "no limit" — the actor's own clearance still applies. But a tenant, or the whole deployment, can refuse any retrieval that declares none. Where that is on, an instance without max_classification gets 403 model_ceiling_required and the turn answers with no context. The tenant's Access page reports how many recent requests declared none, so this is checkable before the switch is turned on rather than after.

Use the level id, not a display name. "机密" is a name; the id is something like "restricted". A value that is not a legal id cannot be sent as an HTTP header, so the retrieval request fails and the turn answers with no context at all — the startup self-check names the instance when this is wrong.

Leaving it unset asserts no limit, which is right where there is no scheme. Once one exists, set it on every instance: the engine can be configured to refuse a request that declares none.

Multiple Instances

[llm.instances.opus]
type = "anthropic"
api_key = "$ANTHROPIC_API_KEY"
default_model = "claude-opus-5-5"

[llm.instances.fast]
type = "openai"
api_key = "$OPENAI_API_KEY"
default_model = "gpt-6-luna"

[llm.instances.local]
type = "ollama"
base_url = "http://localhost:11434"
default_model = "llama4:70b"

Switch at runtime with /providers use <instance>, send one message to another instance with /with, or let the orchestrator pick an instance per delegated task.

Which Model a Turn Runs On

Each turn runs on one model, decided when the turn starts:

1. the model the turn names: /with <instance>[@<model>] <prompt> in the TUI or REPL, a cron task's provider_target, or the model field of a chat request;
2. for a channel or web user, the model they chose with /model, then their caller's default (see Per-User Model Selection);
3. the default instance's default_model.

In the TUI and REPL, /providers use <instance> changes the default instance and /providers model <name> its model; /with changes neither. Headless, ugent -x "<prompt>" --instance <name> --model <id> picks both for the run.

A turn's model belongs to that turn. A plain message after a /with message runs on the default again, and a model one chat request names is not the model that user's next message gets.

Delegating and resumed turns

A turn that delegates is resumed by the runtime once its workers finish. The resumption runs on the model the turn ran on, on whichever instance, so a turn on a model you or the user named finishes on it rather than on the default.

That model is checked again first, by the rule that admitted it:

Who named it	Checked against
A user (/model, a request's model, a caller default)	The user-selection policy (allow_user_select, models_allowlist)
The operator (/with, a cron task's provider_target)	The instance still being configured

If the check fails, the resumption runs on the default model and the log says why; it is never refused, since a refused resumption would lose its workers' answer. A turn on the default model resumes on the default. Workers run on their own task's model, not their parent's, and a worker that delegates in turn continues on its own task's model.

Failover

Chain fallback instances so UGENT automatically retries on a backup when the primary fails:

[llm.instances.opus]
type = "anthropic"
api_key = "$ANTHROPIC_API_KEY"
default_model = "claude-opus-5-5"
fallback_instances = ["fast", "local"]

When the primary instance exhausts all transport retries, UGENT tries each fallback in order.

Reasoning (Extended Thinking)

For models that support extended thinking (Claude, Gemini, GPT-5 and later, the OpenAI o-series, DeepSeek):

Modes

[llm.instances.opus.reasoning]
mode = "adaptive"            # disabled | enabled | adaptive
effort = "high"              # minimal | low | medium | high | xhigh | max

Mode	Behavior
disabled	Turn thinking off where the model allows it. Gemini 3, GPT-6.1 Sol and GPT-6 Astra cannot, and get their lowest setting; Claude Opus 5.5 cannot either, so lower effort there.
enabled	Always think before responding. On Claude this is manual thinking with budget_tokens, which current Claude models refuse; UGENT upgrades it to adaptive for them.
adaptive	The model decides how much to think. Native on Anthropic and the right mode for every current Claude model. Gemini treats it as enabled; OpenAI and OpenAI-compatible fall back to enabled with a warning.

Effort Levels

Effort	Description
minimal	Briefest reasoning, fastest response
low	Light reasoning
medium	Balanced
high	Deep reasoning (recommended for complex tasks)
xhigh	Extra-deep reasoning
max	Maximum reasoning budget

Each provider is sent the nearest level the model accepts. minimal becomes low on Claude, on Gemini 3 and on GPT-5.1 and later, which have no minimal.

Provider Differences

Anthropic. Claude models from 4.7 on (Opus 5.5, Sonnet 5.5 and Haiku 5.5 among them) think adaptively and refuse manual budget_tokens thinking; UGENT upgrades mode = "enabled" to adaptive and warns. Their default effort is medium on Opus 5.5 and Haiku 5.5 and high on the others, so set effort. See Anthropic.

[llm.instances.opus.reasoning]
mode = "adaptive"
effort = "high"
# budget_tokens = 32000      # manual thinking, older Claude models only

OpenAI. GPT-5 and later and the o-series take effort, sent as reasoning.effort on the Responses API and reasoning_effort on Chat Completions. GPT-5.6 and later reason at medium by default; an instance without a [reasoning] block asks for medium with a reasoning summary. See OpenAI.

[llm.instances.reasoning-pro.reasoning]
mode = "enabled"
effort = "high"
# No budget_tokens: OpenAI manages the budget internally

Google (Gemini). Gemini 3 takes a thinking level, which effort sets (xhigh and max become high); a budget is ignored there, and thinking cannot be switched off, so disabled sends low. budget_tokens applies to Gemini 2.x. See Google AI.

[llm.instances.gemini.reasoning]
mode = "enabled"
effort = "medium"

OpenAI-compatible (DeepSeek, Zhipu, Qwen and others). UGENT recognises the vendor from the base URL and sends its own thinking switch. Anything else the endpoint takes goes through extra_body, merged into the request body:

[llm.instances.work.reasoning]
mode = "enabled"
effort = "high"

[llm.instances.work.reasoning.provider.openai_compatible.extra_body]
enable_thinking = true         # a vendor-specific field, sent as written

Reasoning is replayed within a conversation only to the provider that produced it, and for OpenAI only to the same endpoint and key. Switching providers mid-conversation is safe; the new model just does not see the earlier reasoning. Reasoning summaries and Claude's progress notes show as a one-line status in the TUI, never in the answer.

Prompt Caching

Prompt caching is automatic. UGENT places cache breakpoints on the stable system prompt prefix and injects dynamic context (memory, code context, date) after the breakpoint — so cached tokens survive across turns.

[llm.instances.opus.cache]
auto = true                   # auto-activate provider-appropriate strategy

Provider-specific cache telemetry is shown in the TUI status bar.

Transport Reliability

All providers share a unified transport layer with:

• Bounded HTTP retries with exponential backoff + jitter
• Stream idle watchdog (configurable timeout)
• Retry-After header support
• Separate interactive and background profiles

[llm.transport.default]
request_timeout_secs = 300
stream_idle_timeout_secs = 120
http_max_retries = 2
stream_max_retries = 2

[llm.transport.retry]
initial_backoff_ms = 400
max_backoff_ms = 5000

The values shown are the defaults. [llm.transport.interactive] and [llm.transport.background] take the same keys as default and override it for foreground turns and for background work; the background profile already allows 600 s per request and 180 s of stream silence.

Per-instance overrides take a profile too:

[llm.transport_instance_overrides.fast.default]
request_timeout_secs = 30

Image generation is held to less. An error the provider answers with is returned at once, because asking again gets the same answer while the person waits; only a request that got no answer at all, a dropped connection or a timeout, is sent once more. http_max_retries still caps both.

Sub-Agent Cost Control

Reserve expensive models for the parent agent and let sub-agents use cheaper ones:

[llm.instances.opus]
type = "anthropic"
allow_subagent_use = false    # reserved for parent only

[llm.instances.fast]
type = "openai"
default_model = "gpt-6-luna"
allow_subagent_use = true     # sub-agents may use this

When any instance opts in with allow_subagent_use = true, delegated sub-agents may only use the opted-in instances. If the orchestrator omits the provider target, it auto-substitutes the first approved instance so the worker never silently inherits the parent's expensive default.

The rule holds at every dispatch, not only when delegate_task is called: a fallback chain offers a worker only approved instances, a worker's own failover skips unapproved instances and the parent's default, and a worker created without a target runs on an approved instance. A refused dispatch fails with Provider target '<target>' is not approved for sub-agent use, and the parent's handoff names it.

An optional models_allowlist further restricts which model names a sub-agent may select:

[llm.instances.fast]
allow_subagent_use = true
models_allowlist = ["gpt-6-luna", "gpt-6-sol"]

Per-User Model Selection (Web Channel)

For multi-user web deployments, each user can choose their own model:

[llm.instances.fast]
type = "openai"
default_model = "gpt-6-luna"
allow_user_select = true

When any instance opts in, only opted-in instances are selectable by web users. This is default-deny — an instance is user-selectable only when it explicitly sets allow_user_select = true. Operators can toggle at runtime with /providers policy <instance> user-select on|off.

Users choose with the /model command in their chat:

Command	Effect
/model (or /models)	List the models this user may choose
/model <name>	Use <name> for this user's later messages
/model clear	Go back to the default
/model check <name>	Say whether <name> may run a turn; stores nothing

A name is a bare model id (gpt-6-luna), or <provider>:<instance>@<model> to pick one instance's copy of it (openai:fast@gpt-6-luna). When two selectable instances offer a bare id, the default instance wins. The choice is stored for that user only and never changes anyone else's model. It is kept under the user's own identity, their federated sign-in or else their channel user, and does not follow an identity link.

A chat request can also name a model for one turn, in the model field of POST /v1/chat/completions. That model is the turn's own. It is checked before anything runs, so one the policy does not allow is a 400, and it is stored nowhere: it does not become the user's model for their next message, and two of one user's turns in flight each keep their own. ugent-default names no model. A model removed between that check and the start of the turn fails the turn rather than running it on another model.

A turn that names no model runs on the user's /model choice, then on the default model set for their caller, then on the global default. The caller default is keyed by the federation issuer the caller asserts, takes the same forms as a request's model, and must be user-selectable itself:

[caller_defaults."reporting-service"]
model = "openai:fast@gpt-6-luna"

Multimodal Input

Images, audio, video, and documents are supported per provider capability:

[llm.instances.vision]
type = "openai"
api_key = "$OPENAI_API_KEY"
default_model = "gpt-6-sol"

[llm.instances.vision.multimodal]
unsupported_policy = "error"        # error | fallback

[llm.instances.vision.multimodal.interactive]
max_parts_per_message = 12
max_image_inline_bytes = 20971520   # 20 MiB

[llm.multimodal] takes the same keys for every instance at once.

Which modalities a model accepts comes from the model catalog, and for audio from the route the request takes as well. On native OpenAI only the audio chat models, which go to Chat Completions, hear a recording; the Responses API takes none. Gemini always can, and Anthropic and Ollama never do. An attachment a route cannot carry meets unsupported_policy: error refuses the request, fallback sends a note in its place.

Voice messages are transcribed before they reach any model. See Voice.