Google AI (Gemini)

Google AI provides the Gemini model family for chat, reasoning, image generation, video generation, text-to-speech, speech-to-text, and real-time multimodal interaction via the Live API. UGENT integrates with all of these through the native google provider type.

Overview

Capability	Models	API
Chat / reasoning	gemini-3.8-flash, gemini-3.1-pro-preview	generateContent
Image generation	gemini-3-pro-image, gemini-3.1-flash-image	generateContent (Nano Banana)
Video generation	veo-3.1-generate-001, veo-3.1-fast-generate-001	predictLongRunning (Veo, async)
Text-to-speech	gemini-3.8-flash-tts (default)	generateContent with AUDIO modality
Speech-to-text	Any Gemini chat model	generateContent with audio input
Live interaction	gemini-3.8-live	WebSocket (BidiGenerateContent)
Built-in search	Google Search grounding	generateContent with tools

Quick Start

[llm]
default_instance = "gemini"

[llm.instances.gemini]
type = "google"
api_key = "$GOOGLE_API_KEY"
default_model = "gemini-3.8-flash"
context_window = 1048576
max_tokens = 65536

temperature is not sent to Gemini 3 models; see Gemini 3 Request Rules.

Store your API key securely. Copy it to your clipboard, then pipe it in, so it never appears in your shell history:

macOS

pbpaste | ugent secret add @google_api_key --provider google --kind api-key --stdin

Linux X11

xclip -selection clipboard -o | ugent secret add @google_api_key --provider google --kind api-key --stdin

Vault & Secrets has the Wayland and Windows forms. Then reference the key with its vault handle:

[llm.instances.gemini]
api_key_ref = "@google_api_key"

Token Limits

[llm.instances.gemini]
type = "google"
api_key_ref = "@google_api_key"
default_model = "gemini-3.8-flash"
context_window = 1048576    # total context window (input + output)
max_tokens = 65536          # max output tokens

Gemini 3.8 Flash and Gemini 3.1 Pro (preview) both take about a million tokens of context (1,048,576) and write up to 65,536 tokens of output.

Reasoning (Thinking Mode)

Gemini models support a Thinking mode with configurable thinking level and budget. UGENT maps the standard mode / effort fields to Google's native parameters:

[llm.instances.gemini.reasoning]
mode = "enabled"           # disabled | enabled | adaptive
effort = "high"            # minimal | low | medium | high | xhigh | max

On Gemini 3, effort sets the thinking level: minimal and low send low, medium sends medium, and high, xhigh and max send high. enabled and adaptive behave the same; with neither an effort nor a level set, the model uses its own default.

For finer control, use Google-specific overrides:

[llm.instances.gemini.reasoning]
mode = "enabled"

[llm.instances.gemini.reasoning.provider.google]
include_thoughts = true    # return thought summaries in the response
thinking_level = "high"    # Gemini 3: low | medium | high
# thinking_budget = 8192   # Gemini 2.x only

• include_thoughts = true — the model's intermediate reasoning is visible in the response, useful for debugging and transparency.
• thinking_level wins over effort. On Gemini 3, minimal is sent as low, which Gemini 3.7 and 3.8 Flash require.
• thinking_budget (or the generic budget_tokens) applies to Gemini 2.x. Gemini 3 refuses a budget beside a level, so UGENT ignores it there and logs why.
• On Gemini 3, mode = "disabled" cannot switch thinking off; UGENT sends thinking_level = "low" and logs it. On Gemini 2.x it sends a budget of 0.

Gemini 3 Request Rules

From the Gemini 3 family on, Google refuses a set of request shapes, mostly with a 400 that names nothing useful. UGENT keeps to them for you:

• No temperature. It is ignored from Gemini 3.6 and documented to fail later, and Google warns that a value below its default of 1.0 can cause looping or degraded reasoning. Any temperature line is inert on these models.
• Thinking by level only, as described above, with each substitution logged.
• No prefill. A request may not end with a model turn. A model turn whose tool calls were never answered gets synthetic responses, and a genuine trailing assistant message is reported with a clear error rather than an opaque 400.
• Tool loops replay as Google specifies. The thought signature Gemini puts on the first function call of each step goes back on that call, every call id is echoed in its functionResponse, and results go back as functionResponse parts, one per call, in call order. A conversation that started on Claude or GPT continues on Gemini 3: an unsigned call gets Google's documented placeholder instead of failing.
• The model id decides. The version is read from the last path segment of the id, so a resource name (models/gemini-3.8-flash), a gateway slug (my-route/gemini-3.8-flash) or a versionless alias (gemini-flash-latest, treated as Gemini 3) gets the right rules. A per-turn model override is built with its own model's rules, not the instance default's.

An explicit context cache belongs to the instance's default model, so a turn on another model goes without it, and the log says so.

Built-in Google Search

Ground responses with real-time Google Search results without configuring a separate web search tool:

[llm.instances.gemini.builtin_search]
enabled = true

[llm.instances.gemini.builtin_search.google]
mode = "google_search"     # google_search | dynamic_retrieval
# dynamic_threshold = 0.3  # dynamic_retrieval only

When enabled, the model can access current information from the web during generation. This is separate from UGENT's own web search tools — it uses Google's native search grounding built into the Gemini API.

Image Generation (Nano Banana)

Gemini's Nano Banana models generate and edit images natively through the generateContent API with image response modalities:

[llm.instances.gemini.media.generation.image]
enabled = true
model = "gemini-3-pro-image"          # or gemini-3.1-flash-image
default_size = "1024x1024"
response_format = "b64_json"
max_images_per_request = 1

Nano Banana supports both text-to-image and image-to-image (provide reference image URLs). The model returns base64-encoded image bytes inline.

Imagen deprecated

Imagen models (imagen-3, imagen-4) use a separate predict endpoint and are deprecated with shutdown scheduled for August 2026. Use Nano Banana models for all new integrations.

Video Generation (Veo)

Google Veo generates videos asynchronously through a long-running operation pattern. UGENT handles the full lifecycle: create task, poll operation, download video.

[llm.instances.gemini.media.generation.video]
enabled = true
model = "veo-3.1-generate-001"           # or veo-3.1-fast-generate-001
default_ratio = "16:9"                   # the default
default_resolution = "720p"              # the default
default_duration = 8                     # seconds; the default
max_poll_duration_secs = 600
poll_schedule = [30, 5, 120, 15, 600, 30]

A request's own ratio, resolution and duration win over these.

Video modes:

• Text-to-video — prompt only
• Image-to-video — prompt + image_url (single image animated)
• Multi-image — prompt + image_urls (array)
• Keyframe — prompt + image_urls + mode = "keyframes"

Image inputs for Veo use base64-encoded format (Vertex AI style). UGENT handles the encoding automatically through the media transport layer.

Text-to-Speech (TTS)

Gemini TTS models generate natural speech in 30 prebuilt voices:

[llm.instances.gemini.media.tts]
enabled = true
model = "gemini-3.8-flash-tts"           # the default
voice = "Kore"                           # the default; Puck, Zephyr, Charon, Leda, ...

Available voices include: Kore, Puck, Zephyr, Charon, Leda, Orus, Aoede, and many more. Speech is single-speaker. gemini-3.8-flash-tts returns WAV; the earlier preview model returns bare PCM (16-bit, 24 kHz, mono), which UGENT wraps in a WAV header so any player opens it.

An instance with [media.tts] is what reads answers aloud; see Voice.

Speech-to-Text (STT)

Any Gemini chat model can transcribe audio natively — no dedicated STT model needed. Pass audio as an input part alongside a transcription prompt:

[llm.instances.gemini.media.stt]
enabled = true
model = "gemini-3.8-flash"                 # unset: the instance's default_model

The model transcribes audio content with high accuracy, including support for long-form audio and multiple speakers. Transcription goes through the instance's own Gemini client and key. A Gemini chat model can also hear a voice message itself (WAV, MP3, AIFF, AAC, OGG or FLAC) beside its transcript; see Voice.

Live API (Real-time Multimodal)

The Gemini Live API enables bidirectional WebSocket streaming for real-time audio and text interaction. UGENT opens a WebSocket to the BidiGenerateContent endpoint and manages the full session lifecycle. Open a session from the TUI with /live gemini.

Configuration

[llm.instances.gemini.media.live]
enabled = true
model = "gemini-3.8-live"           # set it: /live refuses a chat default_model
voice = "Kore"                      # unset: Google's default voice
input_audio = true                  # send microphone audio to Gemini
output_audio = true                 # receive spoken responses

Field	Default	Description
enabled	false	Read by live calls from the web workspace, which only OpenAI serves; /live works without it
model	none for /live	The Live model. /live needs it set, or a live default_model
voice	Google's default	The prebuilt voice spoken replies use
input_audio	true	Stream microphone audio input to the model (needs the live-audio build)
output_audio	true	Receive spoken audio output from the model

Set model here: /live refuses an instance whose live model is not a realtime or live model, and with model unset it checks the instance's default_model, which is usually a chat model.

Checked against recorded frames

Gemini Live support follows Google's current Live API reference and is tested against recorded protocol frames; it has not yet been run against Google's live service. Report anything that behaves differently.

How It Works

1. Connect — UGENT opens a WebSocket to wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent. The API key travels in the URL, as in Google's own examples, and is scrubbed from every error message.
2. Setup — Sends the model, the reply modality and voice, transcription of both sides, and session resumption, then waits up to 15 seconds for setupComplete.
3. Stream — Microphone audio goes up as PCM16 and typed turns as text. The model answers with audio and transcripts as it generates; what both sides say appears in the chat.
4. Interruptions — Talking over a reply interrupts it; the session goes on.
5. Resumption — When the server announces it will disconnect (goAway), the session resumes on a new connection with its last resumption handle.

Cache (Cached Content)

Gemini supports server-side context caching for large system prompts or documents that are reused across requests. Create a cache deliberately with the CLI, since it bills its tokens at about a tenth of the input rate plus storage for as long as it lives:

ugent cache create --instance gemini --system-file handbook.md --ttl 3600
ugent cache list --instance gemini
ugent cache delete --instance gemini cachedContents/abc123

Put the resource name it prints in the instance's config:

[llm.instances.gemini.cache]
enabled = true

[llm.instances.gemini.cache.provider.google]
cached_content = "cachedContents/abc123"

A cache is bound to the model and the credentials that created it.

Gemini Through an OpenAI-Compatible Gateway

Gemini also works behind a gateway, as an openai-compatible instance. Whether a request is Gemini is decided per request from its model id (for the instance's default model, its catalog_model counts too), so one gateway instance can serve several vendors turn by turn, and a non-Gemini turn gets none of Gemini's fields.

[llm.instances.gateway]
type = "openai-compatible"
base_url = "https://gateway.example.com/v1"
api_key_ref = "@gateway_key"
default_model = "gemini-3.8-flash"

• The temperature rule above applies, and thought signatures travel in tool_calls[].extra_content.google.thought_signature, read and sent back. A replayed turn with no signature at all (another provider's history, or a gateway that dropped the field) gets Google's placeholder.
• Two settings apply Gemini's rules to every request of an instance: a Google host as base_url, and vendor_profile = "gemini_openai" under [llm.instances.<name>.reasoning.provider.openai_compatible]. Neither belongs on an instance that also serves other vendors.
• [reasoning.provider.google] settings are sent only with vendor_profile = "gemini_openai", by the same rules as the native path: no budget on Gemini 3, and minimal and disabled as low.

Complete Configuration Example

[llm]
default_instance = "gemini"

[llm.instances.gemini]
type = "google"
api_key_ref = "@google_api_key"
default_model = "gemini-3.8-flash"
context_window = 1048576
max_tokens = 65536

# Reasoning
[llm.instances.gemini.reasoning]
mode = "enabled"
effort = "high"

[llm.instances.gemini.reasoning.provider.google]
include_thoughts = true

# Built-in search
[llm.instances.gemini.builtin_search]
enabled = true

[llm.instances.gemini.builtin_search.google]
mode = "google_search"

# Image generation
[llm.instances.gemini.media.generation.image]
enabled = true
model = "gemini-3-pro-image"
default_size = "1024x1024"

# Video generation
[llm.instances.gemini.media.generation.video]
enabled = true
model = "veo-3.1-generate-001"
default_ratio = "16:9"
default_resolution = "720p"
default_duration = 8
max_poll_duration_secs = 600

# Text-to-speech
[llm.instances.gemini.media.tts]
enabled = true
model = "gemini-3.8-flash-tts"
voice = "Kore"

# Speech-to-text
[llm.instances.gemini.media.stt]
enabled = true
model = "gemini-3.8-flash"

# Live API
[llm.instances.gemini.media.live]
enabled = true
model = "gemini-3.8-live"

Using Google with Other Providers

[llm]
default_instance = "gemini"

[llm.instances.gemini]
type = "google"
api_key_ref = "@google_api_key"
default_model = "gemini-3.8-flash"
context_window = 1048576

[llm.instances.agnes]
type = "openai-compatible"
base_url = "https://apihub.agnes-ai.com/v1"
api_key_ref = "@agnes_api_key"
default_model = "agnes-2.5-flash"
context_window = 512000

[llm.instances.opus]
type = "anthropic"
api_key_ref = "@anthropic_api_key"
default_model = "claude-opus-5-5"

Switch at runtime with /providers use gemini or /providers use agnes, or send one message elsewhere with /with agnes <prompt>.

Delegating Media Generation to Google

A text-capable model (Claude, GPT) can dispatch a sub-agent that runs on the Google instance to generate images, video, speech, or perform audio transcription. The sub-agent inherits Google's media generation tools based on the media.generation config.

The orchestrator does this through the delegate_task tool with provider_target set to the Google instance:

provider_target: "google:gemini"

The format is {provider_type}:{instance_name}. The main model stays on its reasoning provider while the delegated sub-agent handles media tasks on Google.

Google is particularly powerful as a media sub-agent because it covers the widest range of capabilities — image (Nano Banana), video (Veo), TTS, STT, and real-time Live API — all from one instance.