Google AI (Gemini)

Google AI provides the Gemini model family for chat, reasoning, image generation, video generation, text-to-speech, speech-to-text, and real-time multimodal interaction via the Live API. UGENT integrates with all of these through the native google provider type.

Overview

Capability	Models	API
Chat / reasoning	gemini-3.5-pro, gemini-2.5-flash	generateContent
Image generation	gemini-3-pro-image, gemini-3.1-flash-image	generateContent (Nano Banana)
Video generation	veo-3.0, veo-2.0	predictLongRunning (Veo, async)
Text-to-speech	gemini-3.1-flash-tts-preview	generateContent with AUDIO modality
Speech-to-text	Any Gemini chat model	generateContent with audio input
Live interaction	gemini-2.5-flash-native-audio	WebSocket (BidiGenerateContent)
Built-in search	Google Search grounding	generateContent with tools

Quick Start

[llm]
default_instance = "gemini"

[llm.instances.gemini]
type = "google"
api_key = "$GOOGLE_API_KEY"
default_model = "gemini-3.5-pro"
context_window = 1000000
max_tokens = 65536
temperature = 0.7

Store your API key securely:

echo -n "your-google-api-key" | ugent secret add --stdin google_api_key

Then reference it with a vault handle:

[llm.instances.gemini]
api_key_ref = "@google_api_key"

Token Limits

[llm.instances.gemini]
type = "google"
api_key_ref = "@google_api_key"
default_model = "gemini-3.5-pro"
context_window = 1000000    # total context window (input + output)
max_tokens = 65536          # max output tokens
temperature = 0.7

Gemini 3.5 Pro supports up to 1 million tokens of context. Gemini 2.5 Flash supports 1 million tokens as well, with lower latency and cost.

Reasoning (Thinking Mode)

Gemini models support a Thinking mode with configurable thinking level and budget. UGENT maps the standard mode / effort fields to Google's native parameters:

[llm.instances.gemini.reasoning]
mode = "enabled"           # disabled | enabled | adaptive
effort = "high"            # minimal | low | medium | high | xhigh | max

For finer control, use Google-specific overrides:

[llm.instances.gemini.reasoning.google]
include_thoughts = true    # return thinking tokens in the response
thinking_budget = 8192     # max tokens for thinking

• mode = "adaptive" — UGENT activates Thinking based on task complexity.
• include_thoughts = true — the model's intermediate reasoning is visible in the response, useful for debugging and transparency.

Built-in Google Search

Ground responses with real-time Google Search results without configuring a separate web search tool:

[llm.instances.gemini.search]
enabled = true

[llm.instances.gemini.search.google]
mode = "google_search"     # google_search | google_search_retrieval

When enabled, the model can access current information from the web during generation. This is separate from UGENT's own web search tools — it uses Google's native search grounding built into the Gemini API.

Image Generation (Nano Banana)

Gemini's Nano Banana models generate and edit images natively through the generateContent API with image response modalities:

[llm.instances.gemini.media.generation.image]
enabled = true
model = "gemini-3-pro-image"          # or gemini-3.1-flash-image
default_size = "1024x1024"
response_format = "b64_json"
max_images_per_request = 1

Nano Banana supports both text-to-image and image-to-image (provide reference image URLs). The model returns base64-encoded image bytes inline.

Imagen deprecated

Imagen models (imagen-3, imagen-4) use a separate predict endpoint and are deprecated with shutdown scheduled for August 2026. Use Nano Banana models for all new integrations.

Video Generation (Veo)

Google Veo generates videos asynchronously through a long-running operation pattern. UGENT handles the full lifecycle: create task, poll operation, download video.

[llm.instances.gemini.media.generation.video]
enabled = true
model = "veo-3.0"                        # or veo-2.0
default_width = 1280
default_height = 768
default_num_frames = 121                 # must satisfy 8n+1
default_frame_rate = 24                  # 1-60 FPS
max_poll_duration_secs = 600
poll_schedule = [30, 5, 120, 15, 600, 30]

Video modes:

• Text-to-video — prompt only
• Image-to-video — prompt + image_url (single image animated)
• Multi-image — prompt + image_urls (array)
• Keyframe — prompt + image_urls + mode = "keyframes"

Image inputs for Veo use base64-encoded format (Vertex AI style). UGENT handles the encoding automatically through the media transport layer.

Text-to-Speech (TTS)

Gemini TTS models generate natural speech in 30 prebuilt voices:

[llm.instances.gemini.media.tts]
enabled = true
model = "gemini-3.1-flash-tts-preview"
default_voice = "Kore"                   # Kore, Puck, Zephyr, Charon, Leda, etc.

Available voices include: Kore, Puck, Zephyr, Charon, Leda, Orus, Aoede, and many more. Multi-speaker mode is supported for dialogue-style content.

Output is PCM audio (16-bit signed, 24kHz, mono), automatically saved to the media store.

Speech-to-Text (STT)

Any Gemini chat model can transcribe audio natively — no dedicated STT model needed. Pass audio as an input part alongside a transcription prompt:

[llm.instances.gemini.media.stt]
enabled = true
model = "gemini-3.5-pro"                 # or gemini-2.5-flash

The model transcribes audio content with high accuracy, including support for long-form audio and multiple speakers.

Live API (Real-time Multimodal)

The Gemini Live API enables bidirectional WebSocket streaming for real-time audio, video, and text interaction. UGENT opens a WebSocket to the BidiGenerateContent endpoint and manages the full session lifecycle.

Configuration

[llm.instances.gemini.media.live]
enabled = true
# transport = "websocket"          # websocket (default) or webrtc
input_audio = true                  # send microphone audio to Gemini
output_audio = true                 # receive spoken responses
input_video = false                 # send video frames (camera/screen)
realtime_transcript = true          # live transcription overlay

Field	Default	Description
enabled	false	Enable the Live API for this instance
transport	websocket	Transport protocol: websocket or webrtc
input_audio	true	Stream microphone audio input to the model
output_audio	true	Receive spoken audio output from the model
input_video	false	Stream video frames (camera or screen share)
realtime_transcript	true	Show live text transcription alongside audio

Model

The Live API uses a dedicated model optimized for real-time audio:

[llm.instances.gemini]
# The Live API defaults to this model when no explicit model is set:
default_model = "gemini-2.5-flash-native-audio"

You can override the model per session, but the native-audio variant is recommended for the best real-time experience.

How It Works

The protocol runs in four phases:

1. Connect — UGENT opens a WebSocket to wss://generativelanguage.googleapis.com/ws/...BidiGenerateContent?key=YOUR_KEY
2. Setup — Sends a setup message with the model name, generation config, and enabled modalities (audio input/output, video, transcript). Waits for a setupComplete confirmation.
3. Stream — Sends realtime input chunks: AudioChunk (PCM16 bytes), VideoFrame (JPEG bytes), or Text. The model responds with audio and text as it generates — no waiting for a full turn.
4. Close — The session ends when the user disconnects or sends EndTurn.

Using Live Mode

Start a voice session from the TUI or web interface. UGENT handles audio capture, encoding, and streaming automatically. You speak, Gemini responds in real time — no request/response round-trip latency.

For video input, set input_video = true and use the web interface to share your camera or screen. The model can analyze live video while you talk.

Cache (Cached Content)

Gemini supports server-side context caching for large system prompts or documents that are reused across requests:

[llm.instances.gemini.cache]
enabled = true

[llm.instances.gemini.cache.provider.google]
cached_content = "cached-content/your-cache-id"

Cached content reduces latency and cost for repeated context by storing it server-side and referencing it by ID.

Complete Configuration Example

[llm]
default_instance = "gemini"

[llm.instances.gemini]
type = "google"
api_key_ref = "@google_api_key"
default_model = "gemini-3.5-pro"
context_window = 1000000
max_tokens = 65536
temperature = 0.7

# Reasoning
[llm.instances.gemini.reasoning]
mode = "enabled"
effort = "high"

[llm.instances.gemini.reasoning.google]
include_thoughts = true
thinking_budget = 8192

# Built-in search
[llm.instances.gemini.search]
enabled = true

[llm.instances.gemini.search.google]
mode = "google_search"

# Image generation
[llm.instances.gemini.media.generation.image]
enabled = true
model = "gemini-3-pro-image"
default_size = "1024x1024"

# Video generation
[llm.instances.gemini.media.generation.video]
enabled = true
model = "veo-3.0"
default_width = 1280
default_height = 768
default_num_frames = 121
default_frame_rate = 24
max_poll_duration_secs = 600

# Text-to-speech
[llm.instances.gemini.media.tts]
enabled = true
model = "gemini-3.1-flash-tts-preview"
default_voice = "Kore"

# Speech-to-text
[llm.instances.gemini.media.stt]
enabled = true
model = "gemini-3.5-pro"

Using Google with Other Providers

[llm]
default_instance = "gemini"

[llm.instances.gemini]
type = "google"
api_key_ref = "@google_api_key"
default_model = "gemini-3.5-pro"
context_window = 1000000

[llm.instances.agnes]
type = "openai-compatible"
base_url = "https://apihub.agnes-ai.com/v1"
api_key_ref = "@agnes_api_key"
default_model = "agnes-2.5-flash"
context_window = 512000

[llm.instances.opus]
type = "anthropic"
api_key_ref = "@anthropic_api_key"
default_model = "claude-opus-4-8"

Switch at runtime with /model gemini or /model agnes.

Delegating Media Generation to Google

A text-capable model (Claude, GPT) can dispatch a sub-agent that runs on the Google instance to generate images, video, speech, or perform audio transcription. The sub-agent inherits Google's media generation tools based on the media.generation config.

The orchestrator does this through the delegate_task tool with provider_target set to the Google instance:

provider_target: "google:gemini"

The format is {provider_type}:{instance_name}. The main model stays on its reasoning provider while the delegated sub-agent handles media tasks on Google.

Google is particularly powerful as a media sub-agent because it covers the widest range of capabilities — image (Nano Banana), video (Veo), TTS, STT, and real-time Live API — all from one instance.