Document Sync

UGENT's context client is what sends a workspace's documents to the context engine's document catalog. It sends:

• every extraction the web workspace keeps in catalog mode;
• each document's latest event, whether a new text, a rename or a removal;
• every receipt back.

This page is the operator's side: what to switch on, in which order, the files involved, and what the client refuses. What readers and agents get is in Document Versions; the tenant's side is Documents.

Document Author: Uni Zhu

Switching it on

Three switches, turned on in this order:

1. The engine and the tenant. The engine needs a build with its documents feature, which release builds include, and [documents] enabled = true in its server.toml. Those platform keys are read at startup, so restart the engine after changing them; a service should also set database_url, artifact_dir and profiles_dir there to absolute paths. A tenant then has documents on where tenant_default = "on", or switches them on itself where tenant_may_change = true. The console's readiness panel shows where they stand.
2. UGENT. [context_client.documents] in ugent.toml, in a build with the documents feature. Release builds include it.
3. The web workspace, last. UGENT_DOCUMENT_CATALOG_ENABLED=true in ugent-web's environment. Catalog mode writes no legacy extraction for a new document, so a document turned on too early is in search only once the catalog holds it.

[context_client.documents]
enabled = true
max_artifact_bytes = 104857600   # 100 MiB
accept_identity_fields = false

Key	Default	Means
enabled	false	Send the documents of every workspace that holds catalog files. A workspace with none, such as a code repository, registers nothing and is left alone.
max_artifact_bytes	104857600	The largest artifact the client uploads. A larger one is refused before anything of its document is sent. Keep it within the engine's limit for the tenant.
accept_identity_fields	false	Let a document's manifest name its own work, variant, version or relationships. The web workspace writes none of them. Turn it on only for a producer you trust to say which document a file is.

UGENT's engine key, the [context_client] token, needs codebase:write and also actor:assert. The web workspace names the person who uploaded each file, and the engine accepts that name only from a key allowed to assert it:

• without actor:assert, every upload is refused with 403 uploader_mismatch, and since catalog mode writes no legacy extraction, those uploads never reach search;
• a key bound to one actor (actor:bind:<id>) may post only that actor's uploads.

How UGENT decides it may send:

• The engine has to say yes. Nothing is sent until the engine's GET /v1/health advertises documents.catalog.v1 and documents.versions.v1. The answer is remembered for five minutes, or for thirty seconds when the engine did not answer.
• A build without the feature logs once that enabled = true has no effect, and sync works as before.
• In every build, the catalog's files stay out of ordinary sync: extraction results, event files, receipts and the root marker. A legacy extraction of a document that happens to be named results is still synced.

The web workspace reads three variables:

Variable	Default	Means
UGENT_DOCUMENT_CATALOG_ENABLED	false	Keep every extraction as an immutable result and write each source's latest event. Off, extraction works as before and writes no catalog file. Switching it off again extracts each cataloged document once more in the old form.
UGENT_DOCUMENT_STAGING_DIR	.ugent-document-staging beside the workspace root	Where a result is assembled before it is renamed into place: outside the workspace (inside it is refused), and on the same filesystem, so the rename is atomic.
UGENT_DOCUMENT_REMOVAL_RECORD_DAYS	30	How many days a removal record is kept in its folder's doca_extraction/sources/.

The web server says at start when it cannot assemble results in the staging folder. It warns, at most hourly, when documents have waited more than six hours for UGENT's receipt.

Citations

Two more settings let a citation of an exact version reach the reader.

Mark the engine as an evidence source. The engine's document_read and document_compare return text with [[cite:doc:...]] pins and list the same pins in their structured result, and workspace_search lists its hits' markers the same way. Only a service marked as an evidence source may supply pins an answer can cite. Without the mark, every document citation, and every citation of a hit the agent found by searching, is dropped as invented:

[mcp.services.ugent-context]
evidence_source = true

Set it only for a context engine you trust: anything the service lists becomes citable. It is off by default.

Give the web chat its citation relay. A citation chip in the web workspace resolves through channel-web's relay, as the reader who sees it. The relay holds its own engine key, scoped to nothing but resolving citations for your web users' issuer:

• the scope is citations:<issuer>:resolve, such as citations:ugent-web:resolve;
• citations:*:resolve covers every issuer.

The relay resolves only for readers of its own federation source: UGENT_WEB_DIRECTORY_SOURCE where it is set, else the first issuer's actor prefix, else ugent-web. The <issuer> in the key's scope must be that source.

Set UGENT_WEB_CITATION_ENGINE_URL and UGENT_WEB_CITATION_CODEBASE_ID in ~/.ugent/plugins.env (or UGENT's environment), and list them in the channel-web plugin's env_refs in plugins.toml: the supervisor passes on only the variables a plugin declares, and silently ignores the rest. Give the key from the vault:

[[plugins]]
id = "web-main"
kind = "channel-web"
enabled = true
command = "/usr/local/bin/channel-web-worker"
env_refs = [
  "UGENT_WEB_API_KEY",
  # ...every other variable the worker reads, then:
  "UGENT_WEB_CITATION_ENGINE_URL",    # the engine's data plane
  "UGENT_WEB_CITATION_CODEBASE_ID",   # the workspace whose documents are cited
]

[plugins.secret_env_refs]
UGENT_WEB_CITATION_ENGINE_KEY = "@context_engine_citation_key"

The relay needs all three: without any one of them it stays off, and answers 501. Given the URL but not the key or the workspace id, it also warns at start. While it is off, every document citation chip reads Source unavailable. When the relay itself fails, the web server's log says why.

Teach the agent to cite by marker. A chip appears for each passage an answer cites by its marker, [[cite:...]]: the one retrieval puts above a passage, the citation of a workspace_search hit, or the one a document tool puts after a paragraph. The web chat shows each marker as a numbered link to its chip. Agents write markers unprompted only when told to, so copy the section "Citing a passage by its marker" from ugent-web's AGENTS.workspace.md template into the workspace AGENTS.md that ugent-web reads. Copy it whole: it names each kind of marker the agent can cite. Keep it out of ~/.agents.md, and out of any workspace that also serves another channel: other channels show a marker as written.

The files involved

In catalog mode the web workspace writes these beside each folder's documents, inside its doca_extraction folder:

<folder>/doca_extraction/results/<request_key>/<result_hash>/   result.json, document.md, pages.json, images/
<folder>/doca_extraction/sources/<source_key>.ugent.json       the latest event: a manifest or a removal record
<folder>/doca_extraction/sources/<source_key>.state.json       the engine's receipt, written by UGENT

• A result is immutable. A result is assembled outside the workspace and renamed in, so nothing half-written is ever sent. The same document under the same settings reuses its result. An OCR or vision-model extraction run again writes a second result beside it.
• Each document has one event file, holding its latest event. A rename, or a change only to the details Google Drive reports, is a new event with no new download. An unchanged rescan writes nothing.
• The root marker, .ugent-producer at the workspace root, holds the workspace's producer key, ugent-<uuid>. UGENT writes it when the engine first registers the producer, and never again.
• UGENT's own state sits beside its sync manifest: document-producer.json (the key and its current epoch), document-envelopes.json (each event's place in its document's order) and document-files.json (what the engine recorded for each event file).

The agent cannot write these control files with its file tools. write_file and edit_file refuse any path under doca_extraction/sources/, a .ugent.json or .state.json file beneath a doca_extraction folder, and .ugent-producer, in any letter case. A shell command is not covered, which is why each event file speaks only for its own folder; see below.

How it runs

Document sync runs beside ordinary sync, so an upload never delays a text push and a failure there never fails a sync.

• Full pass. Whenever a full sync starts (at startup, on /context sync, on a reconcile), UGENT walks the workspace for event files. It posts each one that changed, writes receipts, and runs a complete scan.
• Event pass. When the watcher sees an event file change, UGENT posts that file's event.
• Retries. An event the engine could not take is sent again after five seconds, the wait doubling up to five minutes, with no trigger needed. That covers no answer, retries used up, a producer not yet registered, and documents switched off for the tenant.
• Starting over. /context force-sync with no path forgets what was sent, so every event and its files go again. That is what an engine restored from a backup needs. The engine keeps each event where it first recorded it, so nothing moves twice.
• Uploads are streamed from their files one at a time and hashed again on the way. A file that changed in between fails its upload rather than reach the engine. Each upload is allowed the usual timeout plus the time a 4 Mbit/s link needs for its size.

Order is kept across retries, restarts and restores. The client writes each event's place down before it first sends it, and reuses that place, under the same idempotency key, on every retry. A lost answer is therefore replayed rather than recorded twice.

A restored state directory is behind by one edit

If UGENT's state is restored from an older backup, it takes a new epoch for the events the engine never recorded, and a restored old event file never becomes the current version. The first new event of a document the engine already has further along is recorded as history, and the engine keeps the version it had. Edit or save that document once more, and that becomes current.

Which folder speaks for a document

An event file speaks only for documents in its own folder, the one holding its doca_extraction. Anyone who can write a folder can withdraw documents beneath it, which they could delete anyway, and no more. Before posting, UGENT refuses:

• A copy's claim on the original. It refuses a manifest for a document whose manifest it last posted from another folder's event file, while that file is still there. A copied folder does not take over its original's documents. Once the original's event file is gone, as a move leaves it, the copy is accepted at once, with no restart.
• A removal from elsewhere. It refuses a removal record from outside the folder of the document's last manifest, or the folders above it. The web workspace writes a deleted folder's removals into its parent, so one folder up is all it needs.
• Identity claims. It refuses a manifest that names its own work, variant, version or relationships, unless accept_identity_fields is on.

These refusals depend on other files too, so a refused file is judged again at every pass. Deleting an event file starts a pass.

Removals and complete scans

• A removal is posted at once, when its folder may speak for the document. A file that disappears with no removal record is left to the next complete scan.
• A complete scan lists every document whose event file holds a manifest and whose file is there, in pages of 5,000, and finishes with the root marker. The engine then takes out what the scan no longer lists, as long as that stays within a budget. Beyond the budget it takes out nothing and asks a person instead; see missing-source proposals.
• A walk that went wrong completes the scan as incomplete, and the engine takes nothing out. That covers a directory UGENT could not read, an event file it could not attribute to a document, and a folder whose name contains ...
• A root without its marker begins no scan at all, because an unmounted mount point looks exactly like an empty workspace. The marker is checked again once the walk is done.

Receipts

After posting, UGENT asks the engine for each document's receipt, in batches of 1,000, and writes it as <source_key>.state.json. The web workspace reads the receipt before it deletes a document's old-style extraction. It deletes it only once three things hold:

• the catalog holds the current event;
• selection has run;
• search has indexed the document, or excluded it for a stated reason.

A receipt that is not settled yet is asked for again after five seconds, doubling up to five minutes. While documents are off for the tenant, no receipt reads as indexed, so old-style extractions stay.

When something is refused

UGENT refuses an event before uploading anything when:

• its schema or one of its fields is unknown;
• its event file is named for another document, or is larger than 1 MiB;
• its document's path climbs out of its folder, or is not in the folder holding the event's doca_extraction;
• its result is missing, reached through a link out of its doca_extraction, or does not match its hashes;
• an artifact is larger than max_artifact_bytes;
• its metadata is past the engine's default limits: 64 KiB of raw metadata, or 256 values;
• it names an uploader for a file that was not uploaded through the web workspace, or an uploader who is not a federated user.

Each such refusal follows from the file's bytes alone. It is logged once and not read again until the file changes; a restart reads it again and refuses it again.

The engine may refuse too, for reasons that rest on its side rather than on the file. These are tried again at the next full pass, logged once per file and reason:

• 422 profile_not_requestable and 422 invalid_metadata, which rest on the tenant's configuration;
• 403 uploader_mismatch, which rests on what UGENT's key may assert (see its key).

403 documents_disabled refuses no event: documents are off for the tenant, so the pass stops, nothing is remembered as refused, and the retry asks again until they are back on.

Switching it off

In this order:

1. Turn documents off for the tenant, in the console's Documents, or for the platform. Documents leave search at once, and the catalog stops taking new ones. Nothing is deleted.
2. Clear projections for each workspace, if the index space matters.
3. Switch the web workspace's catalog mode off. Each cataloged document gets its old-style extraction back.
4. Turn off [context_client.documents].

Turning everything back on needs no new extraction: the engine projects its documents again, and its embedding cache serves the text it has seen.