Configuration

Configure the corpus and the Workers

The corpus sync reads targets.json and a few environment variables. Each Worker reads its own wrangler file, its vars and its secrets.

targets.json

A target names one AI Search instance, the R2 bucket it reads, and the repos that go into it. repos are directory names under the clone root. Only git-tracked files are uploaded.

{
  "includePaths": { "example-repo": ["docs/"] },
  "excludePaths": { "example-repo": ["vendor/private-notes/"] },
  "targets": {
    "corpus": {
      "instance": "my-ai-search-instance",
      "bucket": "my-search-corpus",
      "repos": ["example-repo"]
    }
  }
}

With the npm CLIs, the file is ./targets.json, or the path in SEARCH_MCP_TARGETS. In a clone of the repository it is scripts/targets.json. The full example is scripts/targets.json.example.

Check what a target will upload before you trust it:

node scripts/sync.mjs my-target --dry-run

The sync renames files that AI Search does not index natively, such as .ts, .tsx, .service and an extensionless Dockerfile, to .txt keys, so AI Search indexes them.

Bounding a corpus: includePaths and excludePaths

excludePaths is a denylist and fails open. A new top-level file in a repo joins the corpus with no warning. That is fine for a docs site. It is wrong when the corpus boundary matters.
includePaths is an allowlist and fails closed. When a repo has an entry, only paths under those prefixes can go into the corpus. The sync refuses everything else.
They compose by subtraction. includePaths decides what is eligible. Then excludePaths removes paths from that set. A denylist entry can never add a path back.
Two levels. A map at the top level applies to every target that lists the repo. A map nested under a target applies only to that target, and wins for that repo when both levels set it. So one repo can be full-tree in one target and _corpus/ only in another.
An empty entry is an error. Omit a repo from includePaths to admit all of it. An entry that is present but empty ([]) does not mean "index nothing". The sync refuses it.

When the sync refuses to run

An allowlist that matches nothing is worse than no allowlist. The sync would plan zero objects, the prune would delete the corpus that was there, and the reindex would succeed over nothing. So each of these refuses the run with exit code 2, before anything uploads or is pruned:

Refusal	Meaning
`include_paths_no_match`	An entry matched zero git-tracked files: a typo, or a directory that moved.
`include_paths_entry_empty`	The entry is `[]`.
`include_paths_entry_invalid`	An entry cannot match a git path: absolute, `..`, a backslash, or not a string.
`include_paths_entry_not_array`, `include_paths_not_object`	The config has the wrong shape.
`include_paths_unknown_repo`	The entry names a repo that no target lists, so nothing reads it.
`include_paths_repo_not_cloned`	An allowlisted repo is missing from the clone root.
`include_paths_all_excluded`	`excludePaths` removed everything the allowlist selected.
`include_paths_all_filtered`	Nothing the allowlist selected passed the ingest filters.

The shape and repo-name checks run for every target on every sync, not only the target being synced. A stale excludePaths repo name gives a warning, not a refusal.

Size cap

The sync skips objects larger than SYNC_MAX_BYTES (default 4 MB). It lists the skipped files at the end of the run, because a missing file looks the same as a file the corpus never had. Use --fail-on-skip to make an incomplete corpus a failed run:

SYNC_MAX_BYTES=$((16 * 1024 * 1024)) node scripts/sync.mjs corpus --fail-on-skip

Corpus manifest

A corpus producer can publish a manifest.json next to its objects. The widget then shows citations instead of raw R2 keys. Each entry needs a key. The widget uses title, url, page and total_pages when they are present.

{ "pages": [
  { "key": "my-doc/p003.txt", "title": "Deploy guide", "url": "/docs/deploy/", "page": 3, "total_pages": 12 }
] }

Keys match exactly first, then by suffix. The sync stores every object under its repo name (<repo>/<path>), but a producer writes its manifest with its own paths.

Sync environment variables

Variable	Use
`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`	An R2 API token with Object Read and Write on the corpus bucket only.
`CLOUDFLARE_ACCOUNT_ID`	The account that holds the bucket.
`CORPUS_GIT_ORG`	The GitHub organization the sync runner clones from.
`GITHUB_TOKEN`	Read access to the contents of the source repos, for the sync runner.
`CORPUS_ROOT`	The sync runner's own clone directory, kept apart from your working clones.
`SYNC_REPO_ROOT`	The clone root for `search-mcp-sync`. Default: the current directory with the npm CLIs, or the parent of the clone in a repository checkout.
`SEARCH_MCP_TARGETS`	The path to `targets.json`.
`SYNC_MAX_BYTES`	The size cap.

Query Worker settings

Start from wrangler.toml.example.

Setting	Kind	Use
`SEARCH`	AI Search binding	Set `instance_name` to your instance.
`ASK_LIMITER`	Rate limit binding	The per-IP limit on `/ask`. The example allows 20 requests in 60 seconds.
`ALLOWED_ORIGINS`	Var	A comma-separated list of the page origins that may call `/ask`.
`GENERATION_MODEL`	Var	The model that writes the answer.
`ASSISTANT_SYSTEM_PROMPT`	Var, optional	The system prompt for answers.
`ORIGIN_PROFILES`	Var, optional	A JSON object that maps an exact request origin to a system prompt for that site. See per-site system prompts.
`TURNSTILE_SECRET`	Secret, optional	Turns on the Turnstile check. When it is not set, the Worker skips verification.

MCP Worker settings

Start from wrangler.mcp.toml.example.

Setting	Kind	Use
`MCP_TOKEN`	Secret, required	One token, or comma-separated `name=token` pairs so that logs name each consumer. Without it the Worker refuses every request.
`SEARCH`	AI Search binding	Set `instance_name` to your instance.
`CORPUS`	R2 binding, recommended	The bucket the sync uploads to. `get_file` and `corpus_status` need it, and `list_repos` can read repo names from it.
`CORPUS_REPOS`	Var, optional	A JSON array of repo names, or `{"repos":[...],"description":"..."}`. When set, `list_repos` uses it instead of listing R2.
`GENERATION_MODEL`	Var, optional	The model for the `ask` tool.
`GET_FILE_MAX_BYTES`	Var, optional	The default byte limit for `get_file`.

Reindex behavior

AI Search refuses a new reindex job in two cases, and search-mcp-sync-run waits out both before it starts one:

A job is running. A new job does not queue behind it. Cloudflare ends the running job and starts again. So the runner waits for the running job to end.
The cooldown after a job. After a job ends, a new one is refused for a short time with sync_in_cooldown (code 7020). The runner retries until it clears.

The runner also retries unable_to_connect_to_ai_search (code 7017) under the cooldown budget. Each wait has its own budget of 10 minutes. If a budget runs out, the run fails and says so. The files in R2 are already uploaded, so nothing is lost: the index catches up on the next sync.