Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions TOC-ai.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,15 @@

- [Get Started via Python](/ai/quickstart-via-python.md)
- [Get Started via SQL](/ai/quickstart-via-sql.md)
- TiDB Cloud CLI (tdc) (Preview)
- [Overview](/ai/tdc/tdc-overview.md)
- [Quick Start](/ai/tdc/tdc-quick-start.md)

## CONCEPTS

- [Vector Search](/ai/concepts/vector-search-overview.md)
- TiDB Cloud CLI (tdc) (Preview)
- [Concepts and Architecture](/ai/tdc/concepts/tdc-concepts-and-architecture.md)

## GUIDES

Expand All @@ -30,6 +35,14 @@
- [Join Queries](/ai/guides/join-queries.md)
- [Raw SQL Queries](/ai/guides/raw-queries.md)
- [Transactions](/ai/guides/transactions.md)
- TiDB Cloud CLI (tdc) (Preview)
- [Install, Configure, and Update tdc](/ai/tdc/guides/tdc-install-configure-update.md)
- [Organization](/ai/tdc/guides/tdc-organization.md)
- [Starter Database](/ai/tdc/guides/tdc-starter-database.md)
- [Filesystem](/ai/tdc/guides/tdc-filesystem.md)
- [Filesystem Git](/ai/tdc/guides/tdc-filesystem-git.md)
- [Filesystem Journal](/ai/tdc/guides/tdc-filesystem-journal.md)
- [Filesystem Vault](/ai/tdc/guides/tdc-filesystem-vault.md)

## EXAMPLES

Expand All @@ -44,6 +57,14 @@
- [RAG Application](/ai/examples/rag-with-pytidb.md)
- [Conversational Memory](/ai/examples/memory-with-pytidb.md)
- [Text-to-SQL](/ai/examples/text2sql-with-pytidb.md)
- TiDB Cloud CLI (tdc) (Preview)
- [Agent Sandbox](/ai/tdc/examples/tdc-agent-sandbox-example.md)
- [Daily Workflow](/ai/tdc/examples/tdc-daily-workflow-example.md)
- [Query SQL with Roles](/ai/tdc/examples/tdc-query-sql-with-roles-example.md)
- [Share a Filesystem Across Machines](/ai/tdc/examples/tdc-share-filesystem-across-machines-example.md)
- [Git Workspace for Agents](/ai/tdc/examples/tdc-git-workspace-for-agents-example.md)
- [Journal an Agent Workflow](/ai/tdc/examples/tdc-journal-agent-workflow-example.md)
- [Delegate Vault Secrets](/ai/tdc/examples/tdc-vault-agent-secrets-example.md)

## INTEGRATIONS

Expand Down Expand Up @@ -84,3 +105,8 @@
- [Performance Tuning](/ai/reference/vector-search-improve-performance.md)
- [Limitations](/ai/reference/vector-search-limitations.md)
- [Changelogs](/ai/reference/vector-search-changelogs.md)
- TiDB Cloud CLI (tdc) (Preview)
- [CLI Reference](/ai/tdc/reference/tdc-cli-reference.md)
- [Configuration and Credentials](/ai/tdc/reference/tdc-configuration-and-credentials.md)
- [Regions, Security, and Limitations](/ai/tdc/reference/tdc-regions-security-and-limitations.md)
- [Troubleshooting](/ai/tdc/reference/tdc-troubleshooting.md)
41 changes: 41 additions & 0 deletions ai/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,21 @@ Get up and running quickly with TiDB's AI capabilities.
| [Get Started with Python](/ai/quickstart-via-python.md) | Build your first AI application with TiDB in minutes using Python. |
| [Get Started with SQL](/ai/quickstart-via-sql.md) | Quick start guide for vector search using SQL. |

### TiDB Cloud CLI (tdc) (Preview)

| Document | Description |
| --- | --- |
| [tdc Overview](/ai/tdc/tdc-overview.md) | Understand what tdc manages and how it uses its bundled Filesystem companion. |
| [Get Started with tdc](/ai/tdc/tdc-quick-start.md) | Install and configure tdc, then complete a first database or Filesystem operation. |

## Concepts

Understand the foundational concepts behind AI-powered search in TiDB.

| Document | Description |
| --- | --- |
| [Vector Search](/ai/concepts/vector-search-overview.md) | Comprehensive overview of vector search, including concepts, how it works, and use cases. |
| [tdc Concepts and Architecture (Preview)](/ai/tdc/concepts/tdc-concepts-and-architecture.md) | Understand profiles, regions, credentials, SQL roles, Filesystems, and the Drive9 companion boundary. |

## Guides

Expand All @@ -39,6 +47,18 @@ Step-by-step guides for building AI applications with TiDB using the [`pytidb`](
| [Auto Embedding](/ai/guides/auto-embedding.md) | Automatically generate embeddings on data insertion. |
| [Filtering](/ai/guides/filtering.md) | Filter search results with metadata conditions. |

### TiDB Cloud CLI (tdc) (Preview)

| Document | Description |
| --- | --- |
| [Install, Configure, and Update tdc](/ai/tdc/guides/tdc-install-configure-update.md) | Install release binaries, configure profiles, update, and uninstall tdc. |
| [Organization](/ai/tdc/guides/tdc-organization.md) | List projects and understand virtual-project selection. |
| [Starter Database](/ai/tdc/guides/tdc-starter-database.md) | Manage clusters, branches, SQL users, connection strings, and SQL execution. |
| [Filesystem](/ai/tdc/guides/tdc-filesystem.md) | Manage resources, data, layers, packs, and FUSE or WebDAV mounts. |
| [Filesystem Git](/ai/tdc/guides/tdc-filesystem-git.md) | Clone, hydrate, and manage linked Git worktrees. |
| [Filesystem Journal](/ai/tdc/guides/tdc-filesystem-journal.md) | Record, search, and verify append-only workflow events. |
| [Filesystem Vault](/ai/tdc/guides/tdc-filesystem-vault.md) | Store secrets, delegate access, audit, inject, and mount a read-only vault. |

## Examples

Complete code examples and demos showcasing TiDB's AI capabilities.
Expand All @@ -52,6 +72,18 @@ Complete code examples and demos showcasing TiDB's AI capabilities.
| [Conversational Memory](/ai/examples/memory-with-pytidb.md) | Persistent memory for AI agents and chatbots. |
| [Text-to-SQL](/ai/examples/text2sql-with-pytidb.md) | Convert natural language to SQL queries. |

### TiDB Cloud CLI (tdc) (Preview)

| Document | Description |
| --- | --- |
| [Agent Sandbox](/ai/tdc/examples/tdc-agent-sandbox-example.md) | Give a clean sandbox Filesystem access without TiDB Cloud API keys. |
| [Daily Workflow](/ai/tdc/examples/tdc-daily-workflow-example.md) | Manage one Starter cluster and Filesystem in a routine operator flow. |
| [Query SQL with Roles](/ai/tdc/examples/tdc-query-sql-with-roles-example.md) | Use explicit read-only, read-write, and admin SQL roles. |
| [Share a Filesystem Across Machines](/ai/tdc/examples/tdc-share-filesystem-across-machines-example.md) | Transfer an owner token securely and verify cross-machine visibility. |
| [Git Workspace for Agents](/ai/tdc/examples/tdc-git-workspace-for-agents-example.md) | Prepare a mounted Git workspace and isolated linked worktree. |
| [Journal an Agent Workflow](/ai/tdc/examples/tdc-journal-agent-workflow-example.md) | Record structured events and verify their hash chain. |
| [Delegate Vault Secrets](/ai/tdc/examples/tdc-vault-agent-secrets-example.md) | Grant an agent temporary access to one secret field. |

## Integrations

Integrate TiDB with popular AI frameworks, embedding providers, and development tools.
Expand All @@ -75,3 +107,12 @@ Technical reference documentation for TiDB's AI and vector search features.
| [Vector Search Index](/ai/reference/vector-search-index.md) | Create and manage vector indexes for performance. |
| [Performance Tuning](/ai/reference/vector-search-improve-performance.md) | Optimize vector search performance. |
| [Limitations](/ai/reference/vector-search-limitations.md) | Current limitations and constraints. |

### TiDB Cloud CLI (tdc) (Preview)

| Document | Description |
| --- | --- |
| [CLI Reference](/ai/tdc/reference/tdc-cli-reference.md) | Global flags, output, queries, dry-run, help, errors, and aliases. |
| [Configuration and Credentials](/ai/tdc/reference/tdc-configuration-and-credentials.md) | Profiles, precedence, local state, credentials, mount locators, and logs. |
| [Regions, Security, and Limitations](/ai/tdc/reference/tdc-regions-security-and-limitations.md) | Placement, authentication boundaries, platforms, durability, and Preview constraints. |
| [Troubleshooting](/ai/tdc/reference/tdc-troubleshooting.md) | Diagnose authentication, quota, SQL, companion, selection, and mount failures. |
129 changes: 129 additions & 0 deletions ai/tdc/concepts/tdc-concepts-and-architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
---
title: tdc Concepts and Architecture
summary: Understand tdc profiles, regions, credentials, SQL roles, Filesystem resources, local state, and the bundled Drive9 companion.
---

# tdc Concepts and Architecture

This document explains the concepts needed to use tdc with TiDB Cloud Starter and TiDB Cloud Filesystem.

> **Note:**
>
> tdc is currently in Preview. Its features and command-line interface might change without prior notice.

## Command model

tdc uses service nouns and explicit operation names:

```text
tdc db create-db-cluster
tdc fs copy-file
tdc fs-git clone-git-workspace
```

The command tree is at most two levels. Long, descriptive command and flag names make intent explicit in logs and agent-generated commands. Commands are non-interactive except for `tdc configure`.

Structured commands return JSON by default. Use `--output text` for a terminal-oriented representation and `--query` for a JMESPath projection.

## Profiles and regions

A profile is a named local namespace containing TiDB Cloud placement, a default virtual project, and credentials. The default profile is named `default`; select another with `--profile`.

tdc represents placement with one canonical region code:

```text
aws-us-east-1
aws-ap-southeast-1
ali-ap-southeast-1
```

The prefix identifies the cloud provider. A global `--region` value overrides `TDC_REGION_CODE` and the profile region for one command without changing saved configuration.

During `tdc configure`, tdc calls the organization API and requires one accessible project with `type = "tidbx_virtual"`. It stores that project ID as the default for Starter cluster creation.

## Credential boundaries

tdc uses separate credentials for separate security boundaries:

| Credential | Purpose | Storage |
| --- | --- | --- |
| TiDB Cloud API public/private key | Organization, Starter control plane, Filesystem provisioning and deletion | `~/.tdc/credentials` |
| DB SQL username/password | SQL access to one Starter cluster | `~/.tdc/db_users/<cluster-id>/credentials` |
| Filesystem owner token | Filesystem data plane, mounts, Git, journal, and owner vault operations | Per-resource credentials under `~/.tdc/fs_resources/` or `TDC_FS_TOKEN` |
| Delegated vault token | Limited access to selected secret fields | `TDC_VAULT_TOKEN` or an explicit command input |

TiDB Cloud API keys are never reused as SQL passwords or Filesystem tokens.

## SQL roles

`tdc db create-db-sql-users` creates or repairs three stable users for a cluster:

- `read_only` for queries that must not modify data;
- `read_write` for normal application and agent work;
- `admin` for DDL and privilege administration.

Role selection is explicit with `--read-only`, `--read-write`, or `--admin`. Read-write is the default when no role flag is supplied.

## One profile, multiple Filesystems

One profile can register multiple Filesystem resources. Each resource has an isolated config file and credential file. Select a resource in this order:

1. `--file-system-name`;
2. `TDC_FS_FILE_SYSTEM_NAME`;
3. the profile's default Filesystem;
4. the only registered Filesystem, when exactly one exists.

If multiple resources exist and none is selected, tdc fails instead of guessing. Use `tdc fs set-default-file-system` to choose a default.

## Config-free sandbox access

A clean agent sandbox does not need `tdc configure` or TiDB Cloud API keys to use an existing Filesystem. Supply:

```bash
export TDC_FS_TOKEN="<owner-token>"
export TDC_REGION_CODE="aws-us-east-1"
export TDC_FS_FILE_SYSTEM_NAME="workspace"
```

tdc resolves these values into an in-memory profile namespace. It does not write an `[env]` profile or persist the token.

## Local state

All tdc-owned state is under `~/.tdc/`:

| Path | Contents |
| --- | --- |
| `config` | Non-sensitive profile, default project, and logging settings |
| `credentials` | TiDB Cloud API keys |
| `fs_resources/` | Per-profile, per-Filesystem metadata and owner credentials |
| `db_users/` | Cluster-scoped SQL credentials |
| `mounts/` | Non-secret locators for active background mounts |
| `logs/tdc.jsonl` | Redacted local operation log |
| `bin/` | Installed `tdc` and `tdc-drive9` binaries |

Operation logging is local and is not telemetry. Set `TDC_LOGGING=off` to disable it for one process.

## tdc and the Drive9 companion

tdc installs [Drive9](https://github.com/mem9-ai/drive9) as the private companion name `tdc-drive9`.

tdc owns:

- profile, credential, region, and Filesystem selection;
- TiDB Cloud control-plane calls;
- JSON/text output, queries, errors, and local logging;
- translation from the tdc command surface to the companion.

The companion owns:

- Filesystem reads, writes, metadata, links, search, and layers;
- FUSE and WebDAV mount processes, caches, drain, and unmount;
- pack/unpack, Git workspace, journal, and vault semantics.

A background mount leaves a long-running `tdc-drive9 mount --foreground` process. `tdc fs drain-file-system` asks that process to flush pending FUSE work, and `tdc fs unmount-file-system` stops the mount. Do not terminate a machine with unflushed writes or local-only overlay data that you need to keep.

## What's next

- [Install, Configure, and Update tdc](/ai/tdc/guides/tdc-install-configure-update.md)
- [tdc Configuration and Credentials](/ai/tdc/reference/tdc-configuration-and-credentials.md)
- [tdc Regions, Security, and Limitations](/ai/tdc/reference/tdc-regions-security-and-limitations.md)
118 changes: 118 additions & 0 deletions ai/tdc/examples/tdc-agent-sandbox-example.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
---
title: Use TiDB Cloud Filesystem in an Agent Sandbox
summary: Provision a Filesystem on a trusted machine and give a clean agent sandbox config-free access without TiDB Cloud API keys.
---

# Use TiDB Cloud Filesystem in an Agent Sandbox

This example gives an ephemeral coding agent a durable workspace without copying a user's complete tdc configuration into the sandbox.

> **Note:**
>
> tdc is currently in Preview. Its features and command-line interface might change without prior notice.

## The agent problem

Coding agents often start in clean, short-lived sandboxes. The local disk disappears when the sandbox is replaced, but the agent still needs previous artifacts, repository state, and files produced by other workers. Rebuilding that state wastes task time, while copying `~/.tdc/` or injecting TiDB Cloud API keys gives the sandbox control-plane credentials it does not need.

## Why local storage and full cloud credentials are not enough

Check warning on line 18 in ai/tdc/examples/tdc-agent-sandbox-example.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [PingCAP.Ambiguous] Consider using a clearer word than 'enough' because it may cause confusion. Raw Output: {"message": "[PingCAP.Ambiguous] Consider using a clearer word than 'enough' because it may cause confusion.", "location": {"path": "ai/tdc/examples/tdc-agent-sandbox-example.md", "range": {"start": {"line": 18, "column": 57}}}, "severity": "INFO"}

A sandbox-local directory is fast but not durable or shared. Generic object-storage APIs require application-specific download and upload logic instead of ordinary file operations. Giving every sandbox the user's complete cloud credentials solves access at the cost of a much broader security boundary.

Check warning on line 20 in ai/tdc/examples/tdc-agent-sandbox-example.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [PingCAP.Ambiguous] Consider using a clearer word than 'much' because it may cause confusion. Raw Output: {"message": "[PingCAP.Ambiguous] Consider using a clearer word than 'much' because it may cause confusion.", "location": {"path": "ai/tdc/examples/tdc-agent-sandbox-example.md", "range": {"start": {"line": 20, "column": 272}}}, "severity": "INFO"}

## How tdc changes the workflow

A trusted machine provisions the Filesystem once. The sandbox receives only the Filesystem owner token, region code, and Filesystem name, and can immediately use data-plane, mount, Git, journal, and vault workflows without `tdc configure`. When an agent needs only selected secrets, use a delegated vault token instead of the owner token.

## Prerequisites

- Install and configure tdc on a trusted machine.
- Install tdc in the sandbox. The release installer includes `tdc-drive9`.
- Use a secure secret manager or encrypted sandbox input for token transfer.

## Step 1. Provision on the trusted machine

```bash
export TDC_FS_TOKEN="$(tdc fs create-file-system \
--file-system-name agent-sandbox \
--query fs_token \
--output text)"
```

Record the canonical region code used by the profile, for example `aws-us-east-1`. Do not print the token.

## Step 2. Inject the minimum sandbox environment

Configure the sandbox secret/environment mechanism with:

```bash
TDC_FS_TOKEN=<owner-token>
TDC_REGION_CODE=aws-us-east-1
TDC_FS_FILE_SYSTEM_NAME=agent-sandbox
```

The sandbox does not need `TDC_PUBLIC_KEY`, `TDC_PRIVATE_KEY`, `tdc configure`, or files copied from `~/.tdc/`.

## Step 3. Verify direct access

In the sandbox:

```bash
printf 'sandbox ready\n' | tdc fs copy-file \
--from-stdin \
--to-remote /sandbox/status.txt

tdc fs read-file --path /sandbox/status.txt
```

Expected output:

```text
sandbox ready
```

## Step 4. Optionally mount the Filesystem

On Linux with FUSE:

```bash
mkdir -p /workspace
tdc fs mount-file-system \
--file-system-name agent-sandbox \
--mount-path /workspace \
--driver fuse

cat /workspace/sandbox/status.txt
```

On macOS, omit `--driver fuse` to use the default WebDAV path. Use FUSE only after installing macFUSE.

After mounting, you can use `tdc fs-git`, `tdc fs-journal`, and owner-authorized `tdc fs-vault` commands with the same FS environment. Give agents a delegated `TDC_VAULT_TOKEN` instead of the owner token when they need only selected secret fields.

## Cleanup

Stop writers. For FUSE:

```bash
tdc fs drain-file-system --mount-path /workspace
tdc fs unmount-file-system --mount-path /workspace
```

For WebDAV, close files and run only unmount. Back on the trusted machine:

```bash
tdc fs delete-file-system \
--file-system-name agent-sandbox \
--confirm-file-system-name agent-sandbox
```

## Security notes

- Treat `TDC_FS_TOKEN` as an owner credential.
- Do not place it in an image, repository, command flag, or operation log.
- Deleting the sandbox does not delete the remote Filesystem.
- Draining is required before deleting a FUSE sandbox when pending writes must be durable.

## What's next

- [Manage TiDB Cloud Filesystem with tdc](/ai/tdc/guides/tdc-filesystem.md)
- [tdc Configuration and Credentials](/ai/tdc/reference/tdc-configuration-and-credentials.md)
Loading
Loading