-
Notifications
You must be signed in to change notification settings - Fork 724
docs: add TiDB Cloud Filesystem product documentation #23876
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Icemap
wants to merge
1
commit into
pingcap:release-8.5
Choose a base branch
from
Icemap:docs/filesystem-product
base: release-8.5
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,000
−1
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| <!-- markdownlint-disable MD007 --> | ||
| <!-- markdownlint-disable MD041 --> | ||
|
|
||
| # Table of Contents | ||
|
|
||
| ## Getting Started | ||
|
|
||
| - [Introduction](/tidb-cloud-filesystem/_index.md) | ||
| - [Quick Start](/tidb-cloud-filesystem/filesystem-quick-start.md) | ||
| - Mounting Locally | ||
| - [Overview](/tidb-cloud-filesystem/filesystem-mount.md) | ||
| - [Linux](/tidb-cloud-filesystem/filesystem-mount-linux.md) | ||
| - [macOS](/tidb-cloud-filesystem/filesystem-mount-macos.md) | ||
| - [Docker and Docker Compose](/tidb-cloud-filesystem/filesystem-mount-docker.md) | ||
|
|
||
| ## Concepts | ||
|
|
||
| - [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md) | ||
| - [Sharing Filesystems](/tidb-cloud-filesystem/filesystem-sharing.md) | ||
| - [Branches and Checkpoints](/tidb-cloud-filesystem/filesystem-branches-checkpoints.md) | ||
|
|
||
| ## Reference | ||
|
|
||
| - [Filesystem CLI Commands](/ai/ti/reference/ti-filesystem.md) | ||
| - [Regions and Limitations](/ai/ti/reference/ti-regions-security-and-limitations.md) | ||
| - [Troubleshooting](/ai/ti/reference/ti-troubleshooting.md) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| --- | ||
| title: TiDB Cloud Filesystem | ||
| summary: Learn how TiDB Cloud Filesystem keeps agent files available across sessions, shares workspaces, and isolates changes with layers. | ||
| --- | ||
|
|
||
| # TiDB Cloud Filesystem | ||
|
|
||
| TiDB Cloud Filesystem is persistent, shared file storage for applications and AI agents. Keep a workspace in the cloud, access it from different machines, and retain its files after a sandbox or process ends. | ||
|
|
||
| Use the TiDB Cloud Command Line Interface (`ti`) to create a Filesystem, upload and download files, or mount a remote directory for tools that work with local paths. You do not need to provision or manage a separate database to get started. | ||
|
|
||
| > **Note:** | ||
| > | ||
| > TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. | ||
|
|
||
| ## Keep work beyond one session | ||
|
|
||
| An agent might collect source documents in one sandbox, generate a report in another, and hand the result to a person on a laptop. Files stored only on the sandbox's local disk disappear when that environment is removed. Repeatedly exporting and importing archives also creates separate copies that can drift apart. | ||
|
|
||
| With a Filesystem, these participants use the same remote files. A new environment needs an access token and the Filesystem's region code, not the creator's TiDB Cloud API keys or local configuration directory. | ||
|
|
||
| - **Continue a task in another environment.** Mount the existing workspace instead of recreating its input files. | ||
| - **Share results with a reviewer.** Give the reviewer a read-only scoped token for a report directory. | ||
| - **Compare parallel approaches.** Fork layers for independent drafts, create checkpoints, and commit a selected result to the shared base. | ||
|
|
||
| ## Work with files through the CLI or a mount | ||
|
|
||
| The CLI provides two ways to use the same remote Filesystem: | ||
|
|
||
| - **Direct file commands:** upload, download, list, read, and search files with `ti fs`. These commands do not require FUSE or a mount. | ||
| - **Local mounts:** expose remote files at a local directory so existing tools can use commands such as `cat`, `cp`, and `ls`. Linux uses FUSE. macOS uses WebDAV by default and can use macFUSE for layer and checkpoint mounts. | ||
|
|
||
| For example, after selecting a Filesystem and supplying its token: | ||
|
|
||
| ```bash | ||
| # Read a remote file without mounting it. | ||
| ti fs read-file --path /workspace/report.md | ||
| ``` | ||
|
|
||
| Or mount the workspace on macOS or Linux: | ||
|
|
||
| ```bash | ||
| # Give local tools a directory backed by the remote workspace. | ||
| mkdir -p "$HOME/workspace" | ||
| ti fs mount-file-system --remote-path /workspace --mount-path "$HOME/workspace" | ||
| cat "$HOME/workspace/report.md" | ||
| ``` | ||
|
|
||
| ## Control sharing and changes | ||
|
|
||
| - [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md) distinguishes resource-management credentials from owner and scoped Filesystem tokens. | ||
| - [Sharing Filesystems](/tidb-cloud-filesystem/filesystem-sharing.md) explains how participants access one remote workspace without sharing account credentials. | ||
| - [Branches and Checkpoints](/tidb-cloud-filesystem/filesystem-branches-checkpoints.md) explains layers, saved history points, and publishing changes to the base Filesystem. | ||
|
|
||
| ## Before you begin | ||
|
|
||
| Choose how to access the Filesystem: | ||
|
|
||
| - **Create and manage a Filesystem:** obtain a TiDB Cloud API public key and private key from the [API Keys page](https://tidbcloud.com/org-settings/api-keys), then run `ti configure`. Follow the [Quick Start](/tidb-cloud-filesystem/filesystem-quick-start.md) to create a Filesystem and write your first file. | ||
| - **Use an existing Filesystem:** if someone has supplied an FS token, set `TI_FS_TOKEN` and `TI_REGION_CODE`. You do not need TiDB Cloud API keys or `ti configure` to access files within the token's permissions. Follow [Mounting Locally](/tidb-cloud-filesystem/filesystem-mount.md#use-a-token-without-configuring-a-profile). | ||
|
|
||
| See [Authorization](/tidb-cloud-filesystem/filesystem-authorization.md) for the differences between API keys, owner tokens, and scoped tokens. An FS token does not grant permission to create or delete a Filesystem through the CLI. | ||
|
|
||
| Choose a supported Filesystem region: `aws-us-east-1`, `aws-ap-southeast-1`, `aws-us-west-2`, or `alicloud-ap-southeast-1`. Keep compute close to that region when possible to reduce network latency. | ||
|
|
||
| Mount support depends on the operating system and its permissions. Windows supports direct file commands but not native mounting through `ti`. WebDAV does not support layer or checkpoint mounts. Buffered writes must reach the service before you remove an environment; a successful local write alone is not a durability guarantee. | ||
|
|
||
| These guides focus on file storage and access. TiDB Cloud Filesystem does not provision an agent's compute environment. A layer is not a security sandbox, and checkpoints do not snapshot the entire live base Filesystem. | ||
|
|
||
| ## What's next | ||
|
|
||
| - [Create a Filesystem and write your first file](/tidb-cloud-filesystem/filesystem-quick-start.md). | ||
| - [Mount an existing Filesystem locally](/tidb-cloud-filesystem/filesystem-mount.md). | ||
| - [Try the interactive agent sandbox lab](https://labs.tidb.io/labs/demo_901). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| --- | ||
| title: TiDB Cloud Filesystem Authorization | ||
| summary: Choose TiDB Cloud API keys, owner tokens, or scoped tokens to separate Filesystem administration from application data access. | ||
| --- | ||
|
|
||
| # TiDB Cloud Filesystem Authorization | ||
|
|
||
| The person who creates a Filesystem and the agent that uses its files do not need the same credentials. Keep account-level management on a trusted machine and give each application only the Filesystem access it needs. | ||
|
|
||
| > **Note:** | ||
| > | ||
| > TiDB Cloud Filesystem is currently in public preview. Its features and interfaces are subject to change without notice. | ||
|
|
||
| ## Choose the credential type | ||
|
|
||
| ### TiDB Cloud API keys | ||
|
|
||
| A TiDB Cloud public/private API key pair authorizes resource-management operations according to the account's permissions. Use it to create, list, describe, and delete Filesystems, generate owner tokens, and configure Filesystem AI providers. | ||
|
|
||
| Configure the keys with `ti configure`, or provide `TIDB_CLOUD_PUBLIC_KEY` and `TIDB_CLOUD_PRIVATE_KEY` together. They are not FS tokens and do not directly replace the token used by a mount. | ||
|
|
||
| ### Owner FS tokens | ||
|
|
||
| An owner token grants broad access within one Filesystem, including reading, writing, and deleting files. It can issue scoped tokens, list token metadata, and revoke tokens in that Filesystem. It can enable or disable scoped tokens, but enabling or disabling owner tokens requires TiDB Cloud API keys. It is a high-privilege secret, not a read-only mount credential. | ||
|
|
||
| An owner FS token is not interchangeable with TiDB Cloud API keys: it cannot create or delete the Filesystem resource or generate another owner token through `ti`. In particular, deleting a file and deleting its Filesystem are different permissions. | ||
|
|
||
| Filesystem creation returns an owner token and stores it locally for the creating profile. To generate another owner token, use `ti fs generate-file-system-token` with TiDB Cloud API keys. | ||
|
|
||
| ### Scoped FS tokens | ||
|
|
||
| A scoped token limits access to specified path prefixes and operations. The supported operations are `read`, `list`, `search`, `write`, and `delete`. The service enforces these permissions, including when requests arrive through a mount. | ||
|
|
||
| - A reporting agent might need `read,list` for `/inputs` and `read,list,write` for `/reports`. | ||
| - A reviewer might need only `read,list` for `/reports`. | ||
| - `search` also requires `read` on the scope. | ||
|
|
||
| Scoped tokens cannot generate child tokens or manage token inventory. A scoped token can refresh itself while valid; refresh does not turn it into an owner token or broaden its permissions. | ||
|
|
||
| ## Delegate access to an agent | ||
|
|
||
| On a trusted machine with a locally stored owner token, select the Filesystem and generate a limited token: | ||
|
|
||
| ```bash | ||
| # Capture the one-time token response without printing the secret. | ||
| SCOPED_TOKEN="$(ti fs generate-file-system-scoped-token \ | ||
| --file-system-id "<file-system-id>" \ | ||
| --subject report-agent \ | ||
| --ttl 24h \ | ||
| --allow /workspace:read,list,write \ | ||
| --query fs_token --output text)" | ||
| ``` | ||
|
|
||
| Transfer `SCOPED_TOKEN` through a secret manager. In the agent's environment, inject it as `TI_FS_TOKEN` and provide the Filesystem's region: | ||
|
|
||
| ```bash | ||
| # In the agent environment, these values normally come from secret injection. | ||
| export TI_FS_TOKEN="<scoped-token>" | ||
| export TI_REGION_CODE="aws-us-east-1" | ||
| ti fs list-files --path /workspace | ||
| ``` | ||
|
|
||
| The remote `/workspace` directory must already exist. For a mount, select the allowed subtree with `--remote-path /workspace`. Mounting the root `/` is not appropriate for a token that has access only to `/workspace`. | ||
|
|
||
| ## Understand local selection | ||
|
|
||
| One Filesystem can have multiple remote tokens, but one CLI profile stores at most one selected local token for that Filesystem. Local state is a credential selection, not the authoritative remote token inventory. | ||
|
|
||
| - `--fs-token` takes precedence over `TI_FS_TOKEN` for token-based operations. | ||
| - Without an explicit token, data-access commands use the locally stored credential for `--file-system-id` or `TI_FS_FILE_SYSTEM_ID`. | ||
| - An explicit token contains the Filesystem ID. A clean environment therefore needs only `TI_FS_TOKEN` and `TI_REGION_CODE` for data access. | ||
| - Generating a token does not select it locally unless you pass `--store-locally`. Replacing the local selection does not revoke the previous remote token. | ||
|
|
||
| For token list, enable, disable, and delete commands, an explicitly supplied owner token selects bearer authentication; otherwise the CLI uses TiDB Cloud API keys and requires a Filesystem ID. A scoped token does not gain administrative capability merely because account keys also exist in the profile. | ||
|
|
||
| ## Rotate and revoke credentials | ||
|
|
||
| List token metadata to identify the token you want to manage: | ||
|
|
||
| ```bash | ||
| # Use TiDB Cloud API keys when no explicit FS token is present. | ||
| ti fs list-file-system-tokens --file-system-id "<file-system-id>" --output text | ||
| ``` | ||
|
|
||
| The list does not return token plaintext. Preserve newly generated or refreshed tokens in a secret manager. If you lose an owner token, generate a replacement using TiDB Cloud API keys; do not expect to recover the original secret by listing tokens. | ||
|
|
||
| To retire a token, stop its consumers and use its token ID: | ||
|
|
||
| ```bash | ||
| # Revoke only the retired token, not the Filesystem itself. | ||
| ti fs delete-file-system-token --file-system-id "<file-system-id>" --token-id "<token-id>" | ||
| ``` | ||
|
|
||
| Token changes can take time to propagate through authorization caches. Rotate by distributing and validating a replacement before retiring the old token. Do not treat disabling an owner token as an implicit replacement for reviewing and revoking previously issued scoped tokens. | ||
|
|
||
| > **Warning:** | ||
| > | ||
| > Stop writes and unmount consumers before refreshing, disabling, or deleting their token. The CLI checks known local mounts but cannot discover every remote machine using the secret. Token refresh is non-idempotent: if the request might have succeeded but the response was lost, do not blindly retry it with the old token. | ||
|
|
||
| ## What's next | ||
|
|
||
| - [Share one Filesystem with multiple machines](/tidb-cloud-filesystem/filesystem-sharing.md). | ||
| - [Look up token management commands](/ai/ti/guides/manage-filesystem-tokens.md). | ||
| - [Mount a token-scoped directory](/tidb-cloud-filesystem/filesystem-mount.md). |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Fix the Filesystem TOC links or add the published route mapping. The link checker sends
/tidb-cloud-filesystem/...directly tohttps://docs.pingcap.com/tidb-cloud-filesystem/..., and the tested entry points return 404. The proposed/tidbcloudfs/path also currently returns 404, so use the actual deployed route or add a redirect before changing these links.