Skip to content

*: fix CSE keyspace GC safepoint compatibility - #5870

Draft
pingyu wants to merge 10 commits into
pingcap:masterfrom
pingyu:compat-essential-v1-gc-check
Draft

*: fix CSE keyspace GC safepoint compatibility#5870
pingyu wants to merge 10 commits into
pingcap:masterfrom
pingyu:compat-essential-v1-gc-check

Conversation

@pingyu

@pingyu pingyu commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #5785

Related PD issue: tikv/pd#11108

Problem Summary:

TiCDC SchemaStore can fail to initialize on TiDB Cloud Essential because PD-CSE and client-go select different keyspace transaction safepoints when GetGCState is unavailable. PD-CSE returns the keyspace-v2 service safepoint, while client-go falls back to the legacy cluster-level etcd key when CSE keyspace-level GC metadata is not recognized. The resulting stale snapshot causes ListDatabases to fail with error 9006 and blocks changefeed creation.

What is changed and how it works?

  • Adopt the TiDB compatibility fix from metaservice: support CSE keyspace-level GC tidb#70322 and the client-go compatibility fix from tikv: support CSE keyspace-level transaction safepoint tikv/client-go#2040 through temporary fork replacements.
  • Align the TiDB nested parser module and Arrow dependency so the upgraded TiDB graph builds while retaining the API-v2 kvproto and PD client line.
  • Add a hidden cdc cli unsafe verify-gc-safepoint --keyspace <name> command that:
    1. Resolves the keyspace and reads its current GC state from PD.
    2. By default, uses GCState.TxnSafePoint as snapshot S, matching the initial metadata snapshot selected by SchemaStore instead of using the latest TSO.
    3. With --legacy-safepoint, reads the keyspace-v2 minimum service safepoint through LegacyClientV2.GetMinServiceSafePointV2 and uses it as S.
    4. Warns that no service safepoint blocks GC advancement during verification.
    5. Opens the keyspace TiKV storage with GC disabled and runs ListDatabases at snapshot S.
    6. Reports the database count or propagates keyspace, GC-state, service-safepoint, and snapshot errors.

Both verifier modes are read-only. They do not create, update, or delete a service safepoint or GC barrier.

Check List

Tests

  • Unit test
  • Manual test

Commands run:

  • make fmt
  • go test ./cmd/cdc/cli -count=1
  • go test -race -shuffle=on ./cmd/cdc/cli -run ^TestVerifyGCSafepoint(Run|Flags)$ -count=1
  • gopls check cmd/cdc/cli/cli_unsafe_verify_gc_safepoint.go cmd/cdc/cli/cli_unsafe_verify_gc_safepoint_test.go
  • make cdc
  • ./bin/cdc cli unsafe verify-gc-safepoint --help
  • ./bin/cdc cli unsafe verify-gc-safepoint --legacy-safepoint --no-confirm

A live Essential v1 cluster was not available for an end-to-end snapshot read.

Questions

Will it cause performance regression or break compatibility?

No performance regression is expected. The command is hidden and only runs when explicitly invoked. The dependency compatibility changes make client-go use the keyspace-scoped safepoint path for CSE keyspaces.

Do you need to update user documentation, design documentation or monitoring documentation?

No. This is a compatibility fix and unsafe diagnostic command for TiDB Cloud Essential.

Release note

Fix changefeed creation failures caused by inconsistent keyspace GC safepoints on TiDB Cloud Essential.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@ti-chi-bot ti-chi-bot Bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed labels Aug 4, 2026
@ti-chi-bot

ti-chi-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot

ti-chi-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign flowbehappy for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0a74d480-da13-48a5-bcae-a992f422065e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 4, 2026
pingyu and others added 2 commits August 4, 2026 11:37
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@pingyu pingyu changed the title cli: add GC safepoint verifier cli: add read-only GC safepoint verifier Aug 4, 2026
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@pingyu pingyu changed the title cli: add read-only GC safepoint verifier *: fix Essential keyspace GC safepoint compatibility Aug 4, 2026
@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels Aug 4, 2026
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@ti-chi-bot ti-chi-bot Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 4, 2026
pingyu added 2 commits August 4, 2026 16:33
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
pdClient, err := pd.NewClientWithContext(
f.ctx, "cdc-factory", pdEndpoints, credential.PDSecurityOption(),
pdopt.WithMaxErrorRetry(maxGetPDClientRetryTimes),
pdopt.WithEnableRouterClient(false),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this used for? Why didn't we need it before?

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Signed-off-by: Ping Yu <yuping@pingcap.com>
@pingyu pingyu changed the title *: fix Essential keyspace GC safepoint compatibility *: fix CSE keyspace GC safepoint compatibility Aug 5, 2026
pingyu and others added 2 commits August 5, 2026 11:50
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Signed-off-by: Ping Yu <yuping@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

schemastore: GC safepoint mismatch blocks changefeed creation on TiDB Cloud Essential

2 participants