columnar: pass gcs_auth_mode through to the hub's DFS client

[iosmanthus]

What problem does this PR solve?

Issue Number: close #11095

Problem Summary:

Follow-up to #11096, which bumped kvengine for GCP support. No issue filed.

On a GCS-backed deployment the columnar read node starts, registers with PD as
engine=tiflash_compute, and decrypts its master key through GCP KMS — but no
query ever returns. EXPLAIN with tidb_enforce_mpp=1 produces a full
mpp[tiflash] plan, and executing it hangs forever.

build_dfs() built the hub's S3 client with S3Fs::new(), which hardcodes an
empty gcs_auth_mode:

// kvengine/src/dfs/s3.rs
fn use_gcs_oauth(endpoint: &str, key_id: &str, auth_mode: &str) -> bool {
    key_id.is_empty()
        && CloudProvider::from_endpoint(endpoint) == CloudProvider::Gcs
        && auth_mode.eq_ignore_ascii_case(GCS_AUTH_MODE_OAUTH)
}

The first two conditions hold; only auth_mode is missing. So
GcsOAuthDispatcher is never installed and the client falls back to the AWS
credential chain. On GKE that chain ends at EC2 IMDS, which answers 404, and
every columnar file read retries forever:

[cloud_helper.rs:867] ["13628:19 request_snapshot_from_leader, uri: ...  cost: 0.003 ...
                        columnar_creates { id: 469127498656382994 ... }"]
[WARN] ["provider failed to provide credentials provider=Ec2InstanceMetadata
        error=... StatusCode(404) ..."]
[WARN] [util.rs:121] ["aws request fails"] [context=get_cred_on_premise] [retry?=true]
[WARN] [s3.rs:1294] ["retry read file 469127498656382994.col,
                      error Credentials(CredentialsError { message: \"Timeout getting credentials\" })"]

Note the snapshot fetch from TiKV succeeds — the hub gets as far as knowing
which columnar files to read, and only the object store read fails.

What is changed and how it works?

The value was never missing from the config. run() calls
Config::override_from_env() before build_dfs(), and that already reads
DFS_GCS_AUTH_MODE into Config::gcs_auth_mode. Only the handoff to the client
dropped it.

Both call sites in build_dfs() now go through S3Fs::new_from_config(), which
forwards gcs_auth_mode. That is the same constructor TiKV uses, which is why
tikv-server reads the same bucket with the same environment without hitting
this.

read_only and max_read_throughput are pinned to the values S3Fs::new()
hardcoded (true and 0), so this change stays limited to credentials.
Whether a configured dfs.max-read-throughput should apply to the hub is a
separate decision, deliberately not made here.

The env-overridden branch is fixed the same way. It is currently unreachable —
its condition needs dfs_conf.s3_bucket and s3_endpoint to be empty, but
override_from_env() has already populated them by then — but leaving the two
branches inconsistent would just re-plant the bug if that condition ever
changes. Removing the dead branch is out of scope for this PR.

columnar: pass gcs_auth_mode through to the hub's DFS client

build_dfs() built its S3 client with S3Fs::new(), which hardcodes an empty
gcs_auth_mode, so use_gcs_oauth() was false against a GCS endpoint and the
client fell back to the AWS credential chain. On GKE that chain ends at EC2
IMDS, so every columnar file read retried "Timeout getting credentials"
forever and MPP queries hung. Go through S3Fs::new_from_config() instead,
which forwards the gcs_auth_mode that Config::override_from_env() already
read from DFS_GCS_AUTH_MODE.

Check List

Tests

• Manual test (add detailed scripts or steps below)

Reproduced and verified on a GCP serverless cell (GKE, us-east4, TiKV and
TiFlash both reading one GCS bucket with DFS_GCS_AUTH_MODE=oauth).

Before this change, with an image built from #11096:

1. ALTER TABLE t SET TIFLASH REPLICA 1 — information_schema.tiflash_replica
   reports AVAILABLE=1 within ~15s.
2. SET SESSION tidb_enforce_mpp=1; EXPLAIN SELECT g, COUNT(*), SUM(v) FROM t GROUP BY g ORDER BY g; — full mpp[tiflash] plan.
3. Running the same query never returns; the read node logs the credential
   retry loop quoted above, and keeps retrying after the client disconnects.

Ruled out as causes:

• The pod can fetch a Google access token from the metadata server, as the
  correct Workload Identity service account.
• TiKV in the same namespace, same bucket, same DFS_* environment, logs zero
  credential errors over the same period.
• The TiFlash and TiKV container environments were compared variable by
  variable; the DFS_* and CSE_* sets are identical.

Local cargo check --locked --manifest-path hub-runtime/Cargo.toml passes
against the contrib/cloud-storage-engine revision master pins
(9d4471a5). A rebuilt image has not yet been run on the cell — I will follow
up here once it has.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

None on AWS or on any deployment with a static DFS_S3_KEY_ID: when
gcs_auth_mode is not "oauth", or the endpoint is not GCS, or a key id is set,
use_gcs_oauth() is false and the client is built exactly as before.

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Bug Fixes
  • Improved connectivity to S3-compatible and Google Cloud Storage endpoints by correctly honoring configured authentication settings.
  • Ensured storage access uses read-only mode, preventing unintended write operations.
  • Environment-based storage configuration is now applied consistently, including endpoint, bucket, region, credentials, and path settings.

[ti-chi-bot]

Hi @iosmanthus. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: c170a8a1-3a17-4a53-93d6-f5b5fe996c1c

📥 Commits

Reviewing files that changed from the base of the PR and between dfddd06 and a050c37.

📒 Files selected for processing (1)

• contrib/tiflash-columnar-hub/hub-runtime/src/run.rs

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

The DFS builder now creates S3 clients from cloned, read-only configurations. Environment overrides remain supported, and gcs_auth_mode is preserved for GCS endpoints.

Changes

DFS read-only configuration

Layer / File(s)	Summary
Configure and build read-only DFS contrib/tiflash-columnar-hub/hub-runtime/src/run.rs	build_dfs applies DFS_* environment values before client construction. as_read_only_dfs_config sets read-only mode, disables read-throughput limits, and preserves gcs_auth_mode.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: jayson-huang

Merge Risk: ⚪ Minimal · up to a050c

The DFS client configuration preserves GCS OAuth settings in both construction paths, with no actionable merge-blocking risk identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: passing gcs_auth_mode to the columnar hub's DFS client.
Description check	✅ Passed	The description covers the problem, implementation, manual test steps, side effects, documentation impact, and release note. It also reports the local check result and clearly notes that rebuilt-image…

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{I hop through configs, neat and bright,
Read-only paths now set just right.
GCS keeps its chosen key,
S3 builds carefully.
A carrot cheers the passing byte.}

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

@coderabbitai[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: coderabbitai[bot], JaySon-Huang, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,yongman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-09-17 01:59:43.555992659 +0000 UTC m=+256829.493650253: ☑️ agreed by JaySon-Huang.
• 2026-09-17 06:14:38.045985539 +0000 UTC m=+272123.983643133: ☑️ agreed by yongman.

[yongman]

/cherry-pick release-nextgen-202603

[ti-chi-bot]

@yongman: once the present PR merges, I will cherry-pick it on top of release-nextgen-202603 in the new PR and assign it to you.

Details

In response to this:

/cherry-pick release-nextgen-202603

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[yongman]

/test pull-unit-test

[iosmanthus]

/retest

[ti-chi-bot]

@iosmanthus: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yongman]

/retest-required

[JaySon-Huang]

/test pull-unit-test

[ti-chi-bot]

@iosmanthus: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-sanitizer-asan	a050c37	link	false	/test pull-sanitizer-asan
pull-integration-test	a050c37	link	true	/test pull-integration-test
pull-unit-test	a050c37	link	true	/test pull-unit-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[JaySon-Huang]

/test pull-unit-test

[yongman]

/retest-required