docs: note in the 1.1.0 upgrade guide that native Iceberg reads no longer fall back from IRSA

[andygrove]

Which issue does this PR close?

None. It documents a behavior change from #6025 in the 1.1.0 upgrade guide. It was part of the draft release notes PR #6469, which is closing because 1.1.0 now fixes nearly every regression that PR listed.

Rationale for this change

Since #6025, which fixed #6024, Comet's native Iceberg scan on EKS with IRSA takes its S3 credentials only from the web-identity role, when the IRSA environment variables are set and the catalog configures no credentials. If that call fails, it no longer falls back to the node role or Pod Identity as 1.0.0 did. So a cluster whose IRSA setup is broken, and that was reading S3 as the node role without anyone noticing, fails native Iceberg reads after upgrading. The 1.1.0 upgrade guide doesn't mention it.

What changes are included in this PR?

A "Native Iceberg Reads on EKS with IRSA" subsection under "Upgrading to Comet 1.1.0". It describes the change, points to the "EKS / IRSA" section of the S3 credential providers page, and names the two ways back: spark.sql.catalog.<catalog>.s3.comet.credential.webIdentity.enabled=false for a catalog, and spark.comet.scan.icebergNative.enabled=false for a table loaded by path, which has no catalog to set it on.

This should go to branch-1.1 too, because the archived 1.1.0 docs are built from that branch. The commit applies there cleanly.

How are these changes tested?

Docs only. The setting and the page section it names exist on both main and branch-1.1, and prettier passes on the file.

[sunchao]

Summary

• Prior state and problem: The upgrade guide omitted the IRSA credential fallback change introduced by #6025.
• Design approach: Add a focused migration note describing the failure and the available configuration workarounds.
• Correctness / compatibility analysis: The note matches the native credential selection and error propagation. Verified the catalog opt-out and native scan switch against Comet code, and catalog configuration forwarding against Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources. Iceberg's path-loading source supports the distinction described here.
• Key design decisions: Reuse the existing credential-provider documentation and settings. This documentation-only change adds no runtime overhead or implementation complexity.
• Implementation sketch: The full base-relative diff adds one 13-line subsection to docs/source/user-guide/latest/migration-guide.md. There are no stacked prerequisites.
• Behavioral changes worth calling out: This PR changes documentation only. Compared with branch-1.1, it documents existing behavior. The released 1.0.0 source lacks the IRSA takeover, consistent with the intended upgrade warning.
• Suggested improvements: None meeting the requested severity bar. No introduced P1/P2 issues found within this review.

Reviewed full SHA 5c537ed86bbb1fb1344db7a4225116b9ff09f17e against base 3bc2faa934f04799686705b952d59ed32a707dda. The PR is not a draft. Read AGENTS.md and routed through review-comet-pr. No sibling skill applies. The snapshot and live discussion contain no reviews, issue comments, inline comments or review threads.

Exact-head CI: 13 successful checks and 48 skipped checks, with no failures or pending checks. Preflight, including Markdown formatting, and Required Checks passed. Runtime suites and site deployment were skipped.

Validation: git diff --check and focused documentation link, section and configuration checks passed. Reviewed existing credential tests without executing them. No local Rust/Spark build, Sphinx build or live AWS validation was run.