Skip to content

[FLINK-39709][s3]Update documentation for native-s3 Implementation#28200

Open
Samrat002 wants to merge 1 commit into
apache:masterfrom
Samrat002:FLINK-39709
Open

[FLINK-39709][s3]Update documentation for native-s3 Implementation#28200
Samrat002 wants to merge 1 commit into
apache:masterfrom
Samrat002:FLINK-39709

Conversation

@Samrat002
Copy link
Copy Markdown
Contributor

What is the purpose of the change

Update missing configuration for native-s3-fs.

Brief change log

Update ReadMe doc

Verifying this change

N/A

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude sonnet 4.6

@flinkbot
Copy link
Copy Markdown
Collaborator

flinkbot commented May 19, 2026

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@gaborgsomogyi
Copy link
Copy Markdown
Contributor

gaborgsomogyi commented May 20, 2026

Issue 1 — MinIO section: auto-detection claim is incorrect (~line 263)

Current text:

MinIO-specific optimizations are applied automatically:
- Path-style access enabled
- Chunked encoding disabled (compatibility)
- Checksum validation disabled (compatibility)

s3.chunked-encoding.enabled and s3.checksum-validation.enabled are plain ConfigOptions with defaultValue(true) (NativeS3FileSystemFactory.java:101–115). There is no code that auto-disables them when a custom endpoint is set — only path-style access is auto-enabled (S3ClientProvider.java:481). A user copying the MinIO example as-is will have both still on, which can cause request failures against MinIO.

Suggested replacement:

When a custom endpoint is configured, path-style access is enabled automatically.
Chunked encoding and checksum validation must be disabled manually for MinIO compatibility:

```yaml
s3.chunked-encoding.enabled: false
s3.checksum-validation.enabled: false

@gaborgsomogyi
Copy link
Copy Markdown
Contributor

Issue 2 — SSE type table: aliases not documented (~line 80)

Current text:

| s3.sse.type | none | Encryption type: `none`, `sse-s3` (AES256), `sse-kms` (AWS KMS) |

S3EncryptionConfig.java:143–147 has explicit case "aes256" and case "aws:kms" branches. The aws:kms value in particular is what Hadoop S3A configs use, so users migrating from Hadoop will set it and silently get no encryption (falls through to none) because it is undocumented.

Suggested replacement:

| s3.sse.type | none | Encryption type: `none`, `sse-s3` or `aes256` (S3-managed keys), `sse-kms` or `aws:kms` (KMS-managed keys) |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants