Skip to content

[FLINK-39701][s3]Fix Concurrency Annotation in native-s3-fs#28190

Open
Samrat002 wants to merge 1 commit into
apache:masterfrom
Samrat002:concurency-fix
Open

[FLINK-39701][s3]Fix Concurrency Annotation in native-s3-fs#28190
Samrat002 wants to merge 1 commit into
apache:masterfrom
Samrat002:concurency-fix

Conversation

@Samrat002
Copy link
Copy Markdown
Contributor

@Samrat002 Samrat002 commented May 18, 2026

What is the purpose of the change

This PR adds missing thread-safety annotations across flink-s3-fs-native to aid future maintainability.

Brief change log

  • NativeS3FileSystemFactory: flinkConfig and bucketConfigProvider are mutable fields written in configure() and read in create(), potentially from different threads with no
    happens-before guarantee. Added volatile to both fields and stashed bucketConfigProvider into a local variable before use (prevents a TOCTOU race between the null-check and
    dereference on a volatile read).
  • NativeS3InputStream: closed was annotated both @GuardedBy("lock") and volatile — a contradiction, since volatile implies lock-free access while @GuardedBy requires the
    lock. All reads and writes of closed are already inside the ReentrantLock, so volatile was misleading and unnecessary. Removed volatile.

Annotation hygiene:

  • Added @threadsafe class annotation to: NativeS3FileSystem, S3ClientProvider, NativeS3InputStream, NativeS3OutputStream, NativeS3BulkCopyHelper, NativeS3FileSystemFactory,
    NativeS3RecoverableWriter, NativeS3ObjectOperations, NativeS3Committer, NativeS3DelegationTokenReceiver, NativeS3DelegationTokenProvider,
    DynamicTemporaryAWSCredentialsProvider
  • Added @NotThreadSafe to S3ClientProvider.Builder (mutable builder, standard pattern)
  • Added @GuardedBy("lock") to NativeS3OutputStream.position and fileUploaded
  • Added @immutable to four inner result classes in NativeS3ObjectOperations: UploadPartResult, PutObjectResult, CompleteMultipartUploadResult, ObjectMetadata

Verifying this change

Existing UT and e2e test over the large cluster

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: yes

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? no

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

@flinkbot
Copy link
Copy Markdown
Collaborator

flinkbot commented May 18, 2026

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Comment on lines +303 to +304
@Nullable private volatile Configuration flinkConfig;
@Nullable private volatile BucketConfigProvider bucketConfigProvider;
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without volatile, the JVM can legally:

  • Cache flinkConfig in a CPU register on thread B's core
  • Never re-read it from main memory
  • Thread B sees null even though thread A wrote a real Configuration

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label May 18, 2026
@Samrat002
Copy link
Copy Markdown
Contributor Author

@gaborgsomogyi PTAL

import javax.annotation.concurrent.ThreadSafe;

@Internal
@ThreadSafe
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gaborgsomogyi :
if we say thread safe , from what thread we are protecting against ?
if not then we dont care .

If there are multiple thread involved. list them !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants