[FLINK-37504] Handle TLS Certificate Renewal by JTaky · Pull Request #27034 · apache/flink

JTaky · 2025-09-23T19:33:38Z

https://issues.apache.org/jira/browse/FLINK-37504

What is the purpose of the change

Implementation of the TLS Renewal for SSL.

Adds functionality to be notified if SSL keys were changed on the Flink container. Different networking mechanisms triggers the reload of new ssl keystore/truststore.
Covers the functionality with appropriate Unit Tests and Integration Tests.

More details are in design doc: https://cwiki.apache.org/confluence/display/FLINK/FLIP-523%3A+Handle+TLS+Certificate+Renewal

Brief change log

Adds new configuration to enable the SSL certificates reload
Adds watch service, which is able to watch certain directories and notify subscribers if those directories were changed
Netty, Pekko and Blob Server components subscribes to the new Watch Service and reloads SSL Context if needed
BlobServer recreates the Socket on the certificate reload, we count on BlobClient retries to handle temporary connectivity issues
Test functionality

Verifying this change

This change added tests and can be verified as follows:

Added integration tests for end-to-end deployments, which ensures that certificates are reloaded, not reloaded, not used according to the provided ssl options
Added unit tests to test the watch service behaviour, with multiple writers, writes, readers. Ensures that proposed mechanism with Dirty state machine works fine. Executed test 100 times locally to eliminate flackiness
If needed if is easy to experiment with given test manually. Running 100 of threads, changes seems to be too slow to be executed regularly
Covered in particular the BlobServer reload mechanism. Ensured that certificate is reloaded if changed. Also run multiple times locally to eliminate flackiness
Deployed the server on the local environment, triggered certificate change
Run given implementation in staging environment for several months. Note: only application mode is used in staging environment

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? New configurations are added

flinkbot · 2025-09-23T19:39:21Z

CI report:

01c4139 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

JTaky · 2025-10-07T09:39:27Z

@flinkbot run azure

JTaky · 2025-10-27T13:45:20Z

@flinkbot run azure

agresch · 2025-11-04T16:45:59Z

    public static ServerSocketFactory createSSLServerSocketFactory(Configuration config)
            throws Exception {
-        SSLContext sslContext = createInternalSSLContext(config, false);
+        SSLContext sslContext = createInternalSSLContext(config, false, false);


I'm not really familiar with Flink code, but can you explain why the server socket wouldn't need to watch for cert reload? I think that would be worth a comment as well.

thanks for the comment, it is really could be confusing. Added a comment to make it a bit more explicit.
TL;DR; BlobServer uses the plain socket, there is not built-in API in java to reload the certificates, so we went with solution to simply recreate the server socket.

My understanding: It could result in temporary unavailability of the Blob Server, but with client retries practically it should lead to the slight increase of the Job deployment time for clusters which relies on the SSL reload mechanism.

JTaky · 2026-02-11T20:27:15Z

@flinkbot run azure

mateczagany

Thank you for this change, I've left some minor comments, otherwise all looks good.

The endpoint verification change looks good to me: internal SSL connections don't have endpoint verification and it's only enabled on the client side. Thank you for rebasing!

mateczagany · 2026-02-24T07:25:54Z

+ * <p>Implementations monitor directories for file modifications and invoke registered {@link
+ * LocalFSWatchServiceListener} callbacks when changes occur.
+ */
+public interface LocalFSDirectoryWatcher {


I think this could be annotated as @Internal

mateczagany · 2026-02-24T07:26:36Z

+ * }
+ * }</pre>
+ */
+public interface LocalFSWatchServiceListener {


This could also be annotated as @Internal

mateczagany · 2026-02-24T09:31:14Z

-                                    .ciphers(ciphers)
-                                    .clientAuth(clientAuth));
+            Path[] pathsToWatch = new Path[certificatePaths.size()];
+            certificatePaths.toArray(pathsToWatch);


I think in any of the modern JVMs, it's unnecessary and slower to use pre-sized arrays, so it's safe to get rid of the unnecessary line:

certificatePaths.toArray(Path[]::new);

or

certificatePaths.toArray(new Path[0]);

mateczagany · 2026-02-24T09:36:45Z

+import java.nio.file.attribute.FileTime;
+import java.util.Optional;
+
+import static org.junit.Assert.assertTrue;


Please try to use JUnit 5 assertions:

import org.junit.jupiter.api.Test; ... import static org.assertj.core.api.Assertions.assertThat; ... assertThat(maybeCertDate) .as("No certificates on rpc port should be accessible when SSL is disabled: " + maybeCertDate) .isEmpty();

mateczagany · 2026-02-24T09:37:03Z

+
+import org.apache.flink.tests.util.flink.ClusterController;
+
+import org.junit.Test;


Please try to use JUnit 5 here too

mateczagany · 2026-02-24T09:37:17Z

+
+import org.apache.flink.tests.util.flink.ClusterController;
+
+import org.junit.Test;


Please try to use JUnit 5 here too

mateczagany · 2026-02-24T09:37:54Z

+import java.time.Duration;
+import java.util.Optional;
+
+import static org.junit.Assert.assertEquals;


Please try to use JUnit 5 here too

JTaky · 2026-02-24T15:24:20Z

@mateczagany thanks a lot for review and feedback

* Unit test LocalFSWatchSingleton/SSLContextLoader * Test BlobStoreSsl to ensure certificates are reloaded

* test Blob server certificate exposure and reload in case of different ssl configs * check files are not read after certificate reload when reload/ssl is disabled

* javadoc * flacky tests * another tests

…rver case

* bump e2e test module version * Fix formatting after rebase

* Remove flacky test, which is already covered by e2e * Increase e2e retries to reduce flackiness (cluster didn't start in time) * Run 20 times locally, all are green

* added for ip v4/v6 * locally test is stable with > 30 consecutive runs

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch 2 times, most recently from 4e21d4b to ba980e3 Compare September 23, 2025 19:38

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch 5 times, most recently from 61f9501 to cc24032 Compare September 30, 2025 12:45

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch 13 times, most recently from c69e821 to 8481eea Compare October 7, 2025 08:05

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch 3 times, most recently from b0392e4 to c6d586a Compare October 7, 2025 11:33

JTaky changed the title ~~[DRAFT][IN_PROGRESS][FLINK-37504]~~ [FLINK-37504] Handle TLS Certificate Renewal Oct 7, 2025

JTaky marked this pull request as ready for review October 7, 2025 16:00

JTaky mentioned this pull request Oct 9, 2025

[DRAFT][IN_PROGRESS][FLINK-37504] JTaky/flink#1

Closed

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch from c6d586a to 32840cf Compare October 10, 2025 06:14

agresch reviewed Nov 4, 2025

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Nov 5, 2025

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch 3 times, most recently from ab0044f to eb1766f Compare February 9, 2026 12:12

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch from eb1766f to 248569b Compare February 11, 2026 15:48

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch 6 times, most recently from 0ccdae5 to 294bdc4 Compare February 23, 2026 11:23

mateczagany reviewed Feb 24, 2026

View reviewed changes

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch from 4d48691 to acd53ce Compare March 9, 2026 09:28

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch 2 times, most recently from 2bb6343 to b5fc855 Compare May 8, 2026 08:21

ashangit and others added 11 commits August 18, 2026 23:29

[FLINK-37504] Add ssl certificate reload mechanism

ed6bcce

[FLINK-37504] Refactor for test compatibility

a105261

* Unit test LocalFSWatchSingleton/SSLContextLoader * Test BlobStoreSsl to ensure certificates are reloaded

[FLINK-37504] Develop an Integration test for SSL certificate reload

a126021

* test Blob server certificate exposure and reload in case of different ssl configs * check files are not read after certificate reload when reload/ssl is disabled

Fix CI

e3f504d

* javadoc * flacky tests * another tests

Add comment which explains why we do not reload ssl context in BLobSe…

37fc8d7

…rver case

Rebase and resolve conflicts with master

e74d7eb

* bump e2e test module version * Fix formatting after rebase

Reduce test flackiness for e2e ith retryies

bc2240d

* Remove flacky test, which is already covered by e2e * Increase e2e retries to reduce flackiness (cluster didn't start in time) * Run 20 times locally, all are green

Add localhost/dns name to generated ceritifates for e2e

fe186c7

* added for ip v4/v6 * locally test is stable with > 30 consecutive runs

Add modern encryptions JVM algo

73a3277

Address PR comments: junit5 + @internal + array

6e969dc

Bump snapshot version for parent

01c4139

JTaky force-pushed the oleksandr.nitavskyi/flink_37504 branch from b5fc855 to 01c4139 Compare August 19, 2026 09:46


		import org.apache.flink.tests.util.flink.ClusterController;

		import org.junit.Test;

Uh oh!

Conversation

JTaky commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

JTaky commented Oct 7, 2025

Uh oh!

JTaky commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JTaky commented Feb 11, 2026

Uh oh!

mateczagany left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JTaky commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JTaky commented Sep 23, 2025 •

edited

Loading

flinkbot commented Sep 23, 2025 •

edited

Loading

mateczagany left a comment •

edited

Loading