Skip to content

IGNITE-29211 Hot-reload TLS certificates automatically before expiry - #13786

Open
anton-vinogradov wants to merge 413 commits into
apache:masterfrom
anton-vinogradov:ignite-29211
Open

anton-vinogradov wants to merge 413 commits into
apache:masterfrom
anton-vinogradov:ignite-29211

Conversation

@anton-vinogradov

@anton-vinogradov anton-vinogradov commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

IGNITE-29211 adds optional automatic TLS certificate hot reload. Nodes attempt to read replacements from the configured source before expiry. Ignite does not issue certificates.

Behavior

  • Hot reload is disabled by default. SslContextFactory calculates hot reload dates from the key store actually loaded, including custom sources. Hot reload becomes due hotReloadBeforeExpiryMs before the earliest certificate expiry, 7 days by default; the next hourly check starts the attempt. Keys without an X.509 chain are ignored, as TLS ignores them. Without such a key, the node logs a warning and skips hot reload until a reload brings one.
  • Manual and automatic replacements use the same init() path. The factory does not reject its own certificates because they have expired or are not yet valid, including at startup. Replacements may have equal, earlier or later expiry, including unchanged certificates and trust-store-only changes. Peers apply their configured trust checks during TLS connections.
  • Loading or context creation failures preserve the current context and hot reload date. A successful load installs the context for new connections; established connections remain open. The manual command reports reloaded, including when the material has not changed.
  • At startup, each node checks its enabled factories. One hour after it finishes checking all of them, it starts the next check. A manual reload of an enabled factory runs in the caller thread and waits for that factory's running check. If an automatic load leaves hot reload due, the attempt counts as a failure, and the next hourly check retries. The installed context remains in use. A successful manual reload clears that factory's failures, and so does a check that finds the factory no longer due.
  • ssl.ConsecutiveHotReloadFailures sums consecutive automatic failures on the node. ssl.ManualReloadAttempts and ssl.AutomaticHotReloadAttempts count per-factory calls, including errors. They exclude startup and direct application calls.
  • The log shows when the loaded certificates are due for hot reload and when each attempt starts or succeeds. On failure, it gives the reason and the earliest time the node can retry automatically.
  • Shutdown stops the hourly check and interrupts a running check without waiting for it. That check may still finish its load after the node stops, but it does not start loads of other factories.

The existing final void init() contract is preserved. The factory publishes the hot reload date only after successful context creation. During loading, readers get the previous date without waiting. The registry checks whether loading is due, calls init() and evaluates the resulting hot reload date. Nodes sharing a factory check it separately and may load it sequentially.

Hot reload covers TCP transports. HTTPS remains in IGNITE-29212. Factories extending only AbstractSslContextFactory retain their own manual loading behavior.

Tests cover certificate dates, loading expired and not yet valid certificates, peer trust checks, equal or earlier expiry, trust-only changes, failure recovery, scheduling, shutdown and metrics. They also cover the reload command and replacement on a live TLS connection.

Depends on IGNITE-29210, merged into Apache master.

# Conflicts:
#	modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
# Conflicts:
#	docs/_docs/security/ssl-tls.adoc
#	modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java
#	modules/core/src/main/java/org/apache/ignite/internal/ssl/SslContextProvider.java
@ignite-pr-checker

ignite-pr-checker commented Nov 16, 2026 •

Copy link
Copy Markdown

@anton-vinogradov Ignite PR Checker verdict · RunAll build 9499819 · tested a5cfeb9 · 149 suites ran, 0 reused

⚠️ This run doesn't cover the PR fully:

  • 2 suites have no reliable result (non-zero exit code; execution timeout · non-zero exit code)

Everything below is what it did manage to say.

⚠️ Broken suites (no reliable result):

  • [Check Code Style Ducktests]: non-zero exit code
  • Thin client: Node.js: execution timeout · non-zero exit code

🔎 No blockers found — but the run above can't prove the PR is clean. 113 failing tests are not blamed on this PR. Re-run after you handle what is listed above.

What each label means and what to do: verdict glossary.

♻️ Settled after 2 auto re-run waves: #1 — 5 suites that failed mid-run; #2 — 3 broken suites.

🔁 Superseded by the newer RunAll 9502804 — its verdict is on the checker's page.

anton-vinogradov and others added 25 commits November 16, 2026 13:28
…urly

Hot reload starts hotReloadBeforeExpiry before the earliest certificate expiry, 7 days by default,
instead of at 15% of the chain lifetime. Failed attempts retry every hour instead of with a growing delay.
Keys without an X.509 chain are skipped, as the JDK key managers skip them.
The log shows when each attempt starts and succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test checks that the first load accepts a chain outside its validity period.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The setting becomes hotReloadBeforeExpiryMs, and its default becomes DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If the key store has no private key with an X.509 chain, the factory loads it without a hot reload date.
The registry logs a warning and schedules nothing until a reload brings such a key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only a certificate that expires exactly 7 days after 1 January 1970 gives a zero date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop cancels queued work and interrupts the running attempt. Nothing breaks if that attempt finishes
after the node stops: rescheduling is rejected and factory loads are serialized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One factory is often registered for several transports. Grouping first visits each factory once
and builds its transport list in one pass, so start no longer needs to skip factories it has seen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ttempt

With a fixed one-hour retry, one periodic check covers all factories: it runs at startup and then
every hour, and reloads each factory whose hot reload time has come. Pending attempts, rescheduling
after manual reloads and rejected-schedule handling are gone. The registry calls init() itself, so
SslContextFactory.hotReload() is removed. The hot reload date is taken from the certificate chains
directly, and the logger lives in the registry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without per-attempt scheduling, the hot reload thread no longer needs to own manual reloads.
A manual reload now runs in the caller thread and waits for the factory's running check on
its monitor, so the failure count still matches the last load. Queued reloads, their
cancellation on stop and the rejection before start are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reload command and the hot reload start both grouped transports by factory on their own.
SslFactoryRegistry.factories() now returns each factory once, under its transports joined
by commas, in registration order, and both callers use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SslFactoryRegistry.factories() returns each factory with its transports joined by commas,
in registration order, instead of keying factories by the joined transport names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… they hold

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…due, store grouped factories

- The hourly check skips the remaining factories once the node stops.
- A check that finds a factory no longer due ends its failure streak, for example after
  another node sharing the factory loaded fresh certificates.
- The registry stores each factory with its transports and no longer rejects a repeated
  transport registration: every transport registers once per node start.
- The trust store test checks a TLS handshake with a client from the added CA.
- The docs say that an attempt can start up to an hour after hot reload becomes due.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SslContextFactory computes the hot reload date on every load, so the registry no longer
  reloads factories at start: every transport loads its factory before registering it.
- createKeyManagers() keeps the master structure and adds only the date calculation.
- The hourly check no longer clears the operation context: it is scheduled once, at start.
- The lead time is an ordinary setting, like the other factory settings.
- Tests of removed paths and duplicate failure stages are gone; the reload command test is
  back to its master version.
- The docs no longer restate unchanged manual reload behavior or the metric descriptions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rence

A node has a few TLS factories, so a manual reload finds its wrapper by scanning the list,
the same way registration finds a factory. HotReload is renamed to SslFactoryWrapper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each SslFactoryWrapper holds its factory, transports and hot reload state. Start marks the
wrappers whose factories have hot reload enabled, the hourly check skips the others, and a
manual reload finds its wrapper by reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…field

The wrapper checks whether its factory is an SslContextFactory with hot reload enabled when it
needs to. The registry logs nothing until it starts, so a manual reload that arrives during node
startup works without a logger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The class does not wrap the Factory interface; it is the registry's entry for a factory with
its transports and hot reload state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ignite-pr-checker

Copy link
Copy Markdown

@anton-vinogradov Ignite PR Checker verdict · RunAll build 9502804 · tested 541872e · 149 suites ran, 0 reused

⚠️ This run doesn't cover the PR fully:

  • 2 suites have no reliable result (2× execution timeout · non-zero exit code)

Everything below is what it did manage to say.

⚠️ Broken suites (no reliable result):

  • execution timeout · non-zero exit code (2 suites): Thin client: Node.js, [Check Code Style Ducktests]

🔎 No blockers found — but the run above can't prove the PR is clean. 121 failing tests are not blamed on this PR. Re-run after you handle what is listed above.

What each label means and what to do: verdict glossary.

♻️ Settled after 2 auto re-run waves: #1 — 7 suites that failed mid-run; #2 — 2 broken suites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants