diff --git a/docs/_docs/monitoring-metrics/new-metrics.adoc b/docs/_docs/monitoring-metrics/new-metrics.adoc index 249bbc8f120f1..a22aefedf2abb 100644 --- a/docs/_docs/monitoring-metrics/new-metrics.adoc +++ b/docs/_docs/monitoring-metrics/new-metrics.adoc @@ -432,6 +432,20 @@ Register name: `communication.tcp.{node_consistent_id}` |sentMessagesToNode | long | Total number of messages sent by current node to the given node. |=== +== Certificate Hot Reload + +Register name: `ssl` + +[cols="2,1,3",opts="header"] +|=== +|Name | Type | Description +|ConsecutiveHotReloadFailures | long | Sum of consecutive automatic hot reload failures across the node's SSL factories. Zero when hot reload is disabled. +|ManualReloadAttempts | long | Factory reload attempts started by the node for manual commands, including errors. Excludes startup loads. +|AutomaticHotReloadAttempts | long | Automatic factory hot reload attempts, including errors. +|=== + +See link:security/ssl-tls#automatic-certificate-hot-reload[Automatic Certificate Hot Reload] for configuration and recovery. + == Communication Connection Pool Register name: `communication.tcp.connectionPool` diff --git a/docs/_docs/security/ssl-tls.adoc b/docs/_docs/security/ssl-tls.adoc index 9ede647e3ed99..35b057062d557 100644 --- a/docs/_docs/security/ssl-tls.adoc +++ b/docs/_docs/security/ssl-tls.adoc @@ -188,6 +188,91 @@ If any factory on a node has another type, the command reports it as unsupported For these factories, follow the same trust-first sequence and restart affected nodes one at a time instead of running reload. If you configured `trustManagers` directly, the command reuses them instead of reloading a trust store. +== Automatic Certificate Hot Reload + +Automatic hot reload lets nodes load replacement certificates without a reload command. +You must supply the replacements: Ignite reads them from the configured source and does not issue certificates. +Hot reload covers the same TCP transports as <>. +It does not update the HTTP REST endpoint or external client applications. + +=== Enable Hot Reload + +Enable hot reload on each configured `SslContextFactory` before starting the node: + +[source,java] +---- +factory.setHotReloadEnabled(true); +---- + +Changing this setting requires a node restart. +The key store must contain a private key with an X.509 certificate chain. +A chain contains the key's certificate and the certificates of its issuers. +Ignite ignores keys without such a chain, as TLS does. +If the key store has no such key, the node logs a warning and skips hot reload. +After a manual reload of a key store with such a key, hot reload works again. + +Before replacing key stores, prepare peers' trust stores as described in <>. +Once every affected peer trusts the replacement certificates, put the replacement key stores in the configured source. +Put only certificates whose validity period has already started. +Ignite installs the replacements without checking their dates, and peers may reject certificates that are not yet valid. +Keep the configured paths, store types and passwords unchanged. +Ignite loads the replacements automatically when hot reload is due; no reload command is needed for this step. + +=== When Hot Reload Runs + +At startup, Ignite checks each factory with hot reload enabled. +After it finishes checking all factories, it waits one hour before checking them again. +A slow certificate source can therefore delay a factory's next check by more than an hour. + +For each factory, hot reload becomes due `hotReloadBeforeExpiryMs` before the earliest certificate expiry in its private key chains. +This includes issuing certificates. The default lead time is 7 days. +The first check at or after that date attempts a reload, including the startup check. +Allow enough time before expiry to supply the replacements, wait through the one-hour interval and load the stores. +Keep the lead time shorter than the lifetime of the replacement certificates, or they will already be due when loaded. +See link:{javadoc_base_url}/org/apache/ignite/ssl/SslContextFactory.html#getHotReloadTime()[getHotReloadTime(), window=_blank] for the calculation. + +If the loaded certificates are still due for hot reload, Ignite records a failed hot reload attempt even though it installed them. +This can happen when the certificates have not changed, only the trust store changed, or the replacements expire within `hotReloadBeforeExpiryMs`. +The loaded context remains in use until a later reload replaces it. + +After a manual reload, the next hourly check uses the loaded certificates. + +New connections use the loaded certificates; established connections remain open. +To confirm a replacement, open a new TLS connection to each affected endpoint and compare its peer certificate with the replacement. + +=== Check Failures and Recover + +If loading the stores or creating the TLS context fails, Ignite keeps the current certificates and retries on the next check, even after they expire. +Keeping expired certificates does not extend their validity, so new connections may fail until a valid replacement is available. +If the replacements appear after hot reload is due, the next retry loads them. + +The node log shows the affected transports. +At `INFO` level it shows when the loaded certificates are due for hot reload, when an attempt starts and when it succeeds. +At `WARN` level it shows why an attempt failed and the earliest time the node can retry automatically. +The `ssl.ConsecutiveHotReloadFailures` metric sums consecutive failed automatic attempts across the node's factories. +Each factory's count resets after a successful manual reload or when a check finds that the loaded certificates are no longer due for hot reload. + +To recover without waiting for the next retry: + +. Correct the replacement stores and run link:tools/control-script#reloading-tls-certificates[`control.sh --ssl reload`]. +. Check the command result and verify a new connection with the replacement certificates. + +*What happens to automatic hot reload.* +If enabled, automatic hot reload continues after manual reload, as described in <>. +A failed automatic attempt does not undo the manual reload. + +*If the command cannot connect.* +First check the command's certificates and trust settings. +If the problem is on the node, put a valid replacement in its source. +If the node log shows automatic retries, wait for the next attempt. +Otherwise, or to load the corrected stores without waiting, restart the affected nodes. + +=== Custom Certificate Sources + +For automatic hot reload, custom sources require a subclass of javadoc:org.apache.ignite.ssl.SslContextFactory[]. +Factories extending only `AbstractSslContextFactory` support manual reload only. +Limit request duration in custom sources: a blocked load delays other hot reloads and manual reload. + == SslContextFactory Properties `SslContextFactory` supports the following properties: @@ -216,6 +301,14 @@ If you configured `trustManagers` directly, the command reuses them instead of r |The protocol for secure transport. https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#sslcontext-algorithms[Supported algorithms,window=_blank]. |`TLS` +|`hotReloadEnabled` +|Enables automatic certificate hot reload. See <>. +|`false` + +|`hotReloadBeforeExpiryMs` +|How long before the certificates expire hot reload becomes due, in milliseconds. See <>. +|`604800000` (7 days) + |`trustStoreFilePath` |The path to the trust store file. |`N/A` diff --git a/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java b/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java index 15306f1d2b916..6babf0267f3d9 100644 --- a/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java +++ b/modules/commons/src/main/java/org/apache/ignite/ssl/AbstractSslContextFactory.java @@ -113,7 +113,7 @@ public void setProtocols(String... protocols) { * @return Initialized SSL context. * @throws SSLException If SSL context could not be created. */ - private SSLContext createSslContext() throws SSLException { + SSLContext createSslContext() throws SSLException { checkParameters(); KeyManager[] keyMgrs = createKeyManagers(); diff --git a/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java b/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java index 5db5c511c7b1e..41529c1c5a331 100644 --- a/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java +++ b/modules/commons/src/main/java/org/apache/ignite/ssl/SslContextFactory.java @@ -24,11 +24,15 @@ import java.security.GeneralSecurityException; import java.security.KeyStore; import java.security.NoSuchAlgorithmException; +import java.security.cert.Certificate; import java.security.cert.CertificateException; import java.security.cert.X509Certificate; import java.util.Arrays; +import java.util.Enumeration; +import java.util.concurrent.TimeUnit; import javax.net.ssl.KeyManager; import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLContext; import javax.net.ssl.SSLException; import javax.net.ssl.TrustManager; import javax.net.ssl.TrustManagerFactory; @@ -61,6 +65,9 @@ public class SslContextFactory extends AbstractSslContextFactory { /** Default key manager / trust manager algorithm. Specifying different trust manager algorithm is not supported. */ public static final String DFLT_KEY_ALGORITHM = System.getProperty("ssl.KeyManagerFactory.algorithm", "SunX509"); + /** Default lead time for certificate hot reload: 7 days. */ + public static final long DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS = TimeUnit.DAYS.toMillis(7); + /** Key manager algorithm. */ protected String keyAlgorithm = DFLT_KEY_ALGORITHM; @@ -85,6 +92,71 @@ public class SslContextFactory extends AbstractSslContextFactory { /** Trust managers. */ protected TrustManager[] trustMgrs; + /** */ + private boolean hotReloadEnabled; + + /** */ + private long hotReloadBeforeExpiryMs = DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS; + + /** */ + private transient volatile long hotReloadTime; + + /** */ + private transient long pendingHotReloadTime; + + /** + * @return Whether automatic certificate hot reload is enabled. Defaults to {@code false}. + */ + public boolean isHotReloadEnabled() { + return hotReloadEnabled; + } + + /** + * @param enabled Whether automatic certificate hot reload is enabled. Changes require a node restart. + */ + public void setHotReloadEnabled(boolean enabled) { + hotReloadEnabled = enabled; + } + + /** + * @return How long before the certificates expire hot reload becomes due, in milliseconds. + * Defaults to {@link #DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS}. + */ + public long getHotReloadBeforeExpiryMs() { + return hotReloadBeforeExpiryMs; + } + + /** + * @param millis How long before the certificates expire hot reload becomes due, in milliseconds. Must be positive. + */ + public void setHotReloadBeforeExpiryMs(long millis) { + A.ensure(millis > 0, "millis > 0"); + + hotReloadBeforeExpiryMs = millis; + } + + /** + * The hot reload time is calculated from the loaded certificates. + * It comes {@link #getHotReloadBeforeExpiryMs()} before the earliest expiry among the chains of the private keys. + * A chain expires when any certificate in it expires. + * Keys without an X.509 certificate chain are ignored, as TLS cannot use them. + * + * @return Hot reload time in milliseconds since the epoch, or {@code 0} if the certificates have not been loaded yet + * or have no private key with an X.509 chain. + */ + public long getHotReloadTime() { + return hotReloadTime; + } + + /** {@inheritDoc} */ + @Override SSLContext createSslContext() throws SSLException { + SSLContext ctx = super.createSslContext(); + + hotReloadTime = pendingHotReloadTime; + + return ctx; + } + /** * Gets key store type used for context creation. * @@ -266,6 +338,8 @@ public static TrustManager getDisabledTrustManager() { keyMgrFactory.init(keyStore, keyStorePwd); + pendingHotReloadTime = hotReloadTime(keyStore); + return keyMgrFactory.getKeyManagers(); } catch (NoSuchAlgorithmException e) { @@ -277,6 +351,32 @@ public static TrustManager getDisabledTrustManager() { } } + /** + * @param store Loaded key store. + * @return Hot reload time of the chain that expires first, or {@code 0} if the store has no private key + * with an X.509 certificate chain. + * @throws GeneralSecurityException If the key store cannot be read. + */ + private long hotReloadTime(KeyStore store) throws GeneralSecurityException { + long earliestExpiry = Long.MAX_VALUE; + + for (Enumeration aliases = store.aliases(); aliases.hasMoreElements(); ) { + String alias = aliases.nextElement(); + + if (!store.isKeyEntry(alias)) + continue; + + Certificate[] chain = store.getCertificateChain(alias); + + if (chain != null && Arrays.stream(chain).allMatch(X509Certificate.class::isInstance)) { + for (Certificate cert : chain) + earliestExpiry = Math.min(earliestExpiry, ((X509Certificate)cert).getNotAfter().getTime()); + } + } + + return earliestExpiry == Long.MAX_VALUE ? 0 : earliestExpiry - hotReloadBeforeExpiryMs; + } + /** {@inheritDoc} */ @Override protected final TrustManager[] createTrustManagers() throws SSLException { if (trustMgrs != null) diff --git a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java index 81809a7fc91c5..254194524ec92 100644 --- a/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java +++ b/modules/control-utility/src/test/java/org/apache/ignite/util/GridCommandHandlerSslReloadTest.java @@ -36,6 +36,7 @@ import org.apache.ignite.internal.IgniteInternalFuture; import org.apache.ignite.internal.TestRecordingCommunicationSpi; import org.apache.ignite.internal.processors.rest.client.message.GridClientPingPacket; +import org.apache.ignite.spi.metric.LongMetric; import org.apache.ignite.ssl.SslContextFactory; import org.apache.ignite.testframework.ListeningTestLogger; import org.apache.ignite.testframework.LogListener; @@ -97,6 +98,7 @@ public void testSharedFactoryReloadedOnceOnServerAndClient() throws Exception { : "communication, client connector, binary REST, discovery"; assertContains(log, testOut.toString(), nodes.get(i).localNode().id() + ": " + transports + ": reloaded"); + assertEquals(1, nodes.get(i).context().metric().registry("ssl").findMetric("ManualReloadAttempts").value()); } } diff --git a/modules/core/pom.xml b/modules/core/pom.xml index bbdf058dea191..47fd6cc45de8b 100644 --- a/modules/core/pom.xml +++ b/modules/core/pom.xml @@ -158,6 +158,13 @@ test + + org.bouncycastle + bcpkix-jdk18on + 1.86 + test + + org.eclipse.jetty.ee11 jetty-ee11-servlets diff --git a/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java b/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java index 5d11403186f4e..177c23e680e65 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java @@ -1243,6 +1243,8 @@ public void start( } } + ctx.ssl().start(ctx); + // Start plugins. for (PluginProvider provider : ctx.plugins().allProviders()) provider.onIgniteStart(); @@ -1722,6 +1724,8 @@ private void stop0(boolean cancel) { GridKernalGateway gw = this.gw.get(); if (stopGuard.compareAndSet(false, true)) { + ctx.ssl().stop(); + // Only one thread is allowed to perform stop sequence. boolean firstStop = false; diff --git a/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java b/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java index 1731563527170..116a8ade16929 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/management/ssl/SslReloadTask.java @@ -19,12 +19,10 @@ import java.util.ArrayList; import java.util.Collection; -import java.util.Collections; import java.util.HashMap; -import java.util.IdentityHashMap; import java.util.List; import java.util.Map; -import java.util.Set; +import java.util.Map.Entry; import java.util.UUID; import javax.cache.configuration.Factory; import javax.net.ssl.SSLContext; @@ -120,7 +118,7 @@ private SslReloadJob(NoArg arg, boolean debug) { /** {@inheritDoc} */ @Override protected String run(NoArg arg) { - Map> factories = ignite.context().ssl().factories(); + List, String>> factories = ignite.context().ssl().factories(); UUID nodeId = ignite.localNode().id(); String initiator = ""; @@ -138,9 +136,9 @@ private SslReloadJob(NoArg arg, boolean debug) { return report; } - for (Map.Entry> entry : factories.entrySet()) { - if (!(entry.getValue() instanceof AbstractSslContextFactory)) { - String report = entry.getKey() + ": unsupported: " + entry.getValue().getClass().getName(); + for (Entry, String> entry : factories) { + if (!(entry.getKey() instanceof AbstractSslContextFactory)) { + String report = entry.getValue() + ": unsupported: " + entry.getKey().getClass().getName(); ignite.log().warning("TLS reload: " + nodeId + ": " + report + initiator); @@ -148,20 +146,13 @@ private SslReloadJob(NoArg arg, boolean debug) { } } - Set> processed = Collections.newSetFromMap(new IdentityHashMap<>()); List reports = new ArrayList<>(); - for (Factory factory : factories.values()) { - if (!processed.add(factory)) - continue; - - String transports = factories.entrySet().stream() - .filter(entry -> entry.getValue() == factory) - .map(Map.Entry::getKey) - .collect(joining(", ")); + for (Entry, String> entry : factories) { + String transports = entry.getValue(); try { - ((AbstractSslContextFactory)factory).init(); + ignite.context().ssl().reload((AbstractSslContextFactory)entry.getKey()); } catch (Exception e) { ignite.log().warning("TLS reload: " + nodeId + ": " + transports + ": failed" + initiator, e); diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java index ae3db72a6640c..106d253159f22 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistry.java @@ -17,24 +17,224 @@ package org.apache.ignite.internal.processors.ssl; -import java.util.LinkedHashMap; +import java.util.Date; +import java.util.List; import java.util.Map; +import java.util.Map.Entry; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; import javax.cache.configuration.Factory; import javax.net.ssl.SSLContext; +import org.apache.ignite.IgniteException; +import org.apache.ignite.IgniteLogger; +import org.apache.ignite.internal.GridKernalContext; +import org.apache.ignite.internal.processors.metric.MetricRegistryImpl; +import org.apache.ignite.logger.NullLogger; +import org.apache.ignite.ssl.AbstractSslContextFactory; +import org.apache.ignite.ssl.SslContextFactory; +import org.jetbrains.annotations.Nullable; + +import static java.util.concurrent.TimeUnit.HOURS; +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static org.apache.ignite.internal.thread.pool.IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor; /** SSL factories used by running transports. */ public class SslFactoryRegistry { /** */ - private final Map> factories = new LinkedHashMap<>(); + private static final long CHECK_PERIOD = HOURS.toMillis(1); + + /** */ + private final List factories = new CopyOnWriteArrayList<>(); + + /** */ + private volatile ScheduledExecutorService executor; + + /** Logs nothing until the registry starts. */ + private volatile IgniteLogger log = new NullLogger(); + + /** */ + private final AtomicLong manualReloadAttempts = new AtomicLong(); + + /** */ + private final AtomicLong automaticHotReloadAttempts = new AtomicLong(); /** */ public synchronized void register(String transport, Factory factory) { - if (factories.putIfAbsent(transport, factory) != null) - throw new IllegalStateException("SSL factory is already registered [transport=" + transport + ']'); + for (int i = 0; i < factories.size(); i++) { + RegisteredFactory registered = factories.get(i); + + if (registered.factory == factory) { + factories.set(i, new RegisteredFactory(factory, registered.transports + ", " + transport)); + + return; + } + } + + factories.add(new RegisteredFactory(factory, transport)); + } + + /** @return Registered factories in registration order, each with its transports joined by commas. */ + public List, String>> factories() { + return factories.stream().map(registered -> Map.entry(registered.factory, registered.transports)).toList(); + } + + /** */ + public synchronized void start(GridKernalContext ctx) { + MetricRegistryImpl metrics = ctx.metric().registry("ssl"); + + metrics.register("ConsecutiveHotReloadFailures", () -> factories.stream().mapToLong(r -> r.failures.get()).sum(), + "Sum of consecutive automatic certificate hot reload failures across SSL factories."); + metrics.register("ManualReloadAttempts", manualReloadAttempts::get, "Number of manual SSL factory reload attempts."); + metrics.register("AutomaticHotReloadAttempts", automaticHotReloadAttempts::get, + "Number of automatic SSL factory hot reload attempts."); + + log = ctx.log(getClass()); + + boolean hotReload = false; + + for (RegisteredFactory registered : factories) { + SslContextFactory sslFactory = registered.hotReloadFactory(); + + if (sslFactory != null) { + registered.logHotReloadTime(sslFactory.getHotReloadTime()); + + hotReload = true; + } + } + + if (!hotReload) + return; + + executor = newSingleThreadScheduledExecutor("ssl-hot-reload", ctx.igniteInstanceName()); + + executor.scheduleWithFixedDelay(() -> factories.forEach(RegisteredFactory::check), 0, CHECK_PERIOD, MILLISECONDS); + } + + /** */ + public synchronized void stop() { + if (executor != null) + executor.shutdownNow(); + } + + /** + * Reloads a registered factory. For a factory with hot reload enabled, waits for its running check and resets its failure + * count. + * + * @param factory Registered factory to reload. + */ + public void reload(AbstractSslContextFactory factory) { + for (RegisteredFactory registered : factories) { + if (registered.factory == factory) + registered.reload(); + } } - /** @return Snapshot of active transport factories. */ - public synchronized Map> factories() { - return new LinkedHashMap<>(factories); + /** Registered factory with its transports and hot reload state. Its check and manual reload hold its monitor. */ + private class RegisteredFactory { + /** */ + private final Factory factory; + + /** */ + private final String transports; + + /** */ + private final AtomicInteger failures = new AtomicInteger(); + + /** */ + private RegisteredFactory(Factory factory, String transports) { + this.factory = factory; + this.transports = transports; + } + + /** @return The factory if hot reload is enabled for it, {@code null} otherwise. */ + @Nullable private SslContextFactory hotReloadFactory() { + return factory instanceof SslContextFactory sslFactory && sslFactory.isHotReloadEnabled() ? sslFactory : null; + } + + /** Reloads the factory if its hot reload time has come, and ends the failure streak if it has not. */ + private synchronized void check() { + SslContextFactory sslFactory = hotReloadFactory(); + + if (sslFactory == null || executor.isShutdown()) + return; + + if (!due(sslFactory.getHotReloadTime())) { + failures.set(0); + + return; + } + + automaticHotReloadAttempts.incrementAndGet(); + + if (log.isInfoEnabled()) + log.info("TLS hot reload started [transports=" + transports + ']'); + + try { + sslFactory.init(); + } + catch (Exception e) { + fail(e); + + return; + } + + long hotReloadTime = sslFactory.getHotReloadTime(); + + if (due(hotReloadTime)) { + fail(new IgniteException("Loaded certificates are still due for hot reload [hotReloadTime=" + + new Date(hotReloadTime) + ']')); + } + else { + failures.set(0); + + if (log.isInfoEnabled()) + log.info("TLS hot reload succeeded [transports=" + transports + ']'); + + logHotReloadTime(hotReloadTime); + } + } + + /** @return Whether the loaded certificates have a hot reload time, and it has come. */ + private static boolean due(long hotReloadTime) { + return hotReloadTime != 0 && hotReloadTime <= System.currentTimeMillis(); + } + + /** */ + private void fail(Exception cause) { + if (executor.isShutdown()) + return; + + failures.incrementAndGet(); + + log.warning("TLS hot reload failed [transports=" + transports + ", nextAttemptNotBefore=" + + new Date(System.currentTimeMillis() + CHECK_PERIOD) + ']', cause); + } + + /** */ + private synchronized void reload() { + manualReloadAttempts.incrementAndGet(); + + ((AbstractSslContextFactory)factory).init(); + + SslContextFactory sslFactory = hotReloadFactory(); + + if (sslFactory != null) { + failures.set(0); + + logHotReloadTime(sslFactory.getHotReloadTime()); + } + } + + /** */ + private void logHotReloadTime(long hotReloadTime) { + if (hotReloadTime == 0) { + log.warning("TLS hot reload will not run: no private key with an X.509 certificate chain was found [transports=" + + transports + ']'); + } + else if (log.isInfoEnabled()) + log.info("TLS hot reload due [transports=" + transports + ", hotReloadTime=" + new Date(hotReloadTime) + ']'); + } } } diff --git a/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java b/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java index fe517fe19f12a..bad785880aa6d 100644 --- a/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java +++ b/modules/core/src/test/java/org/apache/ignite/internal/processors/ssl/SslFactoryRegistryTest.java @@ -17,23 +17,760 @@ package org.apache.ignite.internal.processors.ssl; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.atomic.AtomicInteger; +import javax.net.ssl.SSLException; +import org.apache.ignite.IgniteException; +import org.apache.ignite.internal.GridKernalContext; +import org.apache.ignite.internal.processors.metric.GridMetricManager; +import org.apache.ignite.internal.processors.metric.MetricRegistryImpl; +import org.apache.ignite.internal.thread.pool.IgniteScheduledThreadPoolExecutor; +import org.apache.ignite.internal.util.typedef.X; +import org.apache.ignite.internal.util.typedef.internal.U; +import org.apache.ignite.logger.NullLogger; +import org.apache.ignite.spi.metric.LongMetric; +import org.apache.ignite.ssl.AbstractSslContextFactory; import org.apache.ignite.ssl.SslContextFactory; +import org.apache.ignite.ssl.TestCertificateAuthority; +import org.apache.ignite.testframework.GridTestThread; +import org.apache.ignite.testframework.ListeningTestLogger; +import org.junit.After; +import org.junit.Before; import org.junit.Test; +import org.mockito.MockedStatic; -import static org.junit.Assert.assertSame; +import static java.util.concurrent.TimeUnit.DAYS; +import static java.util.concurrent.TimeUnit.HOURS; +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static java.util.stream.Collectors.toList; +import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD; +import static org.apache.ignite.ssl.TestCertificateAuthority.save; +import static org.apache.ignite.testframework.GridTestUtils.runAsync; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertThrows; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.mockStatic; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; -/** */ +/** + * A manual reload is {@link SslFactoryRegistry#reload}, which the TLS reload command calls. + * An attempt is an automatic hot reload attempt that the hourly check makes. + */ public class SslFactoryRegistryTest { + /** Upper bound for waiting for an event that must happen. */ + private static final long TIMEOUT_MS = 30_000; + + /** */ + private final SslFactoryRegistry registry = new SslFactoryRegistry(); + + /** */ + private final SslContextFactory factory = mock(SslContextFactory.class); + + /** */ + private final GridKernalContext ctx = mock(GridKernalContext.class); + + /** */ + private final ScheduledExecutorService executor = mock(ScheduledExecutorService.class); + + /** */ + private final MetricRegistryImpl metrics = new MetricRegistryImpl("ssl", null, null, null, new NullLogger()); + + /** */ + private MockedStatic executorFactory; + + /** Hourly hot reload check that the registry scheduled. */ + private Runnable check; + + /** */ + private Path dir; + + /** Key store file of the factory under test. */ + private Path keys; + + /** */ + private Path trust; + /** */ + private TestCertificateAuthority authority; + + /** + * Each test gets a registry whose hot reload executor records the hourly check instead of running it. + * Tests run the check themselves. + * The setup also writes the trust store of a test CA to a temporary directory and stubs the node metrics and logger. + */ + @Before + public void setUp() throws Exception { + dir = Files.createTempDirectory("ssl-factory-registry"); + keys = dir.resolve("keys.jks"); + trust = dir.resolve("trust.jks"); + authority = new TestCertificateAuthority("issuer", at(-DAYS.toHours(365)), at(DAYS.toHours(365))); + + save(authority.trustStore(), trust); + + GridMetricManager metricMgr = mock(GridMetricManager.class); + + when(ctx.metric()).thenReturn(metricMgr); + when(metricMgr.registry("ssl")).thenReturn(metrics); + when(ctx.log(SslFactoryRegistry.class)).thenReturn(new NullLogger()); + + when(executor.scheduleWithFixedDelay(any(Runnable.class), anyLong(), anyLong(), eq(MILLISECONDS))).thenAnswer(call -> { + check = call.getArgument(0); + + return mock(ScheduledFuture.class); + }); + + executorFactory = mockStatic(IgniteScheduledThreadPoolExecutor.class); + executorFactory.when(() -> IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor(any(), any())) + .thenReturn(executor); + } + + /** After each test, the registry stops, and the static executor mock and the temporary directory are released. */ + @After + public void tearDown() throws Exception { + try { + registry.stop(); + } + finally { + executorFactory.close(); + U.delete(dir); + } + } + + /** + * Given: communication and discovery register the same factory, with a different factory registered in between. + * When: the registry lists its factories. + * Then: each factory appears once under its transports, in the order of first registration. + */ @Test - public void testDuplicateRegistrationKeepsOriginalFactory() { - SslFactoryRegistry registry = new SslFactoryRegistry(); - SslContextFactory factory = new SslContextFactory(); + public void testFactoriesAreGroupedInRegistrationOrder() { + SslContextFactory clientConnector = new SslContextFactory(); registry.register("communication", factory); + registry.register("client connector", clientConnector); + registry.register("discovery", factory); + + assertEquals(List.of(Map.entry(factory, "communication, discovery"), + Map.entry(clientConnector, "client connector")), registry.factories()); + } + + /** + * Given: a factory with hot reload disabled and valid certificates. + * When: the registry starts. + * Then: no hot reload thread starts, and the failure metric reads zero. + */ + @Test + public void testDisabledFactoryDoesNotStartHotReload() throws Exception { + writeFreshCertificates(keys); + registerTransport("communication", factory(keys)); + registry.start(ctx); + + executorFactory.verifyNoInteractions(); + assertEquals(0, failures()); + } + + /** + * Given: communication and discovery share a factory, and the client connector has an equal factory for the same file. + * When: the key store is replaced and the hourly check runs. + * Then: each factory instance loads the new certificates once. + */ + @Test + public void testEachFactoryInstanceReloadsOnce() throws Exception { + SslContextFactory communication = factory(keys, new KeyStorePathFactory()); + SslContextFactory clientConnector = factory(keys, new KeyStorePathFactory()); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + clientConnector.setHotReloadEnabled(true); + registerTransport("communication", communication); + registerTransport("discovery", communication); + registerTransport("client connector", clientConnector); + registry.start(ctx); + + long prevTime = communication.getHotReloadTime(); + + writeFreshCertificates(keys); + check.run(); + + assertEquals(2, metric("AutomaticHotReloadAttempts")); + assertTrue(communication.getHotReloadTime() > prevTime); + assertEquals(communication.getHotReloadTime(), clientConnector.getHotReloadTime()); + } + + /** + * Given: communication and the client connector use equal but separate factories for the same key store file. + * When: the key store is replaced, and the client connector's factory is reloaded manually. + * Then: the client connector's factory loads the new certificates, and the communication factory keeps its own. + */ + @Test + public void testManualReloadLoadsOnlyItsFactoryInstance() throws Exception { + SslContextFactory communication = factory(keys, new KeyStorePathFactory()); + SslContextFactory clientConnector = factory(keys, new KeyStorePathFactory()); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + clientConnector.setHotReloadEnabled(true); + registerTransport("communication", communication); + registerTransport("client connector", clientConnector); + registry.start(ctx); + + long prevTime = communication.getHotReloadTime(); + + writeFreshCertificates(keys); + registry.reload(clientConnector); + + assertTrue(clientConnector.getHotReloadTime() > prevTime); + assertEquals(prevTime, communication.getHotReloadTime()); + } + + /** + * Given: hot reload is enabled, and the key store holds only a CA certificate. + * When: the registry starts and the hourly check runs, then the operator puts a key store with a private key in place + * and reloads it. + * Then: the registry logs that hot reload will not run, and the check makes no attempt. + * After the reload, the log shows when hot reload is due. + */ + @Test + public void testStoreWithoutPrivateKeyIsReportedUntilKeysAreReloaded() throws Exception { + List logged = captureLog(); + SslContextFactory communication = factory(keys); + + save(authority.trustStore(), keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + check.run(); + + assertEquals(1, logged.size()); + assertTrue(logged.get(0), logged.get(0).startsWith("TLS hot reload will not run")); + assertEquals(0, metric("AutomaticHotReloadAttempts")); + + writeFreshCertificates(keys); + registry.reload(communication); + + assertEquals("TLS hot reload due [transports=communication, hotReloadTime=" + new Date(communication.getHotReloadTime()) + ']', + logged.get(logged.size() - 1)); + } + + /** + * Given: hot reload is enabled for due certificates, and the key store file is deleted after start. + * When: the hourly check runs three times. + * Then: the check runs at start and then every hour, and each run counts one more failed attempt. + */ + @Test + public void testHotReloadIsCheckedEveryHourFromStart() throws Exception { + SslContextFactory communication = factory(keys); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + Files.delete(keys); + + verify(executor).scheduleWithFixedDelay(any(Runnable.class), eq(0L), eq(HOURS.toMillis(1)), eq(MILLISECONDS)); + + for (int failure = 1; failure <= 3; failure++) { + check.run(); + + assertEquals(failure, failures()); + } + } + + /** + * Given: hot reload is enabled for due certificates, and the key store file is deleted after start. + * When: one automatic attempt fails, and the next one loads fresh certificates. + * Then: the log shows when the loaded certificates are due for hot reload, and the start and result of each attempt. + */ + @Test + public void testLogShowsDueTimeStartAndResultOfAttempts() throws Exception { + List logged = captureLog(); + SslContextFactory communication = factory(keys); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + + long due = communication.getHotReloadTime(); + + Files.delete(keys); + check.run(); + writeFreshCertificates(keys); + check.run(); + + assertEquals(List.of("due", "started", "failed", "started", "succeeded", "due"), + logged.stream().map(e -> e.substring("TLS hot reload ".length(), e.indexOf(" ["))).collect(toList())); + assertTrue(logged.toString(), logged.stream().allMatch(e -> e.contains("[transports=communication"))); + assertTrue(logged.get(0), logged.get(0).endsWith("hotReloadTime=" + new Date(due) + ']')); + assertTrue(logged.get(2), logged.get(2).contains("nextAttemptNotBefore=")); + assertTrue(logged.get(5), logged.get(5).endsWith("hotReloadTime=" + new Date(communication.getHotReloadTime()) + ']')); + } + + /** + * Given: hot reload is enabled for due certificates, and the key store file is deleted after start. + * When: one check fails, the source then gets certificates due for hot reload in 23 days, the next check loads them, + * and the check runs again. + * Then: both attempts are counted, the success resets the failure count, and the last check makes no attempt. + */ + @Test + public void testSuccessResetsFailuresAndLaterChecksWait() throws Exception { + SslContextFactory communication = factory(keys); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + Files.delete(keys); + check.run(); + + assertEquals(1, failures()); + assertEquals(1, metric("AutomaticHotReloadAttempts")); + + writeFreshCertificates(keys); + check.run(); + + assertEquals(0, failures()); + assertEquals(2, metric("AutomaticHotReloadAttempts")); + + check.run(); + + assertEquals(2, metric("AutomaticHotReloadAttempts")); + } + + /** + * Given: an automatic attempt has failed because the loaded certificates are still due. + * When: a load outside this registry installs fresh certificates, as another node sharing the factory does, + * and the next check runs. + * Then: the check resets the failure count without loading the factory. + */ + @Test + public void testCheckResetsFailuresWhenFactoryIsNoLongerDue() throws Exception { + SslContextFactory communication = factory(keys); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + check.run(); + + assertEquals(1, failures()); + + writeFreshCertificates(keys); + communication.init(); + check.run(); + + assertEquals(0, failures()); + assertEquals(1, metric("AutomaticHotReloadAttempts")); + } + + /** + * Given: a factory with hot reload enabled is registered, and the registry has not started yet. + * When: the key store is replaced, and a manual reload runs. + * Then: the factory loads the new certificates. + */ + @Test + public void testManualReloadBeforeStartLoadsFactory() throws Exception { + SslContextFactory communication = factory(keys); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + + long prevTime = communication.getHotReloadTime(); + + writeFreshCertificates(keys); + registry.reload(communication); + + assertTrue(communication.getHotReloadTime() > prevTime); + } + + /** + * Given: a factory with hot reload disabled. + * When: a manual reload succeeds, and a second one fails because the key store file is deleted. + * Then: both count as manual attempts, and none counts as automatic. + */ + @Test + public void testManualReloadWithoutHotReloadCountsAppliedAndFailedLoads() throws Exception { + SslContextFactory communication = factory(keys); + + writeFreshCertificates(keys); + registerTransport("communication", communication); + registry.start(ctx); + registry.reload(communication); + + assertEquals(1, metric("ManualReloadAttempts")); + + Files.delete(keys); + + assertThrows(IgniteException.class, () -> registry.reload(communication)); + assertEquals(2, metric("ManualReloadAttempts")); + assertEquals(0, metric("AutomaticHotReloadAttempts")); + } + + /** + * Given: a factory with hot reload enabled and a key store deleted after startup. + * When: a manual reload fails to read the file. + * Then: the manual attempt counter includes the failed load. + */ + @Test + public void testFailedManualReloadWithHotReloadEnabledIsCounted() throws Exception { + SslContextFactory communication = factory(keys); + + writeFreshCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + Files.delete(keys); + + IgniteException err = assertThrows(IgniteException.class, () -> registry.reload(communication)); + + assertTrue(X.hasCause(err, "key store file was not found", SSLException.class)); + assertEquals(1, metric("ManualReloadAttempts")); + } + + /** + * Given: a registered factory that is not an {@link SslContextFactory}. + * When: a manual reload runs. + * Then: the factory is loaded. + */ + @Test + public void testManualReloadSupportsAbstractFactory() throws Exception { + AbstractSslContextFactory factory = mock(AbstractSslContextFactory.class); + + registry.register("client connector", factory); + registry.start(ctx); + registry.reload(factory); + + verify(factory).init(); + } + + /** + * Given: an automatic attempt has failed on a missing key store file. + * When: a manual reload runs while the file is still missing. + * Then: the reload fails, and the failure count stays. + */ + @Test + public void testFailedManualReloadKeepsFailures() throws Exception { + SslContextFactory communication = factory(keys); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + Files.delete(keys); + check.run(); + + IgniteException err = assertThrows(IgniteException.class, () -> registry.reload(communication)); + + assertTrue(X.hasCause(err, "key store file was not found", SSLException.class)); + assertEquals(1, failures()); + } + + /** + * Given: an automatic attempt has started, failed to load a missing key store file, and pauses at its stop check + * before it counts the failure. + * When: the operator puts fresh certificates in place and reloads them during the pause. + * Then: the manual reload waits until the attempt finishes, so the failure count ends at zero. + */ + @Test + public void testManualReloadWaitsForRunningAttempt() throws Exception { + SslContextFactory communication = factory(keys); + CountDownLatch attemptFailed = new CountDownLatch(1); + CountDownLatch resumeAttempt = new CountDownLatch(1); + + writeDueCertificates(keys); + communication.setHotReloadEnabled(true); + registerTransport("communication", communication); + registry.start(ctx); + Files.delete(keys); + + when(executor.isShutdown()).thenAnswer(call -> { + if (metric("AutomaticHotReloadAttempts") > 0) { + attemptFailed.countDown(); + + assertTrue(resumeAttempt.await(TIMEOUT_MS, MILLISECONDS)); + } + + return false; + }); + + GridTestThread attempt = new GridTestThread(check); + GridTestThread manualReload = new GridTestThread(() -> registry.reload(communication)); + + attempt.start(); + + try { + assertTrue(attemptFailed.await(TIMEOUT_MS, MILLISECONDS)); + + writeFreshCertificates(keys); + manualReload.start(); + + long deadline = System.nanoTime() + MILLISECONDS.toNanos(TIMEOUT_MS); + + while (manualReload.getState() != Thread.State.BLOCKED && manualReload.isAlive()) { + assertTrue(System.nanoTime() < deadline); + + Thread.sleep(10); + } + } + finally { + resumeAttempt.countDown(); + attempt.join(TIMEOUT_MS); + manualReload.join(TIMEOUT_MS); + } + + attempt.checkError(); + manualReload.checkError(); + assertEquals(0, failures()); + } + + /** + * Given: communication has fresh certificates, and the client connector has due ones, so its attempt fails. + * When: communication loads due certificates manually, the next check fails for both factories, + * and the client connector then loads fresh certificates manually. + * Then: the metric sums the failures of both factories, and each manual reload clears only its own factory's count. + */ + @Test + public void testManualReloadResetsOnlyItsFactoryFailures() throws Exception { + Path clientKeys = dir.resolve("client-keys.jks"); + SslContextFactory communication = factory(keys); + SslContextFactory clientConnector = factory(clientKeys); + + writeFreshCertificates(keys); + writeDueCertificates(clientKeys); + communication.setHotReloadEnabled(true); + clientConnector.setHotReloadEnabled(true); + registerTransport("communication", communication); + registerTransport("client connector", clientConnector); + registry.start(ctx); + check.run(); + + assertEquals(1, failures()); + + writeDueCertificates(keys); + registry.reload(communication); + + assertEquals(1, failures()); + + check.run(); + + assertEquals(3, failures()); + + writeFreshCertificates(clientKeys); + registry.reload(clientConnector); + + assertEquals(1, failures()); + } + + /** + * Given: an automatic load stays blocked even after it receives an interrupt. + * When: the registry stops. + * Then: it interrupts the load and returns while the load is still blocked. + */ + @Test + public void testStopInterruptsActiveAttemptWithoutWaitingForIt() throws Exception { + executorFactory.when(() -> IgniteScheduledThreadPoolExecutor.newSingleThreadScheduledExecutor(any(), any())) + .thenReturn(new IgniteScheduledThreadPoolExecutor("ssl-hot-reload", null, 1)); + + CountDownLatch attemptStarted = new CountDownLatch(1); + CountDownLatch attemptInterrupted = new CountDownLatch(1); + CountDownLatch allowCompletion = new CountDownLatch(1); + + enableHotReload(factory, due()); + doAnswer(call -> { + attemptStarted.countDown(); + + try { + allowCompletion.await(); + } + catch (InterruptedException e) { + attemptInterrupted.countDown(); + + U.awaitQuiet(allowCompletion); + } + + return null; + }).when(factory).init(); + + registry.register("communication", factory); + + try { + registry.start(ctx); + + assertTrue(attemptStarted.await(TIMEOUT_MS, MILLISECONDS)); + + runAsync(registry::stop).get(TIMEOUT_MS); + + assertTrue(attemptInterrupted.await(TIMEOUT_MS, MILLISECONDS)); + } + finally { + allowCompletion.countDown(); + } + } + + /** + * Given: the node starts stopping during an automatic attempt. + * When: the attempt fails. + * Then: the failure is not counted. + */ + @Test + public void testAttemptFailingDuringStopIsNotCounted() { + enableHotReload(factory, due()); + doAnswer(call -> { + when(executor.isShutdown()).thenReturn(true); + + throw new IgniteException("Load interrupted by node stop"); + }).when(factory).init(); + + registry.register("communication", factory); + registry.start(ctx); + check.run(); + + assertEquals(0, failures()); + } + + /** + * Given: two factories with due certificates. + * When: the node stops while the check loads one of them. + * Then: the check does not start loading the other one. + */ + @Test + public void testStopDuringCheckSkipsRemainingFactories() { + SslContextFactory clientConnector = mock(SslContextFactory.class); + AtomicInteger loads = new AtomicInteger(); + + enableHotReload(factory, due()); + enableHotReload(clientConnector, due()); + + for (SslContextFactory sslFactory : List.of(factory, clientConnector)) { + doAnswer(call -> { + loads.incrementAndGet(); + when(executor.isShutdown()).thenReturn(true); + + return null; + }).when(sslFactory).init(); + } + + registry.register("communication", factory); + registry.register("client connector", clientConnector); + registry.start(ctx); + check.run(); + + assertEquals(1, loads.get()); + } + + /** Makes the registry log to a listener and returns the hot reload lines it logs, in order. */ + private List captureLog() { + ListeningTestLogger log = new ListeningTestLogger(); + List logged = new ArrayList<>(); + + log.registerListener(line -> { + if (line.startsWith("TLS hot reload ")) + logged.add(line); + }); + when(ctx.log(SslFactoryRegistry.class)).thenReturn(log); + + return logged; + } + + /** Loads a file-backed factory before registration, as the TCP transports do during startup. */ + private void registerTransport(String transport, SslContextFactory factory) { + factory.create(); + registry.register(transport, factory); + } + + /** */ + private long failures() { + return metric("ConsecutiveHotReloadFailures"); + } + + /** */ + private long metric(String name) { + return metrics.findMetric(name).value(); + } + + /** + * Makes the factory mock look like a factory with hot reload enabled and loaded certificates with the given date. + * The mock returns {@code true} from {@code isHotReloadEnabled()} and the date from {@code getHotReloadTime()}. + * + * @param factory Factory mock. + * @param hotReloadTime Hot reload time of the loaded certificates, or {@code 0} if they were loaded without hot reload. + */ + private static void enableHotReload(SslContextFactory factory, long hotReloadTime) { + when(factory.isHotReloadEnabled()).thenReturn(true); + when(factory.getHotReloadTime()).thenReturn(hotReloadTime); + } + + /** @return Hot reload time that has already come. */ + private static long due() { + return System.currentTimeMillis(); + } + + /** + * Creates a factory with hot reload disabled that reads its key store from the given file. + * The factory trusts the test CA through the trust store file from the setup. + */ + private SslContextFactory factory(Path keyStore) { + return factory(keyStore, new SslContextFactory()); + } + + /** Points the factory at the supplied key store and the test trust store. */ + private SslContextFactory factory(Path keyStore, SslContextFactory sslFactory) { + sslFactory.setKeyStoreFilePath(keyStore.toString()); + sslFactory.setKeyStorePassword(PASSWORD); + sslFactory.setTrustStoreFilePath(trust.toString()); + sslFactory.setTrustStorePassword(PASSWORD); + + return sslFactory; + } + + /** + * Writes a key store that is already due for hot reload with the default lead time. + * Its certificate expires in 10 hours. + */ + private void writeDueCertificates(Path keyStore) throws Exception { + save(authority.keyStore("node", at(-90), at(10)), keyStore); + } + + /** + * Writes a key store that is due for hot reload in 23 days with the default lead time. + * Its certificate became valid an hour ago and expires in 30 days. + */ + private void writeFreshCertificates(Path keyStore) throws Exception { + save(authority.keyStore("node", at(-1), at(DAYS.toHours(30))), keyStore); + } + + /** */ + private static long at(long hours) { + return System.currentTimeMillis() + HOURS.toMillis(hours); + } + + /** Factories with the same key store path compare equal and cache their own SSL contexts. */ + private static class KeyStorePathFactory extends SslContextFactory { + /** */ + private static final long serialVersionUID = 0; + + /** {@inheritDoc} */ + @Override public boolean equals(Object obj) { + return obj instanceof KeyStorePathFactory other && getKeyStoreFilePath().equals(other.getKeyStoreFilePath()); + } - assertThrows(IllegalStateException.class, () -> registry.register("communication", new SslContextFactory())); - assertSame(factory, registry.factories().get("communication")); + /** {@inheritDoc} */ + @Override public int hashCode() { + return getKeyStoreFilePath().hashCode(); + } } } diff --git a/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateHotReloadTest.java b/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateHotReloadTest.java new file mode 100644 index 0000000000000..f6c952403250b --- /dev/null +++ b/modules/core/src/test/java/org/apache/ignite/ssl/SslCertificateHotReloadTest.java @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.ssl; + +import java.nio.file.Files; +import java.nio.file.Path; +import javax.net.ssl.SSLContext; +import org.apache.ignite.Ignition; +import org.apache.ignite.client.IgniteClient; +import org.apache.ignite.client.SslMode; +import org.apache.ignite.configuration.ClientConfiguration; +import org.apache.ignite.configuration.ClientConnectorConfiguration; +import org.apache.ignite.internal.IgniteEx; +import org.apache.ignite.internal.util.typedef.internal.U; +import org.apache.ignite.spi.metric.LongMetric; +import org.apache.ignite.testframework.ListeningTestLogger; +import org.apache.ignite.testframework.LogListener; +import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest; +import org.junit.Test; + +import static java.util.concurrent.TimeUnit.DAYS; +import static java.util.concurrent.TimeUnit.MINUTES; +import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD; +import static org.apache.ignite.ssl.TestCertificateAuthority.save; +import static org.apache.ignite.testframework.GridTestUtils.waitForCondition; + +/** */ +public class SslCertificateHotReloadTest extends GridCommonAbstractTest { + /** Upper bound for waiting for an event that must happen. */ + private static final long TIMEOUT_MS = 30_000; + + /** + * Given: the factory has loaded due certificates of one CA, and the files then get certificates of another CA. + * When: the node starts, and its hot reload attempt runs. + * Then: the node installs the new certificates, and a client that trusts and presents only them connects. + */ + @Test + public void testNodeHotReloadsCertificatesForNewConnections() throws Exception { + long now = System.currentTimeMillis(); + TestCertificateAuthority ca1 = new TestCertificateAuthority("ca1", now - DAYS.toMillis(1), now + DAYS.toMillis(10)); + TestCertificateAuthority ca2 = new TestCertificateAuthority("ca2", now - DAYS.toMillis(1), now + DAYS.toMillis(10)); + Path dir = Files.createTempDirectory("ssl-hot-reload"); + Path keys = dir.resolve("keys.jks"); + Path trust = dir.resolve("trust.jks"); + + try { + save(ca1.keyStore("node", now - MINUTES.toMillis(110), now + MINUTES.toMillis(10)), keys); + save(ca1.trustStore(), trust); + + SslContextFactory serverFactory = configure(new SslContextFactory(), keys, trust); + + serverFactory.setHotReloadEnabled(true); + + SSLContext initCtx = serverFactory.create(); + + assertTrue("Hot reload must be due at startup", serverFactory.getHotReloadTime() < now); + + save(ca2.keyStore("node", now - MINUTES.toMillis(1), now + DAYS.toMillis(10)), keys); + save(ca2.trustStore(), trust); + + IgniteEx node = startGrid(0, cfg -> { + cfg.setClientConnectorConfiguration(sslClientConnector(serverFactory)); + }); + + assertTrue("The reloaded context was not installed", waitForCondition(() -> serverFactory.create() != initCtx, TIMEOUT_MS)); + + ClientConfiguration clientCfg = new ClientConfiguration() + .setAddresses("127.0.0.1:" + node.context().clientListener().port()) + .setSslMode(SslMode.REQUIRED) + .setSslContextFactory(configure(new SslContextFactory(), keys, trust)); + + try (IgniteClient client = Ignition.startClient(clientCfg)) { + client.cacheNames(); + } + } + finally { + stopAllGrids(); + U.delete(dir); + } + } + + /** + * Given: a node with hot reload enabled starts with expired certificates, so hot reload stays due. + * When: the automatic attempt reloads the same certificates. + * Then: the node counts the failure and logs the transport and the reason. + */ + @Test + public void testNodeReportsHotReloadFailureForExpiredCertificates() throws Exception { + long now = System.currentTimeMillis(); + TestCertificateAuthority ca = new TestCertificateAuthority("ca", now - DAYS.toMillis(1), now + DAYS.toMillis(10)); + Path dir = Files.createTempDirectory("ssl-hot-reload"); + Path keys = dir.resolve("keys.jks"); + Path trust = dir.resolve("trust.jks"); + ListeningTestLogger nodeLog = new ListeningTestLogger(log); + LogListener failure = LogListener.matches("TLS hot reload failed [transports=client connector").build(); + LogListener reason = LogListener.matches("Loaded certificates are still due for hot reload").build(); + + nodeLog.registerListener(failure); + nodeLog.registerListener(reason); + + try { + save(ca.keyStore("node", now - MINUTES.toMillis(120), now - MINUTES.toMillis(1)), keys); + save(ca.trustStore(), trust); + + SslContextFactory serverFactory = configure(new SslContextFactory(), keys, trust); + + serverFactory.setHotReloadEnabled(true); + + IgniteEx node = startGrid(0, cfg -> { + cfg.setGridLogger(nodeLog).setClientConnectorConfiguration(sslClientConnector(serverFactory)); + }); + LongMetric failures = node.context().metric().registry("ssl").findMetric("ConsecutiveHotReloadFailures"); + + assertTrue("The automatic attempt did not fail", waitForCondition(() -> failures.value() > 0, TIMEOUT_MS)); + assertTrue(failure.check(TIMEOUT_MS)); + assertTrue(reason.check(TIMEOUT_MS)); + } + finally { + stopAllGrids(); + U.delete(dir); + } + } + + /** + * Creates a client connector configuration that uses the given factory and requires client certificates. + * The connector uses this factory instead of the node factory. + */ + private static ClientConnectorConfiguration sslClientConnector(SslContextFactory factory) { + return new ClientConnectorConfiguration() + .setSslEnabled(true) + .setSslClientAuth(true) + .setUseIgniteSslContextFactory(false) + .setSslContextFactory(factory); + } + + /** + * Points the factory to the given key and trust store files with the test password. + * It returns the same factory, so a test can configure it inline. + */ + private static SslContextFactory configure(SslContextFactory factory, Path keys, Path trust) { + factory.setKeyStoreFilePath(keys.toString()); + factory.setKeyStorePassword(PASSWORD); + factory.setTrustStoreFilePath(trust.toString()); + factory.setTrustStorePassword(PASSWORD); + + return factory; + } +} diff --git a/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryHotReloadTest.java b/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryHotReloadTest.java new file mode 100644 index 0000000000000..0747a33855b7f --- /dev/null +++ b/modules/core/src/test/java/org/apache/ignite/ssl/SslContextFactoryHotReloadTest.java @@ -0,0 +1,524 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.ssl; + +import java.io.IOException; +import java.net.InetAddress; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.KeyStore; +import java.security.PublicKey; +import java.security.cert.Certificate; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.FutureTask; +import javax.crypto.spec.SecretKeySpec; +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLException; +import javax.net.ssl.SSLHandshakeException; +import javax.net.ssl.SSLServerSocket; +import javax.net.ssl.SSLSocket; +import javax.net.ssl.TrustManager; +import org.apache.ignite.IgniteException; +import org.apache.ignite.testframework.GridTestThread; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import static java.util.concurrent.TimeUnit.DAYS; +import static java.util.concurrent.TimeUnit.HOURS; +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static org.apache.ignite.ssl.SslContextFactory.DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS; +import static org.apache.ignite.ssl.TestCertificateAuthority.PASSWORD; +import static org.apache.ignite.ssl.TestCertificateAuthority.merge; +import static org.apache.ignite.ssl.TestCertificateAuthority.save; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNotSame; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertThrows; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; + +/** */ +public class SslContextFactoryHotReloadTest { + /** */ + @Rule public final TemporaryFolder tmp = new TemporaryFolder(); + + /** Upper bound for waiting for an event that must happen. */ + private static final long TIMEOUT_MS = 30_000; + + /** */ + private final long now = System.currentTimeMillis() / 1_000 * 1_000; + + /** */ + private TestCertificateAuthority authority; + + /** + * Each test gets a test CA whose certificate is valid from a year before to a year after {@code now}. + * The tests issue their certificates from this CA with dates counted from {@code now}. + */ + @Before + public void setUp() throws Exception { + authority = new TestCertificateAuthority("issuer", now - DAYS.toMillis(365), now + DAYS.toMillis(365)); + } + + /** + * Given: a key store with only a CA certificate and the default hot reload setting. + * When: the factory loads the store. + * Then: the load succeeds with hot reload disabled and no scheduled reload date. + */ + @Test + public void testDefaultSettingsAcceptStoreWithoutPrivateKeys() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.trustStore()); + + assertFalse(factory.isHotReloadEnabled()); + assertNotNull(factory.create()); + assertEquals(0, factory.getHotReloadTime()); + } + + /** + * Given: a factory with hot reload enabled and the default lead time, which reads its key store from a file. + * When: it loads a certificate that expires in 10 days. + * Then: hot reload is due 7 days before the expiry. + */ + @Test + public void testHotReloadIsDueSevenDaysBeforeExpiryByDefault() throws Exception { + Path keys = tmp.newFile("keys.jks").toPath(); + + save(authority.keyStore("node", at(-1), at(DAYS.toHours(10))), keys); + + SslContextFactory factory = new SslContextFactory(); + + factory.setKeyStoreFilePath(keys.toString()); + factory.setKeyStorePassword(PASSWORD); + factory.setTrustManagers(SslContextFactory.getDisabledTrustManager()); + factory.setHotReloadEnabled(true); + factory.create(); + + assertEquals(at(DAYS.toHours(3)), factory.getHotReloadTime()); + } + + /** + * Given: a factory with the default lead time. + * When: the application sets a zero or negative lead time. + * Then: the setter rejects it and keeps the default. + */ + @Test + public void testNonPositiveHotReloadBeforeExpiryMsIsRejected() { + SslContextFactory factory = new SslContextFactory(); + + assertThrows(IllegalArgumentException.class, () -> factory.setHotReloadBeforeExpiryMs(0)); + assertThrows(IllegalArgumentException.class, () -> factory.setHotReloadBeforeExpiryMs(-1)); + assertEquals(DFLT_HOT_RELOAD_BEFORE_EXPIRY_MS, factory.getHotReloadBeforeExpiryMs()); + } + + /** + * Given: key stores in which the chain that expires first has 15 hours left. + * In one of them, that chain gets its expiry from an issuer that expires before the key certificate. + * Other entries are a chain that expires later, a secret key and a key with a certificate chain that is not X.509. + * When: the factory loads each key store with hot reload enabled. + * Then: hot reload is due now, 15 hours before the chain that expires first. + */ + @Test + public void testHotReloadTimeComesFromEarliestExpiringChain() throws Exception { + TestCertificateAuthority shortIssuer = new TestCertificateAuthority("short-issuer", now - DAYS.toMillis(365), at(15)); + KeyStore node = authority.keyStore("node", at(-85), at(15)); + KeyStore withUnusableKeys = KeyStore.getInstance("JCEKS"); + + withUnusableKeys.load(null, PASSWORD); + withUnusableKeys.setKeyEntry("node", node.getKey("node", PASSWORD), PASSWORD, node.getCertificateChain("node")); + withUnusableKeys.setKeyEntry("secret", new SecretKeySpec(new byte[16], "AES"), PASSWORD, null); + withUnusableKeys.setKeyEntry("other", node.getKey("node", PASSWORD), PASSWORD, + new Certificate[] {new OtherCertificate(node.getCertificate("node").getPublicKey())}); + + assertHotReloadIsDueNow(node); + assertHotReloadIsDueNow(shortIssuer.keyStore("node", at(-85), at(100))); + assertHotReloadIsDueNow(merge(authority.keyStore("first", at(-85), at(15)), authority.keyStore("second", at(-200), at(100)))); + assertHotReloadIsDueNow(merge(authority.keyStore("first", at(-200), at(100)), authority.keyStore("second", at(-85), at(15)))); + assertHotReloadIsDueNow(withUnusableKeys); + } + + /** + * Given: a factory with hot reload enabled and an in-memory key store that counts its reads. + * When: the factory loads its certificates. + * Then: it reads the key store once, so the hot reload date matches the loaded key managers. + */ + @Test + public void testLoadReadsKeyStoreOnce() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-85), at(15))); + + factory.setHotReloadEnabled(true); + factory.create(); + + assertEquals(1, factory.keyStoreLoads); + } + + /** + * Given: the issuer expired an hour ago, and the key certificate becomes valid in 99 hours. + * When: the factory loads the chain for the first time with hot reload enabled. + * Then: the load succeeds, and hot reload has been due since 15 hours before the issuer expired. + */ + @Test + public void testFirstLoadAcceptsChainOutsideValidityPeriod() throws Exception { + TestCertificateAuthority expiredIssuer = new TestCertificateAuthority("expired-issuer", at(-100), at(-1)); + InMemorySslContextFactory factory = new InMemorySslContextFactory(expiredIssuer.keyStore("node", at(99), at(199))); + + factory.setHotReloadEnabled(true); + factory.create(); + + assertEquals(at(-16), factory.getHotReloadTime()); + } + + /** + * Given: the key store returns a certificate chain for an entry without a private key, as Windows-MY does. + * When: the factory loads the key store. + * Then: the load succeeds, and the factory has no hot reload date. + */ + @Test + public void testStoreWithoutPrivateKeyLoadsWithoutHotReloadDate() throws Exception { + KeyStore store = spy(authority.trustStore()); + String alias = store.aliases().nextElement(); + + doReturn(new Certificate[] {store.getCertificate(alias)}).when(store).getCertificateChain(alias); + + InMemorySslContextFactory factory = new InMemorySslContextFactory(store); + + factory.setHotReloadEnabled(true); + + assertNotNull(factory.create()); + assertEquals(0, factory.getHotReloadTime()); + } + + /** + * Given: the server certificate expires in 10 hours, and the client trusts only the issuer of its replacement. + * When: the server reloads a certificate that expires in 10 or 5 hours. + * Then: the client accepts the server's replacement certificate and completes the TLS handshake. + */ + @Test + public void testReloadAcceptsSameOrEarlierExpiry() throws Exception { + TestCertificateAuthority replacementIssuer = new TestCertificateAuthority("replacement-issuer", at(-24), at(24)); + InMemorySslContextFactory client = new InMemorySslContextFactory(authority.keyStore("client", at(-24), at(24))); + + client.trustStore = replacementIssuer.trustStore(); + client.useTrustStore("memory-trust"); + + SSLContext clientCtx = client.create(); + + for (long expiry : new long[] {at(10), at(5)}) { + InMemorySslContextFactory factory = dueFactory(); + + factory.keyStore = replacementIssuer.keyStore("node", at(-1), expiry); + factory.init(); + + assertTrue(serverAccepts(factory.create(), clientCtx)); + } + } + + /** + * Given: a server keeps its keys, and a client presents a certificate from a CA that the server does not trust yet. + * When: only the trust store gets that CA, and the factory reloads. + * Then: the context loaded before the change rejects the client, and the reloaded context accepts it. + */ + @Test + public void testReloadTrustsCaAddedToTrustStore() throws Exception { + InMemorySslContextFactory factory = dueFactory(); + SSLContext ctx = factory.create(); + TestCertificateAuthority otherIssuer = new TestCertificateAuthority("other-issuer", now - DAYS.toMillis(1), now + DAYS.toMillis(1)); + InMemorySslContextFactory client = new InMemorySslContextFactory(otherIssuer.keyStore("client", at(-1), at(23))); + + client.trustStore = authority.trustStore(); + client.useTrustStore("memory-trust"); + + SSLContext clientCtx = client.create(); + + assertFalse(serverAccepts(ctx, clientCtx)); + + factory.trustStore.setCertificateEntry("other-issuer", otherIssuer.trustStore().getCertificate("CN=other-issuer")); + factory.init(); + + assertTrue(serverAccepts(factory.create(), clientCtx)); + } + + /** + * Given: due certificates, and a replacement that has expired, is not valid yet, or comes from an expired issuer. + * When: the factory reloads. + * Then: the factory installs a new context without checking the validity dates. + */ + @Test + public void testReloadAcceptsCertificatesOutsideValidityPeriod() throws Exception { + TestCertificateAuthority expiredIssuer = new TestCertificateAuthority("expired-issuer", at(-100), at(-1)); + + KeyStore[] replacements = { + authority.keyStore("node", at(-101), at(-1)), + authority.keyStore("node", at(1), at(101)), + expiredIssuer.keyStore("node", at(-1), at(99)) + }; + + for (KeyStore replacement : replacements) { + InMemorySslContextFactory factory = dueFactory(); + SSLContext ctx = factory.create(); + + factory.keyStore = replacement; + factory.init(); + + assertNotSame(ctx, factory.create()); + } + } + + /** + * Given: the factory has loaded due keys and a trust store file. + * When: the trust store disappears during key rotation, then returns at its configured path. + * Then: the failed reload keeps the previous context and date. The retry installs the replacement. + */ + @Test + public void testReloadRecoversAfterFailedTrustStoreLoad() throws Exception { + Path trust = tmp.newFile("trust.jks").toPath(); + + save(authority.trustStore(), trust); + + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-90), at(10))); + + factory.useTrustStore(trust.toString()); + factory.setHotReloadEnabled(true); + + SSLContext ctx = factory.create(); + long hotReloadTime = factory.getHotReloadTime(); + + factory.keyStore = authority.keyStore("node", at(-1), at(99)); + Files.delete(trust); + + IgniteException err = assertThrows(IgniteException.class, factory::init); + + assertTrue(err.getCause().getMessage(), err.getCause().getMessage().contains("key store file was not found")); + assertSame(ctx, factory.create()); + assertEquals(hotReloadTime, factory.getHotReloadTime()); + + save(authority.trustStore(), trust); + factory.init(); + + assertNotSame(ctx, factory.create()); + assertEquals(at(84), factory.getHotReloadTime()); + } + + /** + * Given: a reload of certificates with a new date blocks on the trust store load. + * When: another thread reads {@link SslContextFactory#getHotReloadTime()} during the reload. + * Then: the read returns the current date without waiting for the reload. + */ + @Test + public void testHotReloadTimeRemainsReadableDuringLoad() throws Exception { + InMemorySslContextFactory factory = spy(dueFactory()); + long hotReloadTime = factory.getHotReloadTime(); + + factory.keyStore = authority.keyStore("node", at(-1), at(99)); + + CountDownLatch trustLoadStarted = new CountDownLatch(1); + CountDownLatch finishTrustLoad = new CountDownLatch(1); + + doAnswer(call -> { + trustLoadStarted.countDown(); + + assertTrue(finishTrustLoad.await(TIMEOUT_MS, MILLISECONDS)); + + return call.callRealMethod(); + }).when(factory).loadKeyStore(any(), eq("memory-trust"), any()); + + GridTestThread load = new GridTestThread(factory::init); + + load.start(); + + try { + assertTrue(trustLoadStarted.await(TIMEOUT_MS, MILLISECONDS)); + + assertEquals(hotReloadTime, factory.getHotReloadTime()); + } + finally { + finishTrustLoad.countDown(); + load.join(TIMEOUT_MS); + } + + assertFalse(load.isAlive()); + load.checkError(); + } + + /** + * Creates a factory with hot reload enabled that has loaded certificates due for hot reload 5 hours ago. + * It reads both stores from memory. The certificates expire in 10 hours. + */ + private InMemorySslContextFactory dueFactory() throws Exception { + InMemorySslContextFactory factory = new InMemorySslContextFactory(authority.keyStore("node", at(-90), at(10))); + + factory.trustStore = authority.trustStore(); + factory.useTrustStore("memory-trust"); + factory.setHotReloadEnabled(true); + factory.create(); + + assertEquals(at(-5), factory.getHotReloadTime()); + + return factory; + } + + /** */ + private void assertHotReloadIsDueNow(KeyStore keys) { + InMemorySslContextFactory factory = new InMemorySslContextFactory(keys); + + factory.setHotReloadEnabled(true); + factory.create(); + + assertEquals(now, factory.getHotReloadTime()); + } + + /** + * Connects a client to a server that requires a client certificate and reports whether the server accepted it. + * The server side decides, because with TLS 1.3 the client can finish its part of the handshake first. + */ + private static boolean serverAccepts(SSLContext server, SSLContext client) throws Exception { + InetAddress loopback = InetAddress.getLoopbackAddress(); + + try (SSLServerSocket srvSock = (SSLServerSocket)server.getServerSocketFactory().createServerSocket(0, 1, loopback)) { + srvSock.setNeedClientAuth(true); + srvSock.setSoTimeout((int)TIMEOUT_MS); + + FutureTask accept = new FutureTask<>(() -> { + try (SSLSocket sock = (SSLSocket)srvSock.accept()) { + sock.setSoTimeout((int)TIMEOUT_MS); + sock.startHandshake(); + + return true; + } + catch (SSLHandshakeException e) { + return false; + } + }); + + new GridTestThread(accept).start(); + + try (SSLSocket sock = (SSLSocket)client.getSocketFactory().createSocket(loopback, srvSock.getLocalPort())) { + sock.setSoTimeout((int)TIMEOUT_MS); + sock.startHandshake(); + } + catch (IOException ignored) { + // No-op. + } + + return accept.get(TIMEOUT_MS, MILLISECONDS); + } + } + + /** */ + private long at(long hours) { + return now + HOURS.toMillis(hours); + } + + /** + * A factory that reads its key store from a field instead of a file and counts the reads. + * Tests change the field to put new certificates into the source. + * By default the factory trusts all peers. Tests can configure a trust store from memory or a file. + * Hot reload is due 15 hours before expiry, so the tests count certificate dates in hours. + */ + private static class InMemorySslContextFactory extends SslContextFactory { + /** */ + private static final long serialVersionUID = 0L; + + /** */ + private KeyStore keyStore; + + /** */ + private KeyStore trustStore; + + /** */ + private int keyStoreLoads; + + /** */ + private InMemorySslContextFactory(KeyStore keyStore) { + this.keyStore = keyStore; + + setKeyStoreFilePath("memory"); + setKeyStorePassword(PASSWORD); + setTrustManagers(getDisabledTrustManager()); + setHotReloadBeforeExpiryMs(HOURS.toMillis(15)); + } + + /** Selects the trust store that supplies the factory's trust managers. */ + private void useTrustStore(String path) { + setTrustManagers((TrustManager[])null); + setTrustStoreFilePath(path); + setTrustStorePassword(PASSWORD); + } + + /** {@inheritDoc} */ + @Override protected KeyStore loadKeyStore(String type, String path, char[] pwd) throws SSLException { + if ("memory-trust".equals(path)) + return trustStore; + + if (!"memory".equals(path)) + return super.loadKeyStore(type, path, pwd); + + keyStoreLoads++; + + return keyStore; + } + } + + /** A certificate of a type other than X.509, which TLS key managers skip. */ + private static class OtherCertificate extends Certificate { + /** */ + private static final long serialVersionUID = 0L; + + /** */ + private final PublicKey key; + + /** */ + private OtherCertificate(PublicKey key) { + super("Other"); + + this.key = key; + } + + /** {@inheritDoc} */ + @Override public byte[] getEncoded() { + return new byte[0]; + } + + /** {@inheritDoc} */ + @Override public void verify(PublicKey key) { + // No-op. + } + + /** {@inheritDoc} */ + @Override public void verify(PublicKey key, String sigProvider) { + // No-op. + } + + /** {@inheritDoc} */ + @Override public String toString() { + return "OtherCertificate"; + } + + /** {@inheritDoc} */ + @Override public PublicKey getPublicKey() { + return key; + } + } +} diff --git a/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java b/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java new file mode 100644 index 0000000000000..a8d5205edc0c5 --- /dev/null +++ b/modules/core/src/test/java/org/apache/ignite/ssl/TestCertificateAuthority.java @@ -0,0 +1,154 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ignite.ssl; + +import java.io.OutputStream; +import java.math.BigInteger; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.KeyPair; +import java.security.KeyPairGenerator; +import java.security.KeyStore; +import java.security.cert.X509Certificate; +import java.security.spec.ECGenParameterSpec; +import java.util.Collections; +import java.util.Date; +import javax.security.auth.x500.X500Principal; +import org.bouncycastle.asn1.x509.BasicConstraints; +import org.bouncycastle.asn1.x509.Extension; +import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter; +import org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder; +import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder; + +/** + * Test CA that issues certificates with explicit validity dates. + * Each instance has its own EC key and a self-signed certificate. + * Its key stores hold a leaf certificate signed by the CA, followed by the CA certificate. + */ +public class TestCertificateAuthority { + /** */ + public static final char[] PASSWORD = "123456".toCharArray(); + + /** */ + private final X500Principal name; + + /** */ + private final KeyPair keys = keyPair(); + + /** */ + private final X509Certificate cert; + + /** */ + private long serial; + + /** + * Creates a CA with a self-signed certificate valid between the given dates. + * The name becomes the common name of the CA certificate. + */ + public TestCertificateAuthority(String name, long notBefore, long notAfter) throws Exception { + this.name = new X500Principal("CN=" + name); + cert = certificate(this.name, keys, notBefore, notAfter, true); + } + + /** + * Creates a key store with one private key and its chain under the given alias. + * The chain holds a leaf certificate with the given dates, signed by this CA, followed by the CA certificate. + */ + public KeyStore keyStore(String alias, long notBefore, long notAfter) throws Exception { + KeyPair leafKeys = keyPair(); + X509Certificate leaf = certificate(new X500Principal("CN=" + alias), leafKeys, notBefore, notAfter, false); + KeyStore store = emptyStore(); + + store.setKeyEntry(alias, leafKeys.getPrivate(), PASSWORD, new X509Certificate[] {leaf, cert}); + + return store; + } + + /** + * Creates a trust store that trusts only this CA. + * It holds the CA certificate under the CA name. + */ + public KeyStore trustStore() throws Exception { + KeyStore store = emptyStore(); + + store.setCertificateEntry(name.getName(), cert); + + return store; + } + + /** + * Combines the key entries of several key stores into one key store. + * Aliases must differ, because an entry with the same alias replaces the earlier one. + */ + static KeyStore merge(KeyStore... stores) throws Exception { + KeyStore merged = emptyStore(); + + for (KeyStore store : stores) { + for (String alias : Collections.list(store.aliases())) + merged.setKeyEntry(alias, store.getKey(alias, PASSWORD), PASSWORD, store.getCertificateChain(alias)); + } + + return merged; + } + + /** + * Writes the store to the file with the test password. + * It replaces an existing file, so a test can put new certificates in place of old ones. + */ + public static void save(KeyStore store, Path path) throws Exception { + try (OutputStream out = Files.newOutputStream(path)) { + store.store(out, PASSWORD); + } + } + + /** */ + private X509Certificate certificate(X500Principal subject, KeyPair subjectKeys, long notBefore, long notAfter, boolean ca) + throws Exception { + JcaX509v3CertificateBuilder builder = new JcaX509v3CertificateBuilder( + name, + BigInteger.valueOf(++serial), + new Date(notBefore), + new Date(notAfter), + subject, + subjectKeys.getPublic()); + + if (ca) + builder.addExtension(Extension.basicConstraints, true, new BasicConstraints(true)); + + return new JcaX509CertificateConverter().getCertificate( + builder.build(new JcaContentSignerBuilder("SHA256withECDSA").build(keys.getPrivate()))); + } + + /** */ + private static KeyPair keyPair() throws Exception { + KeyPairGenerator generator = KeyPairGenerator.getInstance("EC"); + + generator.initialize(new ECGenParameterSpec("secp256r1")); + + return generator.generateKeyPair(); + } + + /** */ + private static KeyStore emptyStore() throws Exception { + KeyStore store = KeyStore.getInstance("JKS"); + + store.load(null, PASSWORD); + + return store; + } +} diff --git a/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java b/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java index 7b207d1226a0e..0dcd548b85720 100644 --- a/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java +++ b/modules/core/src/test/java/org/apache/ignite/testsuites/SecurityTestSuite.java @@ -83,6 +83,8 @@ import org.apache.ignite.plugin.security.SecurityBasicPermissionSetSerializationTest; import org.apache.ignite.ssl.MultipleSSLContextsTest; import org.apache.ignite.ssl.SSLContextWrapperTest; +import org.apache.ignite.ssl.SslCertificateHotReloadTest; +import org.apache.ignite.ssl.SslContextFactoryHotReloadTest; import org.apache.ignite.ssl.SslContextFactoryInitTest; import org.apache.ignite.ssl.SslContextReloadTest; import org.apache.ignite.tools.junit.JUnitTeamcityReporter; @@ -149,6 +151,8 @@ MultipleSSLContextsTest.class, SSLContextWrapperTest.class, SslContextFactoryInitTest.class, + SslContextFactoryHotReloadTest.class, + SslCertificateHotReloadTest.class, SslFactoryRegistryTest.class, SslReloadTaskTest.class, SslContextReloadTest.class,