BCJSSE cannot select SM2 certificates for RFC 8998 because curveSM2 is missing from ProvX509KeyManager filters

[brightestboy]

Summary

BCJSSE 1.84 and 1.85 advertise support for the RFC 8998 cipher suites:

• TLS_SM4_GCM_SM3
• TLS_SM4_CCM_SM3

However, a TLS 1.3 handshake using curveSM2 and sm2sig_sm3 fails when the standard BCJSSE KeyManagerFactory is used.

The server reports:

handshake_failure(40)
found no selectable cipher suite among the 1 offered:
[TLS_SM4_GCM_SM3]

The problem appears to be that ProvX509KeyManager does not register an EC public-key filter for NamedGroup.curveSM2.

Environment

• JDK: OpenJDK 13.0.1
• bcprov-jdk18on: 1.84
• bcpkix-jdk18on: 1.84
• bctls-jdk18on: 1.84
• Also checked the 1.85 source and the current main branch
• KeyStore: PKCS12 loaded explicitly with the BC provider
• JSSE provider: BouncyCastleJsseProvider explicitly bound to the BC provider

TLS configuration

The following properties are set before initializing BCJSSE:

System.setProperty("jdk.tls.namedGroups", "curveSM2");
System.setProperty("jdk.tls.client.SignatureSchemes", "sm2sig_sm3");
System.setProperty("jdk.tls.server.SignatureSchemes", "sm2sig_sm3");

Both peers are restricted to:

sslSocket.setEnabledProtocols(new String[]{"TLSv1.3"});
sslSocket.setEnabledCipherSuites(new String[]{"TLS_SM4_GCM_SM3"});

Mutual TLS is enabled.

The PKCS12 signing entries contain:

• EC private/public keys on sm2p256v1
• Curve OID: 1.2.156.10197.1.301
• Certificate signature algorithm: SM3withSM2
• Signature algorithm OID: 1.2.156.10197.1.501
• digitalSignature KeyUsage

Reproduction

Providers and KeyManagers are initialized as follows:

BouncyCastleProvider bc = new BouncyCastleProvider();
Security.addProvider(bc);
Security.addProvider(new BouncyCastleJsseProvider(bc));

KeyStore keyStore = KeyStore.getInstance("PKCS12", "BC");
try (InputStream input = Files.newInputStream(keyStorePath)) {
    keyStore.load(input, password);
}

KeyManagerFactory kmf =
    KeyManagerFactory.getInstance("PKIX", "BCJSSE");
kmf.init(keyStore, password);

SSLContext context = SSLContext.getInstance("TLS", "BCJSSE");
context.init(kmf.getKeyManagers(), trustManagers, null);

With the standard BCJSSE KeyManager, the server cannot select credentials and the handshake fails:

found no selectable cipher suite among the 1 offered:
TLS_SM4_GCM_SM3

Source analysis

For TLS 1.3, sm2sig_sm3 is associated with NamedGroup.curveSM2:

sm2sig_sm3(SignatureScheme.sm2sig_sm3, "SM3withSM2", "EC")

JsseUtils.getKeyType13 therefore produces:

EC/curveSM2

ProvTlsServer.selectServerCredentials13 passes that key type to the KeyManager.

However, both ProvX509KeyManager.createFiltersClient() and
ProvX509KeyManager.createFiltersServer() register TLS 1.3 EC filters only for Brainpool and NIST curves:

addECFilter13(filters, NamedGroup.brainpoolP256r1tls13);
addECFilter13(filters, NamedGroup.brainpoolP384r1tls13);
addECFilter13(filters, NamedGroup.brainpoolP512r1tls13);

addECFilter13(filters, NamedGroup.secp256r1);
addECFilter13(filters, NamedGroup.secp384r1);
addECFilter13(filters, NamedGroup.secp521r1);

There is no registration for:

addECFilter13(filters, NamedGroup.curveSM2);

Consequently, when the requested key type is EC/curveSM2, the lookup in FILTERS_SERVER or FILTERS_CLIENT returns null, and the SM2 certificate is rejected regardless of its actual curve and KeyUsage.

Control experiment

I repeated the same mutual TLS handshake using a test-only
BCX509ExtendedKeyManager that directly returns a BCX509Key containing the same SM2 private key and certificate chain.

This bypasses only the ProvX509KeyManager filtering step.

The handshake then succeeds on both sides:

client: TLSv1.3 / TLS_SM4_GCM_SM3
server: TLSv1.3 / TLS_SM4_GCM_SM3

This indicates that:

• SM2 signing works
• curveSM2 key exchange works
• SM3 and SM4 work
• The certificate and private key are usable
• The failure is isolated to the BCJSSE KeyManager selection/filtering path

Possible fix

Would it be appropriate to add the following registration to both
createFiltersClient() and createFiltersServer()?

addECFilter13(filters, NamedGroup.curveSM2);

NamedGroup.getCurveName(NamedGroup.curveSM2) already returns
sm2p256v1, and ECNamedCurveTable can resolve that curve to its standard OID.