Repository navigation
TLS server with an EC key fails on 0.19.x (chooseAlias doesn't match BCJSSE key types) #383
Description
On 0.19.3, a TLS server with an EC certificate fails every handshake; RSA works. 0.16.2 works with both.
require "openssl"
require "socket"
key = ARGV[0] == "rsa" ? OpenSSL::PKey::RSA.new(2048) : OpenSSL::PKey::EC.generate("prime256v1")
cert = OpenSSL::X509::Certificate.new
cert.version = 2
cert.serial = 1
cert.subject = cert.issuer = OpenSSL::X509::Name.parse("/CN=localhost")
cert.public_key = key
cert.not_before = Time.now - 60
cert.not_after = Time.now + 3600
cert.sign(key, "SHA256")
ctx = OpenSSL::SSL::SSLContext.new(:TLSv1_2)
ctx.cert = cert
ctx.key = key
server = OpenSSL::SSL::SSLServer.new(TCPServer.new("127.0.0.1", 0), ctx)
Thread.new { c = server.accept; c.puts(c.gets); c.close }
client = OpenSSL::SSL::SSLSocket.new(TCPSocket.new("127.0.0.1", server.to_io.addr[1]),
OpenSSL::SSL::SSLContext.new(:TLSv1_2))
client.connect
client.puts("ping")
client.gets
puts "OK #{client.ssl_version}"jruby repro.rb rsa prints OK TLSv1.2; jruby repro.rb ec fails (JRuby 9.4.15.0, OpenJDK 21):
OpenSSL::SSL::SSLError: org.bouncycastle.tls.TlsFatalAlert: handshake_failure(40); [server #2 @…] found no selectable cipher suite among …
OpenSSL::SSL::SSLError: Socket closed
The same happens with TLS 1.3 and with P-384 keys.
Cause: 0.19 prefers BCJSSE (SecurityHelper.java#L556-L569). KeyManagerImpl.chooseAlias requires an exact key-type match (SSLContext.java#L1242-L1252), but BCJSSE asks for ECDHE_ECDSA (TLS 1.2) or EC/secp256r1 etc. (TLS 1.3), never plain EC, so no credential is found.
Normalising the key type in chooseAlias (strip /<group> and check the key's curve matches it; map ECDHE_ECDSA to EC, ECDHE_RSA/DHE_RSA to RSA, DHE_DSS to DSA) fixed it in a local build of 0.19.3 for P-256 and P-384 on TLS 1.2 and 1.3.
Metadata
Metadata
Assignees
Labels
No labels
Activity
toabctl commentedon Oct 8, 2026
Clarification on when this fails:
SSLContext.new(:TLSv1_2)/ssl_version=), or always in the FIPS variant. A plainSSLContext.newkeeps protocol"SSL"and uses SunJSSE, which works (that's whytest_add_certificate_multiple_certspasses).Fix and a regression test in #384.
kares commentedon Oct 9, 2026
Thanks Thomas, we'll have to take a closer look.
Custom alias choosing was added due BC-JSSE (to support add_certificate: ee71fd9) but maybe there's another path to approach that.
We've expected BC's JSSE engine to be mostly a drop in replacement for the Sun built-in provider but seems like incompatibilities are piling up - will have to investigate the behavioral differences.