Skip to content

Rework TLS to OpenSSL-native style (memory BIOs, explicit status codes)#8002

Draft
achamayou with Copilot wants to merge 20 commits into
mainfrom
copilot/rework-tls-to-match-openssl-style
Draft

Rework TLS to OpenSSL-native style (memory BIOs, explicit status codes)#8002
achamayou with Copilot wants to merge 20 commits into
mainfrom
copilot/rework-tls-to-match-openssl-style

Conversation

Copilot AI commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Now that MbedTLS is gone from the tree, the TLS layer no longer needs to emulate its style. This reworks ccf::tls::Context and TLSSession to use OpenSSL idioms directly, removing the callback machinery and the "negative return" overloading. Net ~330 fewer lines.

Changes

  • In-memory BIOs replace BIO callbacks (context.h): set_bio() attaches plain BIO_s_mem read/write BIOs to the SSL. TLSSession feeds inbound ciphertext via recv() and drains outbound ciphertext via pending_write()/send() after each SSL operation. No more send_callback/recv_callback indirection.
  • Status separated from byte count (context.h, tls.h): handshake/read/write/close return 0 on success or an OpenSSL SSL_ERROR_* code (from SSL_get_error); bytes transferred are reported through an out-param. Removes the negative-return hack. Only TLS_ERR_X509_VERIFY remains as a CCF-specific sentinel to distinguish a handshake cert-verification failure (reported by OpenSSL as a generic SSL_ERROR_SSL) so callers can treat it as an auth failure.
  • OpenSSL error-queue hygiene (context.h): each SSL operation clears unrelated thread-local errors before calling OpenSSL, as required for reliable SSL_get_error() classification.
  • Callers check status explicitly (tls_session.h): read/flush/do_handshake/close switch on SSL_ERROR_WANT_READ/WANT_WRITE/ZERO_RETURN/etc. instead of inspecting negated/overloaded returns. Outbound ciphertext is drained into pending_out and retried on ring-buffer backpressure. Deletes the obsolete handle_send/handle_recv/*_callback_openssl/write_some paths.
  • PlaintextServer (plaintext_server.h): updated to the new virtual signatures over its own pair of memory BIOs.
  • Unit tests (tls/test/main.cpp): rewritten from a threaded socketpair + callback BIOs to a single-threaded BIO pump; all existing cases preserved. Added coverage for stale errors from unrelated OpenSSL operations, which must be cleared before calls interpreted by SSL_get_error().
  • Docs (tls/README.md, architecture/tls_internals.rst): describe the memory-BIO design and status-code contract; drop the MbedTLS-emulation and "future pure-OpenSSL" sections.

Read path, before/after

// Before: status smuggled through the return value (negated on error)
int n = ctx->read(buf, len);
if (n == TLS_ERR_WANT_READ) { ... }
else if (n < 0) { /* error */ }
else { /* n bytes */ }

// After: status and byte count are distinct
size_t readbytes = 0;
auto rc = ctx->read(buf, len, readbytes);
switch (rc)
{
  case 0: /* readbytes valid */ break;
  case SSL_ERROR_WANT_READ:
  case SSL_ERROR_WANT_WRITE: /* retry later */ break;
  case SSL_ERROR_ZERO_RETURN: /* peer closed */ break;
  default: /* error */ break;
}

Notes for reviewers

  • This is security-critical code; the cert-verification classification (peer_cert_ok() on SSL_ERROR_SSL) is the one place where OpenSSL's status is reinterpreted.
  • The Debug build, tls_test, and programmability_and_jwt pass locally. scripts/ci-checks.sh and all required PR CI jobs are green.

Copilot AI and others added 3 commits June 29, 2026 17:42
Co-authored-by: achamayou <4016369+achamayou@users.noreply.github.com>
Co-authored-by: achamayou <4016369+achamayou@users.noreply.github.com>
Co-authored-by: achamayou <4016369+achamayou@users.noreply.github.com>
Copilot AI changed the title [WIP] Refactor TLS implementation to align with OpenSSL style Rework TLS to OpenSSL-native style (memory BIOs, explicit status codes) Jun 29, 2026
Copilot AI requested a review from achamayou June 29, 2026 17:48
@achamayou

Copy link
Copy Markdown
Member

@copilot check previous CI runs, find out what failed and fix it.

Co-authored-by: achamayou <4016369+achamayou@users.noreply.github.com>

Copilot AI commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

@copilot check previous CI runs, find out what failed and fix it.

Investigated the failing runs (ACI SNP Genoa/Milan + VMSS Virtual C): all hung in programmability_and_jwt / test_jwt_without_key_policy. The node logs show the server generated the expected 400 but never delivered it; the client saw Server disconnected without sending a response. main is green on this test, so it was a regression from this PR.

Root cause: in the new memory-BIO design, TLSSession::read() only drained the write BIO before the read (via flush()). Ciphertext that the TLS layer produces during SSL_read (TLS 1.3 NewSessionTickets / post-handshake messages / key updates) was left stranded in the write BIO. The original BIO send-callback pushed those bytes to the host synchronously, so this is an asymmetry the rework introduced.

Fix (3a8... commit): drain outbound with flush_outbound() immediately after ctx->read(), restoring parity with the original behaviour. Please confirm tls_test and programmability_and_jwt are green in CI.

achamayou and others added 4 commits July 3, 2026 09:42
OpenSSL requires the thread-local error queue to be empty before SSL_get_error is used. Clear stale errors at each TLS operation boundary so unrelated crypto validation failures cannot close healthy sessions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Member

Diagnosed and fixed the remaining programmability_and_jwt failure in bca9375.

The invalid X.509 policy check leaves ASN.1 entries on OpenSSL's thread-local error queue. Because SSL_get_error() requires that queue to be empty before the corresponding SSL operation, the next no-data SSL_read_ex() was misclassified as SSL_ERROR_SSL rather than SSL_ERROR_WANT_READ. The session was then closed before its already-generated 400 response was delivered.

The fix clears stale errors before handshake/read/write/shutdown operations and adds a regression test that reproduces the misclassification. I also merged the latest main. Locally, tls_test and programmability_and_jwt pass through tests.sh, and scripts/ci-checks.sh passes.

@achamayou achamayou added 3.x-todo PRs which should be backported to 3.x bench-ab labels Jul 10, 2026
@achamayou
achamayou marked this pull request as ready for review July 10, 2026 22:54
@achamayou
achamayou requested a review from a team as a code owner July 10, 2026 22:54
Copilot AI review requested due to automatic review settings July 10, 2026 22:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR modernises CCF’s TLS stack to follow OpenSSL-native patterns now that the MbedTLS compatibility layer is gone. It replaces the callback-based I/O emulation with in-memory BIOs and makes TLS APIs return explicit OpenSSL SSL_ERROR_* status codes (with byte counts via out-params), simplifying control flow in TLSSession and related components.

Changes:

  • Replace BIO callbacks with explicit in-memory BIO pump APIs (Context::recv/pending_write/send) and update TLSSession to drive them.
  • Split status and byte counts for handshake/read/write/close, removing the previous negative/overloaded return scheme (keeping TLS_ERR_X509_VERIFY as a sentinel).
  • Update unit tests and docs to reflect the new memory-BIO design and status-code contract.

Custom instructions used:

  • .github/copilot-instructions.md
  • .github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/tls/context.h Switch Context to memory BIOs and explicit OpenSSL-style status codes.
src/enclave/tls_session.h Update session drive loop to feed/drain BIOs and switch on SSL_ERROR_* results.
src/tls/plaintext_server.h Adapt the non-TLS passthrough server to the new BIO-driven virtual interface.
src/tls/tls.h Remove legacy error macros; keep TLS_ERR_X509_VERIFY sentinel.
src/tls/test/main.cpp Rework TLS tests to a single-threaded “BIO pump” model; add stale-error coverage.
src/tls/README.md Update TLS design documentation to describe the new OpenSSL-native approach.
doc/architecture/tls_internals.rst Update architecture docs to describe BIO-driven I/O and status-code contract.
Comments suppressed due to low confidence (1)

src/tls/tls.h:14

  • tls.h uses INT_MIN for TLS_ERR_X509_VERIFY but doesn’t include . Relying on transitive includes is brittle; include (or switch to std::numeric_limits::min()) so this header is self-contained.
// Specific error to flag a certificate validation failure during the handshake.
// This is not emitted by OpenSSL itself (its handshake failures surface as
// SSL_ERROR_SSL), but is set by Context so the caller can distinguish an
// authentication failure from a generic error. We use a bogus negative value
// that won't match any OpenSSL SSL_ERROR_* code.
#define TLS_ERR_X509_VERIFY INT_MIN

#include "ccf/crypto/openssl/openssl_wrappers.h"

#include <string>

Comment thread src/tls/context.h
Comment thread src/tls/context.h
Comment thread src/tls/plaintext_server.h
Comment thread src/tls/plaintext_server.h
Comment thread src/enclave/tls_session.h
Comment thread src/enclave/tls_session.h
Comment thread src/enclave/tls_session.h
@achamayou achamayou added READY TO MERGE and removed 3.x-todo PRs which should be backported to 3.x labels Jul 11, 2026
Use size_t-safe BIO APIs and include SSL statuses in error diagnostics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@achamayou achamayou added the run-long-test Run Long Test job label Jul 13, 2026
GitHub Copilot and others added 3 commits July 13, 2026 15:23
Remove obsolete error-string setup, use the non-owning peer certificate accessor, and drop the unreachable certificate lookup retry while covering exact peer certificate extraction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Description

Comparing this branch (#8002) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The blue line is this branch's latest run; the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show this branch's value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where this branch improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

Throughput (tx/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-4{stroke-width:2px!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarAxisLabel:nth-of-type(1){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(2){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(7){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#008FD3"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 58,700 tx/s ▼ 11%"]
  axis b1["Basic Blocking: 979 tx/s ▼ 3%"]
  axis b2["Basic JS: 4,071 tx/s ▼ 13%"]
  axis b3["Basic Multi-Threaded: 74,476 tx/s ▼ 13%"]
  axis b4["Historical Queries: 178,329 tx/s ▼ 13%"]
  axis b5["Logging: 51,553 tx/s ▼ 15%"]
  axis b6["Logging JWT: 8,893 tx/s ▼ 12%"]
  curve stddev2_high["main EWMA + 2 std dev"]{106.99, 104.13, 106.91, 106.49, 112.29, 108.58, 107.05}
  curve stddev1_high["main EWMA + 1 std dev"]{103.49, 102.06, 103.46, 103.25, 106.14, 104.29, 103.52}
  curve stddev1_low["main EWMA - 1 std dev"]{96.51, 97.94, 96.54, 96.75, 93.86, 95.71, 96.48}
  curve stddev2_low["main EWMA - 2 std dev"]{93.01, 95.87, 93.09, 93.51, 87.71, 91.42, 92.95}
  curve branch["#8002"]{88.77, 97.27, 86.92, 86.87, 87.22, 85.25, 88.07}
  graticule polygon
  max 122
  min 75
  ticks 0
  showLegend false
Loading

Latency (ms)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-4{stroke-width:2px!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#008FD3"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Commit Latency 16ms: 5.19 ms ▬ -5%"]
  axis b1["Commit Latency 1ms: 1.71 ms ▬ -3%"]
  axis b2["Commit Latency 256ms: 210 ms ▲ 1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{150.41, 107.32, 101.11}
  curve stddev1_high["main EWMA + 1 std dev"]{125.21, 103.66, 100.55}
  curve stddev1_low["main EWMA - 1 std dev"]{74.79, 96.34, 99.45}
  curve stddev2_low["main EWMA - 2 std dev"]{49.59, 92.68, 98.89}
  curve branch["#8002"]{94.56, 96.96, 101.46}
  graticule polygon
  max 186
  min 14
  ticks 0
  showLegend false
Loading

Memory (bytes)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-4{stroke-width:2px!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#008FD3"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 88.7 MiB ▬ +1%"]
  axis b1["Basic Blocking: 71.2 MiB ▬ 0%"]
  axis b2["Basic JS: 70.5 MiB ▬ 0%"]
  axis b3["Basic Multi-Threaded: 89.7 MiB ▬ 0%"]
  axis b4["Logging: 76.9 MiB ▲ 2%"]
  axis b5["Logging JWT: 67.7 MiB ▬ 0%"]
  curve stddev2_high["main EWMA + 2 std dev"]{107.12, 100.72, 103.81, 102.17, 102.10, 101.21}
  curve stddev1_high["main EWMA + 1 std dev"]{103.56, 100.36, 101.91, 101.08, 101.05, 100.60}
  curve stddev1_low["main EWMA - 1 std dev"]{96.44, 99.64, 98.09, 98.92, 98.95, 99.40}
  curve stddev2_low["main EWMA - 2 std dev"]{92.88, 99.28, 96.19, 97.83, 97.90, 98.79}
  curve branch["#8002"]{101.13, 100.32, 99.82, 100.38, 101.80, 99.54}
  graticule polygon
  max 113
  min 87
  ticks 0
  showLegend false
Loading

Rate (ops/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-4{stroke-width:2px!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#008FD3"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["CHAMP get: 32,909,114 ops/s ▬ -9%"]
  axis b1["CHAMP put: 4,676,792 ops/s ▼ 11%"]
  axis b2["KV deserialisation: 1,367,802 ops/s ▼ 13%"]
  axis b3["KV serialisation: 1,246,572 ops/s ▼ 13%"]
  axis b4["KV snapshot deserialis...: 3,642 ops/s ▼ 12%"]
  axis b5["KV snapshot serialisation: 3,978 ops/s ▼ 12%"]
  curve stddev2_high["main EWMA + 2 std dev"]{119.55, 116.06, 104.02, 103.64, 103.85, 114.01}
  curve stddev1_high["main EWMA + 1 std dev"]{109.77, 108.03, 102.01, 101.82, 101.93, 107.00}
  curve stddev1_low["main EWMA - 1 std dev"]{90.23, 91.97, 97.99, 98.18, 98.07, 93.00}
  curve stddev2_low["main EWMA - 2 std dev"]{80.45, 83.94, 95.98, 96.36, 96.15, 85.99}
  curve branch["#8002"]{90.92, 88.82, 86.77, 87.03, 87.86, 88.31}
  graticule polygon
  max 134
  min 66
  ticks 0
  showLegend false
Loading

@achamayou
achamayou marked this pull request as draft July 17, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bench-ab run-long-test Run Long Test job

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants