Skip to content

rest: replace select() polling with event-driven CurlIoLoop and support HTTP/2 - #1956

Draft
AustinBenoit wants to merge 2 commits into
mainfrom
libcurl-http2-polish
Draft

AustinBenoit wants to merge 2 commits into
mainfrom
libcurl-http2-polish

Conversation

@AustinBenoit

@AustinBenoit AustinBenoit commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Modernizes the desktop HTTP/REST transport (app/rest/) by replacing the legacy 50 ms select() + curl_multi_fdset() polling loop with an event-driven process-wide CurlIoLoop (curl_multi_poll() + curl_multi_wakeup()), off-thread async completion delivery (CompletionQueue), thread-safe transfer control (TransferControl), and HTTP/2 multiplexing support.

Architecture & Key Changes

1. Event-Driven HTTP Loop (app/rest/curl/curl_io_loop.{h,cc})

  • Replaces curl_multi_fdset() + select() + 33–50 ms polling sleeps with curl_multi_poll() + curl_multi_wakeup().
  • Eliminates the FD_SETSIZE = 1024 Cliff: Under select(), any socket with fd >= 1024 (common in Unreal/Unity desktop games with open asset packs) was invisible to fd_set, causing multi-second stalls when paired with a low-FD transfer or degrading into polling ticks. curl_multi_poll() uses OS poll()/WSAPoll() with no descriptor ceiling.
  • Zero Idle CPU & Sub-Millisecond Wakeups: When idle, the I/O thread blocks up to kIdlePollWaitMs (60 s) with 0% CPU wakeups; new transfers, pauses, resumes, and cancellations wake the loop immediately via curl_multi_wakeup().

2. Off-Thread Asynchronous Completion Delivery (CompletionQueue)

  • Delivers asynchronous Response::MarkCompleted() and Response::MarkFailed() callbacks on a dedicated CompletionQueue worker thread rather than inline on the CurlIoLoop thread.
  • Prevents application callbacks, chained Future continuations, or synchronous token refresh RPCs from blocking or deadlocking the shared network I/O thread.
  • CleanupTransportCurl() drains all queued completions before returning, preserving deterministic teardown semantics.

3. Thread-Safe ControllerCurl & TransferControl

  • Replaces raw cross-thread pointer sharing between ControllerCurl and TransportCurl with a shared std::shared_ptr<TransferControl> state block.
  • Sticky Pause Before First Byte: Calling Controller::Pause() immediately after Perform() (before libcurl has opened a socket) latches paused state and returns CURL_WRITEFUNC_PAUSE / CURL_READFUNC_PAUSE on the first data callback instead of being lost as a no-op.
  • Immediate Cancellation & Exact Progress: Controller::Cancel() wakes the loop immediately and aborts via CURLOPT_XFERINFOFUNCTION, which also keeps BytesTransferred() and TransferSize() exact through 100% completion.

4. HTTP/2 & RFC 9110 Response Parsing

  • Enables CURL_HTTP_VERSION_2TLS (HTTP/2 over TLS with automatic HTTP/1.1 fallback) when LibcurlSupportsHttp2() is true, and builds bundled libcurl alongside nghttp2.
  • Updates Response::ParseHeader() to parse HTTP/2 and HTTP/3 status lines (HTTP/2 200, HTTP/3 204), reasonless status lines (HTTP/1.1 200), and intermediate 100 Continue responses, with case-insensitive RFC 9110 header lookup.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces the legacy uWebSockets and libuv WebSocket transport with a libcurl-based implementation on desktop platforms, consolidating all REST and Realtime Database WebSocket traffic onto a single shared, event-driven I/O thread. It also centralizes the TLS policy configuration and removes the libuv and uWebSockets dependencies. The review feedback highlights potential unsigned integer underflow risks in web_socket_client_impl.cc when performing bounds checks on incoming PING and message payloads, recommending that these subtraction-based checks be rewritten using addition.

Comment on lines +502 to +505
if ((meta->flags & CURLWS_CONT) != 0 ||
rlen > kMaxPingPayloadBytes - incoming_ping_buffer_.size() ||
frame_remaining >
kMaxPingPayloadBytes - (incoming_ping_buffer_.size() + rlen)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent potential unsigned integer underflow when checking the bounds of the incoming PING frame payload, rewrite the subtraction checks using addition. This is a safer, more defensive programming practice.

Suggested change
if ((meta->flags & CURLWS_CONT) != 0 ||
rlen > kMaxPingPayloadBytes - incoming_ping_buffer_.size() ||
frame_remaining >
kMaxPingPayloadBytes - (incoming_ping_buffer_.size() + rlen)) {
if ((meta->flags & CURLWS_CONT) != 0 ||
incoming_ping_buffer_.size() + rlen > kMaxPingPayloadBytes ||
incoming_ping_buffer_.size() + rlen + frame_remaining >
kMaxPingPayloadBytes) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: incoming_ping_buffer_.size() <= kMaxPingPayloadBytes is an invariant enforced before every append(), so kMaxPingPayloadBytes - incoming_ping_buffer_.size() cannot underflow, whereas incoming_ping_buffer_.size() + rlen + frame_remaining could overflow uint64_t if a peer sent a header with bytesleft near INT64_MAX. Also note database/ now lives in base PR #1959 (0 lines touched in #1956).

Comment on lines +544 to +547
if (rlen > kMaxMessagePayloadBytes - incoming_message_buffer_.size() ||
frame_remaining >
kMaxMessagePayloadBytes -
(incoming_message_buffer_.size() + rlen)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similarly, to prevent potential unsigned integer underflow when checking the bounds of the incoming message payload, rewrite the subtraction checks using addition.

      if (incoming_message_buffer_.size() + rlen > kMaxMessagePayloadBytes ||
          incoming_message_buffer_.size() + rlen + frame_remaining >
              kMaxMessagePayloadBytes) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: incoming_message_buffer_.size() <= kMaxMessagePayloadBytes is an invariant so the subtraction form avoids uint64_t addition overflow against untrusted frame_remaining.

@AustinBenoit AustinBenoit changed the title Unify desktop REST and Realtime Database WebSockets on a shared event-driven libcurl I/O loop Desktop networking: event-driven libcurl REST loop; libcurl WebSockets replace uWebSockets/libuv Nov 3, 2026
@AustinBenoit
AustinBenoit changed the base branch from main to libcurl-websockets November 4, 2026 04:33
@AustinBenoit AustinBenoit changed the title Desktop networking: event-driven libcurl REST loop; libcurl WebSockets replace uWebSockets/libuv rest: replace select() polling with event-driven CurlIoLoop and support HTTP/2 Nov 4, 2026
- Drop unmaintained external dependencies uWebSockets (v0.14) and libuv
  (v1.40) from CMake, external rules, patches, and release NOTICES.
- Migrate Realtime Database WebSocket client (WebSocketClientImpl) to
  libcurl's native WebSocket support (curl_ws_send, curl_ws_recv,
  CURLOPT_CONNECT_ONLY = 2L).
- Give each WebSocket connection its own lightweight event thread and private
  CURLM* handle using curl_multi_poll() and curl_multi_wakeup().
- Add stalled handshake deadline watchdog (5000 ms default).
- Centralize SDK transport security in app/rest/curl/tls_policy.{h,cc}
  (TLS 1.2+ minimum, peer/hostname verification, and native OS certificate
  trust store via Schannel on Windows, Keychain on macOS, and runtime CA bundle
  detection on Linux).
- Add thread-safe reference-counted AcquireCurlGlobal() / ReleaseCurlGlobal()
  and RAII handle deleters in app/rest/curl/curl_handles.{h,cc}.
- Initialize DatabaseInternal::persistence_enabled_ to false to prevent
  reading uninitialized member variables on desktop platforms.
- Guard Logger::LogMessageImplV when parent_logger_ is null.
- Fix package.sh symbol hiding script generation for libfirebase_database.a.
- Add comprehensive WebSocket unit tests (16/16 passing) and concurrent
  multi-app / multi-frame integration tests.
@AustinBenoit

Copy link
Copy Markdown
Contributor Author

📚 Stacked PR Navigation

@AustinBenoit
AustinBenoit force-pushed the libcurl-websockets branch 3 times, most recently from 94d75ff to 32dac60 Compare November 4, 2026 15:31
@AustinBenoit
AustinBenoit marked this pull request as draft November 4, 2026 16:50
@AustinBenoit
AustinBenoit changed the base branch from libcurl-websockets to main November 4, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant