[zhouyunding]

Summary

Addresses #896 (server side).

libhv's WebSocket parser accepted protocol-violating frames and delivered them to the application layer. Measured against a 57-case probe (41 frames that RFC 6455 requires a server to reject, 16 that must pass): the baseline accepted 55/57 violating frames; only the two control-frame rules added by #904 were enforced.

Root cause

websocket_parser.c assembles frame flags and dispatches payloads without validating:

1. RSV1/RSV2/RSV3 — must be 0 while no extension is negotiated (RFC 6455 §5.2);
2. reserved opcodes (0x3–0x7, 0xB–0xF) — must be rejected (§5.2);
3. MASK direction — a server must close the connection when it receives an unmasked frame from a client (§5.1).

The RSV bits were folded into the internal flags before any validation could see them, so RSV-marked frames were delivered as ordinary messages.

Approach and trade-offs

• Validation is done in the parser's state machine (both whole-buffer and byte-by-byte paths), rejecting offending frames by refusing to consume the input (nfeed != len), which makes the existing public call sites (HttpHandler.cpp, WebSocketClient.cpp) fail closed and close the connection.
• A parser role flag is introduced and set at the only two construction sites (HttpHandler.cpp:208 server side, WebSocketClient.cpp:149 client side), before any FeedRecvData call.
• Scope decision: server side only. The symmetric client-side rule (clients must reject masked server frames) is not implemented in this PR; the new tests pin the current client behavior explicitly so a future change is visible. This keeps the behavioral delta minimal and reviewable.

Behavior changes (disclosure)

• Servers now close the connection on RSV-set / reserved-opcode / unmasked-client frames instead of delivering them. Payload of well-formed frames is unchanged.
• After a rejection, the parser state (partial body consumption) is consistent with the existing fix(http): keep WebSocket control frames out of fragmented messages #904 design; all public entry points treat nfeed != len as an error and close, so this is fail-closed in practice.
• Conflicts: file-level overlap with open PR fix(websocket): enforce MAX_PAYLOAD_LENGTH cap on message accumulation #875 (both touch http/WebSocketParser.cpp). Semantic interaction with open PR Add HTTP gzip/zstd compression and WebSocket permessage-deflate support #819: this PR rejects RSV1 unconditionally, which is correct while no extension is negotiated; if permessage-deflate (Add HTTP gzip/zstd compression and WebSocket permessage-deflate support #819) lands, RSV1 handling will need to be gated on successful extension negotiation.

Test evidence (before / after, same probe)

Probe of 57 frames (41 must-reject, 16 must-accept), run against baseline 060a4b6 and this commit fc9d417:

• Baseline: 55/57 must-reject frames accepted (only fix(http): keep WebSocket control frames out of fragmented messages #904's two rules caught); all 3 sampled violation classes (unmasked TEXT, RSV1, opcode 0x3) delivered to the app.
• Patched: 41/41 must-reject frames refused (consumed < len, 0 delivered); 16/16 must-accept frames delivered with correct payload (extended lengths, fragmentation, interleaved PING).
• Full role × opcode × RSV × length matrix: 16×8×2×2 = 2134 expectations, zero deviation from RFC.
• RED independently reproduced: the new test file linked against the baseline library aborts in the rsv / opcode / mask assertion groups; against the patched library: PASS: 10 WebSocket parser tests.
• End-to-end against a real hv::WebSocketServer with raw TCP clients: unmasked / RSV / reserved-opcode frames → msgs=0 and connection closed by peer; valid frames still echoed.
• unittest/websocket_parser_test.cpp's main gained an optional single-test-name argument used for the per-group RED/GREEN reproduction; running it without arguments behaves exactly as before.

Environment: Windows 11, MinGW UCRT64 (gcc 16.2.0), CMake Release. Not covered locally: ASan/UBSan (no sanitizer runtime in MinGW), Linux/macOS/MSVC builds (left to CI), the TLS/wss path, and fuzzing.

AI disclosure

This change was prepared with AI assistance: an AI agent wrote the implementation and tests under human direction, and independent AI agents performed the code review and the verification reported above. The human contributor takes responsibility for the change.