Expand Workers compatibility for crypto, filesystem, codecs and networking

README.md

@@ -66,7 +66,7 @@ Install the package in your web project, then follow the
 [complete mobile setup](docs/integration.md#workflow):
 
 ```sh
-npm install --save-exact cf-workers-native@0.1.4
+npm install --save-exact cf-workers-native@0.1.5
 ```
 
 Installation alone does not generate a mobile project. The guide covers adding
@@ -84,7 +84,7 @@ Open your application repository and give your assistant this prompt:
 
 ```text
 Read https://github.com/rdlabo-dev/cf-workers-native/blob/main/docs/integration.md.
-Add an iOS and Android version of this Workers web app using cf-workers-native@0.1.4
+Add an iOS and Android version of this Workers web app using cf-workers-native@0.1.5
 and Capacitor. Preserve the existing pages, backend and Cloudflare deployment configuration.
 Use an existing test Worker for cloud-backed requests; ask before any cloud deployment.
 Start by verifying one page, navigation and a form or data action on a simulator/emulator.

demo/package.json

@@ -75,6 +75,7 @@
     "typescript": "^7.0.2",
     "vite": "^7.3.7",
     "vitest": "^5.0.5",
-    "wrangler": "^4.151.0"
+    "wrangler": "^4.151.0",
+    "ws": "8.21.0"
   }
 }

demo/tests/experiments/node-compat/README.md

@@ -0,0 +1,45 @@
+# Pinned Node compatibility experiments
+
+`exports.json` pins the workerd package and compatibility date. The public
+expansion probe runs shared operation contracts through both build plugins,
+Chromium/WebKit, page and Dedicated Worker. Surface inventory results are separate
+from behavioral acceptance.
+
+From the repository root:
+
+```sh
+node --test demo/tests/experiments/node-compat/expansion-contracts.probe.mjs
+node demo/tests/experiments/node-compat/prepare-events-consumer.mjs
+CONSUMER_PACKAGE=/absolute/path/to/prepared.tgz npm run test:package
+node demo/tests/experiments/node-compat/expansion-native.mjs ios <simulator-udid>
+node demo/tests/experiments/node-compat/expansion-native.mjs android <emulator-id>
+```
+
+Run the native commands sequentially: they share the independent installed
+consumer. Set the Android SDK and `adb` PATH as described in the native test
+procedure. Preparation writes the exact tarball path and SHA-256 to
+`demo/artifacts/node-inventory/events-consumer.json`; the native runner checks that
+hash before compiling the consumer. Do not rebuild repository `dist` while a
+public browser probe reads it.
+
+Native networking needs OpenSSL and Python3. Local HTTP/HTTPS, TLS, STARTTLS and
+WebSocket peers use ephemeral loopback ports. The Python TLS peer explicitly calls
+`unwrap()` to send close_notify; separate peers end or reset the TCP connection
+without that TLS notification. The generated test CA is confined to the debug
+consumer/simulator trust configuration. Workerd uses the same CA and fixture
+ports. FS lifetime probes use two real workerd requests and distinct native
+request scopes.
+
+Page cases run in independent iframe realms and remove each frame on completion.
+Dedicated Workers terminate after each case. This permits cold Wasm measurements
+without retaining a module instance for every case in one page. Progress and
+failure records identify the active case. The overall native deadline is separate
+from individual case and socket deadlines.
+
+`NODE_COMPAT_CASES=name,otherName` filters shared cases for diagnosis. It does not
+filter native network rows, and a filtered run is not full acceptance. Native
+`stage:complete` means all observations were collected; exact assertions run
+subsequently and may still fail. Current TLS and WebSocket implementation gaps
+remain failures, not skipped or accepted differences. The reports, generated
+harness, compiled hash and workerd baseline live under
+`demo/artifacts/node-inventory/native-expansion/<platform>/`.

demo/tests/experiments/node-compat/completion-report.mjs

@@ -23,7 +23,7 @@ const assessments = [
     modules: ["crypto", "zlib"],
     priority: 2,
     category: "implementation-gap",
-    next: "Inventory algorithms, key/codec classes, synchronous and streaming behavior. Evaluate JS/Wasm implementations against all pinned operations; random/hash/HKDF/scrypt, owned secret/asymmetric keys, RSA/EC/Ed25519 signatures, prime generation/checking X509Certificate parsing/fields/name checks/signature checks and Brotli/Zstd profiles now supplement dependency polyfills. Pinned Workers X509 key-check restrictions are documented. Additional key-generation options, remaining algorithms and complete codec contracts remain open. No technical-impossibility claim has been established.",
+    next: "Inventory algorithms, key/codec classes, synchronous and streaming behavior. Evaluate JS/Wasm implementations against all pinned operations; random/Hash/HMAC/PBKDF2/HKDF/scrypt, owned secret/asymmetric keys, RSA/EC/Ed25519 signatures, prime generation/checking X509Certificate parsing/fields/name checks/signature checks and cipher metadata lookup now supplement dependency polyfills. Gzip/deflate now use zlib Wasm; all codec streams use the pinned Workers lifecycle, with option, reset, close, callback and output-limit contracts. Pinned Workers X509 key-check restrictions are documented. The pinned 17 digest names, copying, finalization and KDF length limits have shared contracts; Float16Array input-brand coverage depends on engine availability recorded in native measurements. Additional key-generation options, remaining algorithms and complete codec contracts remain open. No technical-impossibility claim has been established.",
     evidence: [
       "surface-gaps.json",
       "expansion-contracts.probe.mjs",
@@ -35,7 +35,7 @@ const assessments = [
     modules: ["fs", "fs/promises"],
     priority: 2,
     category: "implementation-gap",
-    next: "Compare the Workers virtual filesystem, descriptors, FileHandle, writes, permissions and watch behavior. Request-local memory-backed /tmp now supports writes, descriptors and FileHandle; permission no-ops, statfs and embedded-file copying are tested; compare remaining cross-volume operations, descriptors and stream semantics.",
+    next: "Compare the Workers virtual filesystem, descriptors, FileHandle, writes, permissions and watch behavior. Request-local memory-backed /tmp now supports writes, descriptors and FileHandle; permission no-ops, statfs and embedded-file copying are tested; embedded descriptors/FileHandles, links, immutable copies, cp options, millisecond timestamps and readonly metadata are also compared. Devices, directory iteration, glob, stream fd/custom-fs options, byte streams, URL normalization, FD reuse and request-local cwd now have upstream and comparative coverage. Complete the remaining per-operation/option inventory and all-platform acceptance; these cases do not certify the whole module.",
     evidence: [
       "../../../../src/compat/fs/index.ts",
       "../../../../src/compat/fs/promises/index.ts",

demo/tests/experiments/node-compat/expansion-contracts.probe.mjs

@@ -9,9 +9,37 @@ import { build } from "esbuild";
 import { chromium, webkit } from "@playwright/test";
 import { Miniflare, convertV4MiniflareOptions } from "miniflare";
 import { workersCapacitorPlugin } from "../../../../dist/tooling/esbuild.js";
-import { eventCases } from "../../../../src/compat/expansion/index.spec.mjs";
+import { eventCases as allEventCases } from "../../../../src/compat/expansion/index.spec.mjs";
+
+import {
+  localBoundaryCases as allBoundaryCases,
+  localBoundaryExpected,
+} from "../../../../src/compat/expansion/boundaries.spec.mjs";
+const requestedCases = process.env.NODE_COMPAT_CASES?.split(",");
+if (requestedCases)
+  for (const name of requestedCases)
+    assert.ok(
+      Object.hasOwn(allEventCases, name) ||
+        Object.hasOwn(allBoundaryCases, name),
+      `Unknown compatibility case: ${name}`,
+    );
+const eventCases = requestedCases
+  ? Object.fromEntries(
+      requestedCases
+        .filter((name) => Object.hasOwn(allEventCases, name))
+        .map((name) => [name, allEventCases[name]]),
+    )
+  : allEventCases;
 
 const root = fileURLToPath(new URL("../../../..", import.meta.url));
+const boundaryCases = requestedCases
+  ? Object.fromEntries(
+      requestedCases
+        .filter((name) => Object.hasOwn(allBoundaryCases, name))
+        .map((name) => [name, allBoundaryCases[name]]),
+    )
+  : allBoundaryCases;
+const localCases = { ...eventCases, ...boundaryCases };
 const profile = JSON.parse(
   await readFile(new URL("exports.json", import.meta.url), "utf8"),
 ).profile;
@@ -24,23 +52,32 @@ assert.equal(
 );
 const runtime = new Miniflare(
   convertV4MiniflareOptions({
-    modules: true,
     compatibilityDate: profile.compatibilityDate,
-    script: `import * as tls from 'node:tls'; import * as streams from 'node:stream/web'; import * as module from 'node:module'; import * as httpCommon from 'node:_http_common'; import * as httpOutgoing from 'node:_http_outgoing'; import * as httpServer from 'node:_http_server'; import * as http from 'node:http'; import * as fs from 'node:fs'; import * as zlib from 'node:zlib'; import * as sys from 'node:sys'; import * as util from 'node:util'; import * as types from 'node:util/types'; import * as console from 'node:console'; import * as crypto from 'node:crypto';
+    modulesRoot: "/fixture",
+    modules: [
+      {
+        type: "ESModule",
+        path: "/fixture/main.mjs",
+        contents: `import * as tls from 'node:tls'; import * as classicStreams from 'node:stream'; import * as streams from 'node:stream/web'; import * as module from 'node:module'; import * as httpCommon from 'node:_http_common'; import * as httpOutgoing from 'node:_http_outgoing'; import * as httpServer from 'node:_http_server'; import * as http from 'node:http'; import * as cloudflareNode from 'cloudflare:node'; import * as fs from 'node:fs'; import * as zlib from 'node:zlib'; import * as sys from 'node:sys'; import * as util from 'node:util'; import * as types from 'node:util/types'; import * as console from 'node:console'; import * as crypto from 'node:crypto';
 import { AsyncLocalStorage } from 'node:async_hooks';
 import process from 'node:process';
 const cases = {${Object.entries(eventCases)
-      .map(([name, fn]) => `${JSON.stringify(name)}: (${fn.toString()})`)
-      .join(",")}};
+          .map(([name, fn]) => `${JSON.stringify(name)}: (${fn.toString()})`)
+          .join(",")}};
 export default {async fetch(request) {
  const name = new URL(request.url).pathname.slice(1);
- return Response.json(await cases[name]({process, tls, streams, sys, util, types, console, crypto, fs, zlib, module, httpCommon, httpOutgoing, httpServer, http}));
+ return Response.json(await cases[name]({process, tls, classicStreams, streams, sys, util, types, console, crypto, fs, zlib, module, httpCommon, httpOutgoing, httpServer, http, cloudflareNode}));
 }};`,
+      },
+      { type: "Text", path: "/fixture/readonly.txt", contents: "abcdef" },
+      { type: "Text", path: "/fixture/nested/child.txt", contents: "child" },
+    ],
   }),
 );
 const baseline = {};
 try {
   for (const name of Object.keys(eventCases)) {
+    console.log(`Workers baseline: ${name}`);
     const response = await runtime.dispatchFetch(
       `https://events.invalid/${name}`,
     );
@@ -54,7 +91,12 @@ try {
 } finally {
   await runtime.dispose();
 }
-const output = new URL("../../../artifacts/node-inventory/", import.meta.url);
+const output = new URL(
+  requestedCases
+    ? "../../../artifacts/node-inventory/targeted/"
+    : "../../../artifacts/node-inventory/",
+  import.meta.url,
+);
 await mkdir(output, { recursive: true });
 await writeFile(
   new URL("expansion-contracts-workerd.json", output),
@@ -64,11 +106,13 @@ const compiled = await build({
   absWorkingDir: root,
   stdin: {
     resolveDir: root,
-    contents: `import * as tls from 'node:tls'; import * as streams from 'node:stream/web'; import * as module from 'node:module'; import * as httpCommon from 'node:_http_common'; import * as httpOutgoing from 'node:_http_outgoing'; import * as httpServer from 'node:_http_server'; import * as http from 'node:http'; import * as fs from 'node:fs'; import * as zlib from 'node:zlib'; import * as sys from 'node:sys'; import * as util from 'node:util'; import * as types from 'node:util/types'; import * as console from 'node:console'; import * as crypto from 'node:crypto';
-import { eventCases } from './src/compat/expansion/index.spec.mjs';
+    contents: `import * as tls from 'node:tls'; import * as classicStreams from 'node:stream'; import * as streams from 'node:stream/web'; import * as module from 'node:module'; import * as httpCommon from 'node:_http_common'; import * as httpOutgoing from 'node:_http_outgoing'; import * as httpServer from 'node:_http_server'; import * as http from 'node:http'; import * as cloudflareNode from 'cloudflare:node'; import * as fs from 'node:fs'; import * as zlib from 'node:zlib'; import * as sys from 'node:sys'; import * as util from 'node:util'; import * as types from 'node:util/types'; import * as console from 'node:console'; import * as crypto from 'node:crypto';
+import { eventCases } from './src/compat/expansion/index.spec.mjs'; import {localBoundaryCases} from './src/compat/expansion/boundaries.spec.mjs';
+import { runWithRequestScope as withRequestScope } from './dist/shared/request-scope.js';
 import { AsyncLocalStorage } from 'node:async_hooks';
 import process from 'node:process';
-export const run = name => eventCases[name]({process, tls, streams, sys, util, types, console, crypto, fs, zlib, module, httpCommon, httpOutgoing, httpServer, http});`,
+fs.registerVirtualFiles({'/bundle/readonly.txt':'abcdef','/bundle/nested/child.txt':'child'});
+export const run = name => (eventCases[name] ?? localBoundaryCases[name])({withRequestScope,process, tls, classicStreams, streams, sys, util, types, console, crypto, fs, zlib, module, httpCommon, httpOutgoing, httpServer, http, cloudflareNode});`,
   },
   bundle: true,
   platform: "browser",
@@ -85,11 +129,14 @@ export const run = name => eventCases[name]({process, tls, streams, sys, util, t
 const entry = fileURLToPath(new URL("expansion-vite-entry.mjs", output));
 await writeFile(
   entry,
-  `import * as tls from 'node:tls'; import * as streams from 'node:stream/web'; import * as module from 'node:module'; import * as httpCommon from 'node:_http_common'; import * as httpOutgoing from 'node:_http_outgoing'; import * as httpServer from 'node:_http_server'; import * as http from 'node:http'; import * as fs from 'node:fs'; import * as zlib from 'node:zlib'; import * as sys from 'node:sys'; import * as util from 'node:util'; import * as types from 'node:util/types'; import * as console from 'node:console'; import * as crypto from 'node:crypto';
+  `import * as tls from 'node:tls'; import * as classicStreams from 'node:stream'; import * as streams from 'node:stream/web'; import * as module from 'node:module'; import * as httpCommon from 'node:_http_common'; import * as httpOutgoing from 'node:_http_outgoing'; import * as httpServer from 'node:_http_server'; import * as http from 'node:http'; import * as cloudflareNode from 'cloudflare:node'; import * as fs from 'node:fs'; import * as zlib from 'node:zlib'; import * as sys from 'node:sys'; import * as util from 'node:util'; import * as types from 'node:util/types'; import * as console from 'node:console'; import * as crypto from 'node:crypto';
 import { eventCases } from ${JSON.stringify(fileURLToPath(new URL("../../../../src/compat/expansion/index.spec.mjs", import.meta.url)))};
+import {localBoundaryCases} from ${JSON.stringify(fileURLToPath(new URL("../../../../src/compat/expansion/boundaries.spec.mjs", import.meta.url)))};
+import { runWithRequestScope as withRequestScope } from ${JSON.stringify(fileURLToPath(new URL("../../../../dist/shared/request-scope.js", import.meta.url)))};
 import { AsyncLocalStorage } from 'node:async_hooks';
 import process from 'node:process';
-export const run = name => eventCases[name]({process, tls, streams, sys, util, types, console, crypto, fs, zlib, module, httpCommon, httpOutgoing, httpServer, http});`,
+fs.registerVirtualFiles({'/bundle/readonly.txt':'abcdef','/bundle/nested/child.txt':'child'});
+export const run = name => (eventCases[name] ?? localBoundaryCases[name])({withRequestScope,process, tls, classicStreams, streams, sys, util, types, console, crypto, fs, zlib, module, httpCommon, httpOutgoing, httpServer, http, cloudflareNode});`,
 );
 const viteResult = await viteBuild({
   configFile: false,
@@ -126,7 +173,7 @@ for (const [builder, source] of Object.entries({
         const browser = await launcher.launch();
         const results = {};
         try {
-          for (const name of Object.keys(eventCases)) {
+          for (const name of Object.keys(localCases)) {
             await t.test(name, { timeout: 15000 }, async () => {
               const page = await browser.newPage();
               await page.route("http://localhost/", (route) =>
@@ -186,7 +233,12 @@ for (const [builder, source] of Object.entries({
                   { source, name, mode },
                 );
                 results[name] = result;
-                assert.deepEqual(result, baseline[name]);
+                assert.deepEqual(
+                  result,
+                  Object.hasOwn(boundaryCases, name)
+                    ? localBoundaryExpected[name]
+                    : baseline[name],
+                );
               } finally {
                 await page.close();
               }
@@ -206,6 +258,7 @@ for (const [builder, source] of Object.entries({
                 engine,
                 mode,
                 status: "webview-profile-operation-contracts",
+                localBoundaryCases: Object.keys(boundaryCases),
                 knownGaps: {
                   process:
                     "existing partial process profile; native stack inspection remains engine-dependent",