fix: unblock route handlers, frame lookup, and worker/persona integrity

[DemonMartin]

Six defects found while driving a hardened, anti-detect Chromium build through rustwright. Each was reproduced against a real browser first, then fixed.

Correction to the original description of this PR. It attributed the anti-bot rejection to the navigator.webdriver handling. That was wrong, and the way it was wrong is worth stating: the first attempt guarded the webdriver block with an early return out of the whole init script, which happened to skip everything after it too. The challenge started passing, and the webdriver change got the credit. @chatgpt-codex-connector caught the return scoping the wrong thing; once it was narrowed to the webdriver block alone, the rejection came straight back. Bisecting the four blocks of the init script individually identified the real cause: the Worker constructor rewrite.

Which block actually causes the rejection

The stealth init script has four independent blocks after the webdriver handling. Each was disabled in isolation against a live challenge:

blocks skipped	result
all four	passes
4 — Worker constructor rewrite	passes
1 — window.chrome object	rejected
none (shipped behaviour)	rejected

Only block 4 matters. The other three, including the synthetic window.chrome, are not implicated: a surface diff against Playwright driving the same browser showed window.chrome identical between the two.

1. Dedicated workers are moved off their own script URL

The page's Worker constructor was replaced with one that builds a Blob wrapping the real script:

const source = `${identitySource}\nimportScripts(${JSON.stringify(absoluteUrl)});`;
const blobUrl = URL.createObjectURL(new Blob([source], { type: 'text/javascript' }));
return new NativeWorker(blobUrl, workerOptions);

Every worker the page creates therefore runs from a blob: URL instead of its own. Its location and origin change, worker-src CSP no longer matches, and any worker resolving a URL against its own location resolves against the blob. This is not only detectable — it changes what the worker is.

The mechanism existed because Target.setAutoAttach explicitly excluded worker, so a dedicated worker had no session to install identity over. It is now included, and identity is installed over the worker's own session while it is still paused — exactly how service workers were already handled — so the rewrite is unnecessary and is removed.

Verified directly, driving the same browser with --fingerprint-platform=windows --lang=de-DE:

page   ua: Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/149.0.0.0 Safari/537.36
worker   : {"href": "http://127.0.0.1:43423/w.js", "language": "de-DE",
            "platform": "Win32", "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/149.0.0.0 ..."}
UA matches page      : True
worker keeps real URL: True

The identity guarantees the wrapper provided are unchanged; the worker now keeps its real URL.

2. page.evaluate() rejects an IIFE containing an arrow function

looks_like_function searched the entire source for the first => and accepted the expression when the text before it started with (. Every IIFE starts with (, so one whose body contained an arrow was classified as a function literal, wrapped as const __rw_fn = (<iife>); return __rw_fn() and called a second time — on the value the IIFE had already returned:

(function () { return [1, 2].map((value) => value * 2); })()
Error: __rw_fn is not a function

The head of the expression is now parsed rather than the whole body scanned: the arrow must bind the leading parameter list. Finding that list means matching the opening parenthesis, so the scan skips string literals, template literals, comments, and regular expressions — (a = ")") => a, (a /* ) */) => a and (pattern = /[)]/) => pattern all resolve correctly. Keyword matching is whole-word, so functionality() is no longer read as a function literal.

3. Request.frame inside a route handler stalls the navigation

refresh_page_frame_tree gave each session the caller's entire remaining deadline, for a refresh the function itself treats as optional — a session that does not answer is skipped and the cached tree returned anyway.

A renderer does not answer Page.getFrameTree while a Fetch interception is paused on its document request, which is precisely the state a route handler runs in. Reading Request.frame there burned one full deadline per session walked, and the navigation whose request was paused could not proceed meanwhile. Measured with a 4 s default timeout, page.frames took 38 s, and page.goto then failed with Timeout ... exceeded. With the timeout disabled it never returned, because command_timeout maps 0 to 24 hours.

The refresh is now bounded independently of the caller's budget. Separately, the Python shim re-issued the same command over a raw CDPSession whenever the core's tree came back empty — an empty tree is a legitimate answer for a document that has not committed one yet, and re-asking cost CDPSession.send's fixed 30 s once per frame walked. The core owns this tree, so the shim no longer duplicates the round trip.

4. The stealth override pins Accept-Language, overriding the configured locale

install_stealth_defaults hardcoded "acceptLanguage": "en-US,en" into the user-agent override applied to every session, taking precedence over --accept-lang and --lang. A browser launched for one region reported that region's timezone next to an en-US locale.

Omitting the field leaves whatever the browser was configured with. Verified both directions:

launch	Accept-Language header	navigator.languages
no argument	en-US,en;q=0.9	['en-US', 'en']
--accept-lang=de-DE,de	de-DE,de;q=0.9	['de-DE', 'de']

The first row is what test_default_request_headers_do_not_expose_headless_chrome asserts, so the documented default for a stock launch is unchanged.

5. navigator.webdriver is removed from browsers that report false

The init script deleted Navigator.prototype.webdriver unconditionally. On a stock Chromium advertising automation that is the intent; on a browser already reporting false — what every real Chrome reports — it traded a correct answer for a missing property.

It now guards only that block, so everything after it still runs. That distinction matters: the default launch args include --disable-blink-features=AutomationControlled, so navigator.webdriver === false is the common path, and an early return would have disabled the whole init script for ordinary users. A unit test now asserts the guard closes before the chrome-object setup and that the script contains no early return.

This is a correctness fix, not the fix for the rejection.

6. Frames inside a shadow root are missing from page.frames

Child frames came from a document.querySelectorAll('iframe,frame') result, with the protocol's children used only to enrich that list by position. Two consequences: a frame mounted inside a shadow root is invisible to the query while being an ordinary child of the frame tree, so it was dropped entirely; and when the lists disagreed, a light-DOM frame was handed the identity of a shadow frame that preceded it.

The protocol now decides which frames exist, and the two are correlated by frame identity rather than by index. The DOM query still supplies ordering and names for the frames it can see. This adds shadow DOM support to page.frames and frame.child_frames, so widgets that mount their cross-origin iframe that way are reachable.

Tests

Thirteen Rust unit tests: the expression classifier (IIFEs, parenthesised expressions containing arrows, genuine arrow and function literals, keyword-prefixed identifiers, literals/comments/regexes inside the parameter list, nested parameter lists), the refresh budget including the disabled-timeout case, and the scope of the webdriver guard.

cargo check --locked, cargo fmt --check and cargo test --lib pass (107 tests). pytest tests/test_antibot_benchmarks.py passes (15 tests).

The browser-backed pytest files could not run on this machine: the bundled Chromium 151 aborts with SIGTRAP before the CDP endpoint opens, when launched directly from a shell and independently of rustwright, with no missing shared libraries. Per AGENTS.md that verification belongs on a testbox rather than a developer host, so those suites still need a CI run — in particular the worker-identity tests, which this PR changes the mechanism behind. The equivalent assertions were checked by hand against a real browser, as shown above.

End-to-end verification

Against a live challenge through a proxy, with a route handler reading Request.frame on every request:

• before: page.goto failed with Page.goto: Timeout 45000ms exceeded
• with the frame-tree fix: navigation completed, environment rejected
• with the locale fix: still rejected
• with the worker fix: accepted, and the challenge completes
• with the frame-enumeration fix: the widget's iframe is reachable

Three consecutive runs with the full init script active — no blocks skipped — completed successfully. The reproductions are not included here: they need a proxy and a third-party browser build, so they cannot run in CI.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 71fbb0f9ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[DemonMartin]

Fyi, I used it together with https://github.com/clearcotelabs/clearcote-browser to bypass turnstile