Support Capybara's finder hints in find_css and find_xpath by joeljunstrom · Pull Request #347 · rubycdp/cuprite

joeljunstrom · 2026-10-19T09:02:08Z

Capybara passes uses_visibility, texts, styles and position to drivers whose finders accept them, and reads the answers from Node#initial_cache. Cuprite's finders took only the selector, so Capybara asked every candidate for its visibility and text in round trips of its own.

The browser now answers the hints in the same call that finds the nodes:

Text is filtered case-sensitively against all text or visible text, the strings Capybara compares, so nothing Capybara would match is dropped. Selenium's lowercased textContent filter misses a word split by a hidden element. A node holding the text vouches for its ancestors, so nested candidates don't read the same text over and over.
Candidates found together share one cache of ancestor visibility, so deeply nested candidates don't each walk the whole tree.
The answers come back as one list per hint next to the nodes, and visibility as the indexes of the hidden nodes. Chrome takes time to serialize every object it returns, so a [node, cache] pair per node made first over many candidates slower than without hints.
When no node holds the text, the first 50 come back with their texts, so Capybara's failure messages read as before: Unable to find visible css "li" with text "nope", and assert_selector still lists what else it found.

Like Selenium's, the hints are taken when the nodes are found. Capybara filters a result lazily, so a candidate it hasn't reached yet keeps its hints if the page changes in between.

benchmark-ips on Ferrum 0.20.1 and Chrome 157, interleaved runs, ms per call. Results and the nodes found are the same on both sides.

Page	Query	main	this PR	CDP commands
300 items, every third hidden	`find("li", text: "Item 298")`	383ms	2.1ms	2305 → 10
	`all("li", minimum: 1)`	242ms	5.2ms	1501 → 1
	`assert_selector("li", text: "Item 2", count: 75)`	371ms	119ms	2301 → 456
	`first("li", text: "Item 298", below: …)`	369ms	2.8ms	2299 → 14
	`find("li", text: "nope")`, failing	372ms	1.4ms	2301 → 1
	`all("li", visible: :all)`	3.8ms	3.8ms	1 → 1
	in an iframe: `find("li", text: …)`	373ms	2.6ms	2312 → 12
2000 rows, 10% hidden	`all(".x", minimum: 1)`	1615ms	29ms	10001 → 1
	`all(".x", style: { display: "block" })`	2339ms	34ms	13601 → 1
	`has_css?(".x", text: "row 1999")`	2761ms	3.3ms	17201 → 6
	`has_no_css?("li", text: "nope")`	2766ms	3.5ms	17201 → 1
	`within("#list") { find("li", text: "row 1999") }`	2741ms	5.5ms	17214 → 15
	`first(".x")`	23.1ms	23.1ms	15 → 6
1000 nested divs, text in the leaf	`first("div", text: "needle")`	12.9ms	13.0ms	15 → 11
	`all("div", text: "needle")`	1868ms	1067ms	9010 → 5006
200 sections of 5 divs	`first("div.c", text: "lorem 199-4")`	1512ms	3.5ms	9005 → 10
	`all("div", text: "lorem")`	1878ms	1204ms	10810 → 6006
A small form	`find("#save")`	1.6ms	1.1ms	10 → 6
	`find(:link, "Settings")`	1.8ms	1.1ms	10 → 6
	`find_field("Email")`	2.6ms	2.0ms	15 → 11

No query got slower, and Ruby allocations are at or below main's. Queries with text: still cost a few commands per candidate, because Capybara has no cache key for text and asks each node for it.

On a Rails app's 1509 system tests, the suite sends 23% fewer CDP commands (428,992 → 330,439) with the same outcomes. Wall time doesn't change measurably (346s → 344s), because most finds match few elements.

navidemad

I compared this with the finder-hint work in capybara-lightpanda and checked the PR against its base using Capybara 3.40.0, Chrome 156, Ferrum main fb7d858, and released Ferrum 0.19.1.

The selective text-search improvement is substantial. I would address the two performance regressions below before merging. The separate lazy-cache comment is non-blocking, given the existing Selenium precedent.

The eight added finder-hint examples pass in a focused harness using their original test block and fixture. Targeted base/PR reproductions and measurements were independently rerun. The full upstream suite was not rerun locally. Timings are local measurements of a synthetic worst case, not general performance guarantees.

navidemad · 2026-10-26T09:34:28Z

+      let nodes = this.find(method, selector, within).filter(node => node.nodeType == Node.ELEMENT_NODE);
+
+      if (hints.texts.length > 0) {
+        nodes = nodes.filter(node => this.containsTexts(node, hints.texts, hints.visible));


[P2] Bound eager hint work for large candidate sets

first can stop filtering after its first match, but this path computes text hints and visibility for every retained candidate. In a synthetic worst-case fixture with 1,000 nested divs containing needle at the leaf, page.first(:css, "div", text: "needle", wait: 0) had local three-run medians of approximately 13 ms on base and 174 ms on this branch with Ferrum main fb7d858, excluding DOM construction. The visibility ancestor walks alone perform quadratic work: instrumenting getComputedStyle during hint collection counted 5,250 / 20,500 / 81,000 calls for 100 / 200 / 400 nested candidates.

An experimental fallback above 512 candidates reduced the median to approximately 17 ms. Could we bound eager work and leave larger candidate sets to Capybara’s lazy filtering? The cutoff is experimental; a fallback would still need to exclude non-elements and preserve the node/cache representation.

I'll look into it! Had issues running the local suite for some reason after rebasing so it was a bit of a shot in the dark.

afa3620 shares ancestor visibility results across candidates, 437e673 lets a text match vouch for its node's ancestors, and 60a178f returns the answers as one list per hint instead of a pair per node. No cap needed: your nested first case is 13.1 ms against 13.0 ms on main.

navidemad · 2026-10-26T09:34:29Z

+          position: hints.fetch(:position, false)
+        }
+
+        evaluate("_cuprite.findWithHints(...arguments)", *[method, selector, hints, within].compact)


[P2] Avoid nested transport overhead on supported Ferrum 0.19.1

The gemspec allows Ferrum ~> 0.19.1, whose serializer recursively fetches object properties. Returning [node, {}] pairs adds two Runtime.getProperties requests per match even when no hints are requested. For 300 list items, page.all("li", visible: :all).size increases from 303 to 903 CDP commands; direct Browser#find has the same regression.

Ferrum main’s deep serializer avoids this (both variants use one command), but the Gemfile’s main-branch dependency does not protect released-gem users. Could the no-hints path transport a plain node array and construct empty-cache pairs in Ruby, or require a released Ferrum version containing deep serialization before shipping?

Probably very reasonable to have this as ”on if available” kind of deal so checking the Ferrum capabilities sounds like a good idea

60a178f returns just the nodes when no hint needs a per-node answer.

navidemad · 2026-10-26T09:34:29Z

+      return nodes.map(node => {
+        let cache = {};
+        if (hints.visible) {
+          cache.visible = this.isVisible(node);


[P3, non-blocking] Make the changed lazy-result semantics explicit

This changes how not-yet-filtered candidates behave after the DOM changes. Starting with <div id="first">first</div><div id="second">second</div>:

result = page.all("div", minimum: 1) page.execute_script("document.querySelector('#second').style.display = 'none'") result.map { |node| node[:id] }

Base returns ["first"] while this branch returns ["first", "second"]. Style and position filters behave similarly because later candidates retain their initial hints.

Capybara’s Selenium optimization already uses these caches for larger candidate sets, so this is a compatibility decision rather than an unsupported use of initial_cache. Could we explicitly test and document the intended snapshot behavior?

Fair, will look into it

ceaee43 uses a snapshot, as Selenium does.

route · 2026-10-26T13:04:56Z

I fixed main branch, due to JS API changes. Can you rebase?

route · 2026-11-07T18:27:02Z

good potential in the PR but we should move

Capybara passes uses_visibility, texts, styles and position to finders that take them and reads the answers from Node#initial_cache. Without them it asked each candidate for its visibility and text in round trips of its own. One evaluate returns each node with its answers, which Ferrum's deep serialization brings back in a single call. Texts drop a node only when neither all its text nor its visible text holds them, case sensitively, as Capybara matches. Browser#find and #find_within still return nodes.

Each candidate walked all its ancestors to answer the visibility hint, so nested candidates cost quadratic getComputedStyle calls: 1000 nested divs took 225ms against 12.5ms before the hints. Candidates found together now share one cache of ancestor results, so each element is checked once. An element inside a modal dialog or open popover ignores its ancestors' opacity, so results are cached apart for both cases.

Capybara filters a result lazily, so candidates it hasn't reached yet are judged by the hints from when they were found, even if the page changed since. Selenium's finder hints behave the same way. A spec now covers it and the changelog says so.

Chrome takes time to serialize each object and value an evaluate returns, so a [node, cache] pair per node made first(".x") over 2000 rows 50% slower than without hints, and all("li", visible: :all) 29% slower even though it asks for nothing. The answers now come back as one list per hint next to the nodes, and visibility only as the indexes of the hidden nodes. Both cases are now as fast as without hints.

The text filter read every candidate's textContent, which on nested candidates reads the same text again for each ancestor: first("div.c", text: ...) over 1200 nested divs spent most of its 49ms there. A node's text is part of its ancestors' text, so the filter now walks from the last node first and marks a match's ancestors as matches. The visible text fallback shares the visibility cache, so it no longer walks every ancestor again either.

joeljunstrom · 2026-11-11T12:51:38Z

Made a second pass trying to address the regressions mentioned. Seems more stable now with improvements across the board (of what I measured). Due to lack of insight I do not know what I didn't, but should have, measure though ^^

PR description updated with new measurements.

Ferrum's main branch awaits a returned plain object in case it's a promise, which costs a second round trip, so every find took two CDP commands where main takes one. An app's 1509 system tests sent 40,618 extra commands for it.

Capybara describes a failed find from the nodes it filtered out itself. With the text filtered in the browser it got none back, so `Unable to find visible css "li" with text "nope"` became `Unable to find css "li"`, and assert_selector no longer listed what else it found. When no node holds the texts, the first 50 now come back with the texts Capybara reads to filter and describe them, so the messages are the same as before at about 1ms more on 2000 rows.

The finder returned a [node, cache] pair per node for the callers to build nodes from, two objects more per node than without hints: all("li", visible: :all) over 300 nodes was 6% slower than before the hints. Callers now pass a block that gets each node and its cache, and allocations are back to the level before the hints.

joeljunstrom force-pushed the find-hints branch 2 times, most recently from 5ed6188 to 3fe08cc Compare October 26, 2026 08:44

navidemad suggested changes Oct 26, 2026

View reviewed changes

joeljunstrom added 3 commits November 11, 2026 09:19

joeljunstrom force-pushed the find-hints branch from 3fe08cc to ceaee43 Compare November 11, 2026 10:09

joeljunstrom added 2 commits November 11, 2026 13:09

joeljunstrom requested a review from navidemad November 11, 2026 12:39

joeljunstrom added 3 commits November 11, 2026 14:05

Return finder hint answers in an array, not an object

40c1fab

Ferrum's main branch awaits a returned plain object in case it's a promise, which costs a second round trip, so every find took two CDP commands where main takes one. An app's 1509 system tests sent 40,618 extra commands for it.

Conversation

joeljunstrom commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

navidemad left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

route commented Oct 26, 2026

Uh oh!

route commented Nov 7, 2026

Uh oh!

joeljunstrom commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joeljunstrom commented Oct 19, 2026 •

edited

Loading

joeljunstrom commented Nov 11, 2026 •

edited

Loading