feat: I dump the DOM, I dump the DOM of, on_failure dom

README.md

@@ -46,7 +46,7 @@ resources:
       # headless: false                  (default true; or drive it from an env var: headless: ${BROWSER_HEADLESS:-true})
       # window: 1280x800                 (default)
       # find_timeout_secs: 5             (default; 0 disables the wait)
-      # on_failure: screenshot,console   (default: screenshot,console,network; "none" writes nothing)
+      # on_failure: screenshot,console   (default: screenshot,console,network; add dom for the page source; "none" writes nothing)
       # capabilities: {}                 (raw WebDriver capabilities merged on top of what the plugin builds)
     firefox:                           # managed mode: no url
       browser: firefox
@@ -116,7 +116,7 @@ Selectors are CSS by default; prefix with `xpath=` for an XPath expression, or `
 | 11 | action | `I execute the script:` | runs the doc string as JavaScript in the page; the return value becomes `<<script_result>>` |
 | 12 | action | `I read the "<selector>" element text as "<name>"` | stores an element's rendered text in a variable |
 | 13 | action | `I read the "<attr>" attribute of "<selector>" as "<name>"` | stores an element's attribute in a variable; a missing attribute fails |
-| 14 | action | `I take a screenshot` | writes a PNG of the page into the artifacts directory |
+| 14 | action | `I take a screenshot` | writes a PNG of the page as `screenshot-<n>.png` into the artifacts directory |
 | 15 | assertion | `I should be on "<path>"` | the current URL's path and query (or the whole absolute URL) equal this |
 | 16 | assertion | `the page title should be "<text>"` | the document title equals this exactly |
 | 17 | assertion | `the page should contain "<text>"` | the rendered text of the page contains this |
@@ -134,14 +134,16 @@ Selectors are CSS by default; prefix with `xpath=` for an XPath expression, or `
 | 29 | assertion | `the browser console should have no errors` | no console.error and no uncaught exception since the scenario started |
 | 30 | assertion | `the browser should have sent a "<method>" request to "<path>"` | some request of the scenario has this method and a path starting with this |
 | 31 | assertion | `the last request to "<path>" should have status "<code>"` | the most recent request whose path starts with this has completed with this status |
+| 32 | action | `I dump the DOM` | writes the page's serialized HTML as `dom-<n>.html` into the artifacts directory |
+| 33 | action | `I dump the DOM of "<selector>"` | writes one element's `outerHTML` as `dom-<n>.html`; the selector is CSS, `xpath=` or `text=`, and the element must already be there |
 
 ## Waits
 
-An action (0–14) waits up to `find_timeout_secs` for its element to appear, polling every 100 ms; `0` disables the wait and the action fails immediately on a missing element. An assertion (15–25, 29–31) looks exactly once — it never polls on its own — so a condition that has not settled yet (a click that triggers an async render, a request still in flight) is armed with the host's own eventual assertion: `I expect the next assertion to pass within "N" seconds`. The dump/read-status steps (26–28) read the console and network buffers as they stand at that instant and never wait either — arm an eventual assertion (30 or 31) first when the request they need might still be in flight.
+An action (0–14) waits up to `find_timeout_secs` for its element to appear, polling every 100 ms; `0` disables the wait and the action fails immediately on a missing element. An assertion (15–25, 29–31) looks exactly once — it never polls on its own — so a condition that has not settled yet (a click that triggers an async render, a request still in flight) is armed with the host's own eventual assertion: `I expect the next assertion to pass within "N" seconds`. The dump/read-status steps (26–28) read the console and network buffers as they stand at that instant and never wait either — arm an eventual assertion (30 or 31) first when the request they need might still be in flight. The DOM dumps (32–33) take the page as it is at that instant too, and `I dump the DOM of` fails at once, without waiting, on a missing element — arm an eventual assertion about the element first when it may not have rendered yet.
 
 ## Evidence
 
-A failed step dumps the current page's URL and title, always; a screenshot (PNG, path printed), the console log and the network log, unless `on_failure` says otherwise. `on_failure` is a comma-separated subset of `screenshot`, `console`, `network`, or `none` to write nothing beyond the URL and title.
+A failed step dumps the current page's URL and title, always; a screenshot (`screenshot-<n>.png`, path printed), the console log and the network log, unless `on_failure` says otherwise. `on_failure` is a comma-separated subset of `screenshot`, `console`, `network`, `dom`, or `none` to write nothing beyond the URL and title. `dom` is off by default: it writes the page source as `dom-<n>.html` and prints its path, and that file is raw (see Known limits).
 
 ## Console and network
 
@@ -164,6 +166,8 @@ One browser session per feature file, opened on its first browser step and close
 3. **No iframes, tabs or alerts.** Every lookup runs against the top-level document of the current tab; there is no step to switch frames, open or close a tab, or handle a native `alert`/`confirm`/`prompt`.
 4. **Managed mode is Unix-only.** `Mode::Managed` on a non-Unix build fails naming `url` as the way forward; remote mode is unaffected everywhere.
 
+5. **DOM dumps are the serialized document, written raw.** `I dump the DOM` and `on_failure: dom` use the WebDriver page-source endpoint, which leaves out shadow roots and iframe contents; `I dump the DOM of` is the element's `outerHTML`, which has the same blind spot. Nothing is masked: hidden fields, CSRF tokens and attribute values end up in the file as they are, so keep the artifacts directory out of anything shared. `<n>` counts the dumps of that kind in the scenario, from 1, and starts over at the next scenario.
+
 ## Example
 
 A full demo — a site, five feature files, a compose file — lives under `examples/`; see `examples/README.md` for how to run it.

examples/README.md

@@ -15,7 +15,7 @@ The site is plain HTML and vanilla JavaScript in `examples/site/static/`; edit a
 | `features/login.feature` | `I am on`, `I fill in` by label / name, `I press`, `I should be on` with a query string, the per-scenario reset (a login never leaks into the next scenario) |
 | `features/order.feature` | The bridge: a click in the browser, an eventual assertion that the browser itself sent the `POST /api/orders` request and it answered `201`, the status read into a variable, a clean console, the order id read into a variable, then the host's own HTTP steps read the same order from the API |
 | `features/forms.feature` | select, checkbox, textarea, attribute and script results as variables, a form round trip |
-| `features/evidence.feature` | `I take a screenshot` and where it lands under `I am in debug mode` |
+| `features/evidence.feature` | `I take a screenshot`, `I dump the DOM` and `I dump the DOM of` and where they land under `I am in debug mode`; a second dump of a kind gets the next `-<n>` |
 | `features/console.feature` | `the browser console should have no errors` on a clean page, then `I dump the browser console` / `I dump the network log` for their own sake |
 
 ## Watching the browser

examples/browser.yaml

@@ -12,6 +12,7 @@ resources:
       url: http://localhost:4444        # docker compose up -d
       base_url: http://localhost:3000   # cargo run --example site
       headless: ${BROWSER_HEADLESS:-true}   # BROWSER_HEADLESS=false to watch
+      # on_failure: screenshot,console,network,dom   # add dom to get the page source on a failed step
     firefox:                            # managed mode: no url, the plugin brings the browser
       browser: firefox
       base_url: http://localhost:3000

examples/features/evidence.feature

@@ -5,3 +5,19 @@ Feature: evidence on demand
     And I am in debug mode
     When I take a screenshot
     Then the "text=Sign in" element should be visible
+
+  Scenario: the page source and one element, for writing a selector
+    Given I am on "/login"
+    And I am in debug mode
+    When I dump the DOM
+    And I dump the DOM of "h1"
+    Then the "text=Sign in" element should be visible
+
+  Scenario: a second dump of a kind gets the next number
+    Given I am on "/login"
+    And I am in debug mode
+    When I take a screenshot
+    And I take a screenshot
+    And I dump the DOM
+    And I dump the DOM
+    Then the "text=Sign in" element should be visible

src/config.rs

@@ -102,6 +102,7 @@ pub struct OnFailure {
     pub screenshot: bool,
     pub console: bool,
     pub network: bool,
+    pub dom: bool,
 }
 
 impl OnFailure {
@@ -110,16 +111,18 @@ impl OnFailure {
             screenshot: false,
             console: false,
             network: false,
+            dom: false,
         };
         for token in s.split(',').map(str::trim).filter(|t| !t.is_empty()) {
             match token {
                 "screenshot" => on.screenshot = true,
                 "console" => on.console = true,
                 "network" => on.network = true,
+                "dom" => on.dom = true,
                 "none" => {}
                 other => {
                     return Err(format!(
-                        "\"on_failure\" token {other:?} is not one of screenshot, console, network, none"
+                        "\"on_failure\" token {other:?} is not one of screenshot, console, network, dom, none"
                     ));
                 }
             }
@@ -243,7 +246,7 @@ const FIELDS: &[Field] = &[
         name: "on_failure",
         required: false,
         value_type: None,
-        description: "comma-separated subset of screenshot, console, network to write on a failed step, or none; defaults to all three",
+        description: "comma-separated subset of screenshot, console, network, dom to write on a failed step, or none; defaults to screenshot,console,network (dom is off: the page source is written as is, hidden fields and tokens included)",
         example: Some("screenshot,console"),
     },
     Field {
@@ -437,6 +440,7 @@ impl InstanceConfig {
                 screenshot: true,
                 console: true,
                 network: true,
+                dom: false,
             },
             Some(s) => OnFailure::parse(&s)?,
         };
@@ -497,6 +501,7 @@ mod tests {
         assert_eq!(c.window, (1280, 800));
         assert_eq!(c.find_timeout, Duration::from_secs(5));
         assert!(c.on_failure.screenshot && c.on_failure.console && c.on_failure.network);
+        assert!(!c.on_failure.dom, "the DOM is written raw, so it is opt-in");
         assert!(c.base_url.is_none());
     }
 
@@ -555,6 +560,9 @@ mod tests {
         body["on_failure"] = json!("none");
         let c = InstanceConfig::parse(&body).expect("valid");
         assert!(!c.on_failure.screenshot && !c.on_failure.console && !c.on_failure.network);
+        body["on_failure"] = json!("dom");
+        let c = InstanceConfig::parse(&body).expect("dom");
+        assert!(c.on_failure.dom && !c.on_failure.screenshot);
         body["on_failure"] = json!("screenshot");
         let c = InstanceConfig::parse(&body).expect("valid");
         assert!(c.on_failure.screenshot && !c.on_failure.console);

src/instance.rs

@@ -3,6 +3,7 @@
 
 use std::path::PathBuf;
 use std::sync::Arc;
+use std::sync::atomic::{AtomicU32, Ordering};
 #[cfg(unix)]
 use std::sync::{Mutex, PoisonError};
 use std::time::Duration;
@@ -29,6 +30,10 @@ pub struct Instance {
     pub session: Session,
     pub info: SessionInfo,
     pub bidi: Option<Bidi>,
+    /// How many screenshots and DOM dumps this scenario has written, so a
+    /// file name says which one it is. Cleared by `reset`.
+    pub screenshots: AtomicU32,
+    pub doms: AtomicU32,
     /// The driver process this instance started, managed mode only.
     #[cfg(unix)]
     driver: Mutex<Option<ManagedDriver>>,
@@ -79,6 +84,8 @@ impl Instance {
             session,
             info,
             bidi: None,
+            screenshots: AtomicU32::new(0),
+            doms: AtomicU32::new(0),
             #[cfg(unix)]
             driver: Mutex::new(_driver),
         };
@@ -126,6 +133,8 @@ impl Instance {
         self.session
             .navigate("about:blank")
             .map_err(|e| format!("leaving the page: {e}"))?;
+        self.screenshots.store(0, Ordering::Relaxed);
+        self.doms.store(0, Ordering::Relaxed);
         if let Some(bidi) = &self.bidi
             && let Ok(mut b) = bidi.buffers().lock()
         {

src/reply.rs

@@ -59,6 +59,16 @@ impl Diagnostic {
         }
     }
 
+    /// A file the plugin wrote: the host prints the path, the reader opens it.
+    pub fn file(title: impl Into<String>, kind: &'static str, path: impl Into<String>) -> Self {
+        Self {
+            title: title.into(),
+            kind,
+            content: None,
+            path: Some(path.into()),
+        }
+    }
+
     fn to_json(&self) -> Value {
         json!({
             "title": self.title,

src/steps.rs

@@ -2,6 +2,7 @@
 //! cannot drift. The index of a step in `STEPS` is its identity.
 
 use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicU32, Ordering};
 
 use serde_json::{Map, Value};
 use url::Url;
@@ -93,7 +94,7 @@ pub const STEPS: &[Step] = &[
     Step {
         pattern: r#"^I take a screenshot$"#,
         kind: "action",
-        description: "writes a PNG of the page into the artifacts directory",
+        description: "writes a PNG of the page as screenshot-<n>.png into the artifacts directory",
     },
     Step {
         pattern: r#"^I should be on "(?P<path>[^"]+)"$"#,
@@ -180,6 +181,16 @@ pub const STEPS: &[Step] = &[
         kind: "assertion",
         description: "the most recent request whose path starts with this has completed with this status",
     },
+    Step {
+        pattern: r#"^I dump the DOM$"#,
+        kind: "action",
+        description: "writes the page's serialized HTML as dom-<n>.html into the artifacts directory; shadow roots and iframe contents are not in it",
+    },
+    Step {
+        pattern: r#"^I dump the DOM of "(?P<selector>[^"]+)"$"#,
+        kind: "action",
+        description: "writes one element's outerHTML as dom-<n>.html; the element must be there already, the step never waits",
+    },
 ];
 
 pub fn steps_json() -> String {
@@ -345,6 +356,14 @@ fn write_artifact(ctx: &Ctx, name: &str, bytes: &[u8]) -> Result<PathBuf, Fail>
     Ok(path)
 }
 
+/// `<stem>-<n>.<ext>`, `n` counting up from 1 within the scenario.
+fn numbered(counter: &AtomicU32, stem: &str, ext: &str) -> String {
+    format!(
+        "{stem}-{}.{ext}",
+        counter.fetch_add(1, Ordering::Relaxed) + 1
+    )
+}
+
 /// An action's element: wait up to `find_timeout`, then fail naming it.
 fn act<'a>(instance: &'a Instance, lookup: &Lookup) -> Result<Element<'a>, Fail> {
     let timeout = instance.config.find_timeout;
@@ -442,7 +461,8 @@ fn run(instance: &Instance, index: u32, req: &Request) -> Result<Map<String, Val
         }
         14 => {
             let png = s.screenshot()?;
-            let path = write_artifact(&req.ctx, "screenshot.png", &png)?;
+            let name = numbered(&instance.screenshots, "screenshot", "png");
+            let path = write_artifact(&req.ctx, &name, &png)?;
             if req.ctx.debug {
                 eprintln!("[browser] screenshot: {}", path.display());
             }
@@ -621,6 +641,21 @@ fn run(instance: &Instance, index: u32, req: &Request) -> Result<Map<String, Val
                 Some(_) => {}
             }
         }
+        32 | 33 => {
+            let html = if index == 32 {
+                s.source()?
+            } else {
+                let lookup = find::selector(&arg(0));
+                look(instance, &lookup)?
+                    .ok_or_else(|| fatal(format!("no {} on the page", lookup.what)))?
+                    .outer_html()?
+            };
+            let name = numbered(&instance.doms, "dom", "html");
+            let path = write_artifact(&req.ctx, &name, html.as_bytes())?;
+            if req.ctx.debug {
+                eprintln!("[browser] dom: {}", path.display());
+            }
+        }
         other => return Err(fatal(format!("unknown step index {other}"))),
     }
     Ok(vars)
@@ -645,15 +680,30 @@ fn evidence(instance: &Instance, req: &Request) -> Vec<Diagnostic> {
             .session
             .screenshot()
             .map_err(|e| fatal(e.to_string()))
-            .and_then(|png| write_artifact(&req.ctx, "screenshot.png", &png))
-        {
+            .and_then(|png| {
+                let name = numbered(&instance.screenshots, "screenshot", "png");
+                write_artifact(&req.ctx, &name, &png)
+            }) {
             Ok(path) => out.push(Diagnostic::image("Screenshot", path.display().to_string())),
             Err(e) => out.push(Diagnostic::text(
                 "Screenshot",
                 format!("not taken: {}", e.error),
             )),
         }
     }
+    if instance.config.on_failure.dom {
+        match instance
+            .session
+            .source()
+            .map_err(|e| fatal(e.to_string()))
+            .and_then(|html| {
+                let name = numbered(&instance.doms, "dom", "html");
+                write_artifact(&req.ctx, &name, html.as_bytes())
+            }) {
+            Ok(path) => out.push(Diagnostic::file("DOM", "html", path.display().to_string())),
+            Err(e) => out.push(Diagnostic::text("DOM", format!("not taken: {}", e.error))),
+        }
+    }
     if let Some(bidi) = &instance.bidi
         && let Ok(b) = bidi.buffers().lock()
     {
@@ -685,8 +735,8 @@ mod tests {
         let steps: Vec<Value> = serde_json::from_str(&steps_json()).expect("JSON");
         assert_eq!(
             steps.len(),
-            32,
-            "phase 1 declares 26 steps, task 11 adds 6 more; append, never reorder"
+            34,
+            "phase 1 declares 26 steps, task 11 adds 6, issue 96 adds 2; append, never reorder"
         );
         for (i, s) in steps.iter().enumerate() {
             let p = s["pattern"].as_str().expect("pattern");

src/webdriver.rs

@@ -339,6 +339,11 @@ impl Session {
         self.post("execute/sync", json!({"script": script, "args": args}))
     }
 
+    /// The serialized document. Shadow roots and iframe contents are not in it.
+    pub fn source(&self) -> Result<String, Error> {
+        Ok(self.get("source")?.as_str().unwrap_or("").to_string())
+    }
+
     pub fn screenshot(&self) -> Result<Vec<u8>, Error> {
         let encoded = self.get("screenshot")?;
         let encoded = encoded.as_str().unwrap_or("");
@@ -400,6 +405,14 @@ impl Element<'_> {
         self.session.get(&self.tail(&format!("property/{name}")))
     }
 
+    pub fn outer_html(&self) -> Result<String, Error> {
+        let reference = json!({ELEMENT_KEY: self.id});
+        let html = self
+            .session
+            .execute("return arguments[0].outerHTML;", vec![reference])?;
+        Ok(html.as_str().unwrap_or("").to_string())
+    }
+
     pub fn displayed(&self) -> Result<bool, Error> {
         Ok(self
             .session