[major] Replay auto-heals silently with no opt-out — extraction precision degrades and wrong-target healing can false-pass

[anmolsoin1]

Summary

During testmd run replay, Kane CLI auto-heals locator/extraction failures silently and without any configuration to disable it. Two concrete problems result:

1. Observed: after a semantically-neutral DOM change, the analyzer's extraction precision silently degraded from a single element to a large page blob — and the assertion still passed.
2. Scenario: when multiple elements share a label (e.g. a "Sign In" link in the header nav and a "Sign In" button in the page body), healing can retarget an action to the wrong element — a silent false positive.

Auto-heal in replay should be configurable (opt-out), and healing events should be visible in run output so CI can gate on them.

Environment

• kane-cli: 0.6.4 (npm @testmuai/kane-cli)
• macOS (arm64), Chrome target, headless

Finding 1 — observed: extraction precision degrades under healing, run still reports "passed"

Repro

1. Author a test asserting a page heading, e.g. dashboard shows Welcome back! inside an <h1>. The authored TEXTUAL_ANALYZER step freezes a role-based locator (role=heading[name="Welcome back!"i]) plus the NL query.
2. Change the markup from <h1>Welcome back!</h1> to a <div> styled identically (visually pixel-same page).
3. kane-cli testmd run <test>.md --headless --agent

Actual

Run reports overall_status: passed (replay_decisions: 1, no author decisions). However the extracted memory variable changed character completely — at authoring it captured just the heading text; after the tag change it captured a large blob of the page's main content:

dashboard_welcome_message = "Welcome back!\n\nYou are logged in as demo@acme.shop.\n\n3\n\nOrders\n\n$1,040\n\nTotal Spent\n\n120\n\nReward Points\n\nRecent Orders\n..."

The assertion passed only because "shows 'Welcome back!'" is satisfied as a contains-style match against the blob.

Evidence: session 80937943-b78b-4d0d-851c-35df5f36569b (run_end, variables_out).

Why this is a problem

• The healed extraction is materially fuzzier than the authored one, with no signal in the output that precision changed.
• False-pass risk: if the target text moves to a different component, or the same string exists elsewhere on the page (footer, nav), the assertion can still pass while the actual component under test is broken.

Finding 2 — scenario: wrong-target healing with duplicate labels

Pages commonly have duplicate action labels: e.g. "Sign In" in the header nav and another "Sign In" button in the hero/body. If the body button is removed in a new build, replay healing can retarget the click to the header nav link — the test passes while the component it was written against is gone. This is a silent false positive on exactly the class of regression tests exist to catch.

Expected behavior

1. Configurable healing in replay — e.g. a frontmatter flag (heal: false), a CLI flag (--no-heal), and/or per-step strict mode. Default could stay heal-on, but CI pipelines need a strict mode.
2. Healing visibility — when healing triggers, the run should say so: a distinct step/run status (e.g. passed_healed), an event in --agent NDJSON output, and a count in test_md_summary (alongside replay_decisions). CI must be able to fail or alert on "passed but healed".
3. Extraction drift detection — when an analyzer's healed extraction differs substantially in scope from the authored one (single element → container blob), flag it as a warning at minimum.

Actual behavior

Healing is silent, always on, and indistinguishable from a clean pass in both the console summary and the NDJSON stream.

[anmolsoin1]

Follow-up: both risks confirmed with adversarial repros (kane-cli 0.6.4)

We ran two controlled experiments against the same saved test (acme-login_test.md, asserting verify the dashboard shows 'Welcome back!' — analyzer operator contains, transforms strip):

Experiment A — confirm extraction precision loss

Same run as in the issue body. Comparing the memory variable across runs:

• Authored run: dashboard_welcome_message = "Welcome back!" — precisely the heading text
• Replay after h1→div: dashboard_welcome_message = "Welcome back!\n\nYou are logged in as demo@acme.shop.\n\n3\n\nOrders\n\n..." — entire main-content blob

The analyzer's replay reasoning also drifted: PRIMARY: the welcome message text shown at the top of the dashboard, including the phrase containing 'Welcome back!' — note the goal silently shifted from "extract the heading" to "find the phrase anywhere".

Experiment B — confirm false-pass on a real regression

Setup: changed the heading text itself to Hello again! (the component under test is now wrong) and placed the string Welcome back! only in the page footer.

Result: overall_status: passed. Worse than a blob match — the analyzer extracted the footer itself and treated it as the target:

dashboard_welcome_message = "Welcome back! — © 2026 AcmeShop — demo site for kane-cli testing"

Session: f95f4f7b-65c2-4c04-8ffc-f7ed652e7bd6 (replay_decisions: 1, 8 actions, ~5s, no healing/warning signal in output).

Takeaways

1. In replay, the analyzer satisfies the phrase, not the element — location hints ("top of dashboard, above the stat cards") did not constrain extraction.
2. A genuine regression (heading text changed) produced a green run with zero indication of drift. In a CI gate this ships the bug.
3. Reinforces the asks in the issue: strict/heal-off mode, a passed_healed / drift warning status, and NDJSON events when extraction scope or target changes vs. the authored baseline.

[anmolsoin1]

Verdict summary (closing the loop on both experiments)

Claim 1 — precision loosens under healing: CONFIRMED.

Run	dashboard_welcome_message
Authored	"Welcome back!" — precisely the heading
Replay after h1→div	entire main-content blob (orders, stats, table included)

The analyzer's own replay reasoning shows the goal drifting from "extract the heading" to "find the phrase containing 'Welcome back!' anywhere".

Claim 2 — false-pass on a real regression: CONFIRMED (worse than predicted).

• Heading text changed to Hello again!; Welcome back! present only in the footer
• Result: overall_status: passed — and the analyzer extracted the footer itself as "the top dashboard welcome message text"
• Session: f95f4f7b-65c2-4c04-8ffc-f7ed652e7bd6

Mechanism (from run internals): in replay, text assertions are satisfied at the phrase level, not the element level. Location hints ("top of dashboard, above the stat cards") did not constrain extraction, and the contains operator completes the false-pass. Notably, the strict assertion in the same run (orders table rows: equals + extract_number) behaved correctly — so the leniency is specific to text/contains checks.

Why this matters for suite migration: replay-mode text assertions are phrase-level, not component-level. Migrating existing suites as-is would silently lose regression sensitivity on a very common class of changes (heading tag swapped, text relocated to another component, duplicate labels). Until healing is configurable and drift is surfaced, testmd run results should not be used as a CI gate for text/content regressions.

Asks (unchanged, now evidence-backed):

1. Configurable healing in replay (heal: false frontmatter / --no-heal / per-step strict)
2. passed_healed status + healing/drift counts in test_md_summary and --agent NDJSON events
3. Extraction-scope drift warnings when healed extraction differs materially from the authored baseline

[anmolsoin1]

Evidence bundle

Complete repro package (test file, execution trees, NDJSON run logs, generated code export, and the demo site with step-by-step mutations):

https://gist.github.com/anmolsoin1/e7ceb4f6ca7c6cb4ea2d4cdbc9c301cb

Contents:

• acme-login_test.md — the saved test under review
• execution-authored.json / execution-healed-h1-to-div.json / execution-adversarial-footer.json — full instruction trees for all three runs
• runlog-*.ndjson — --agent output showing dashboard_welcome_message degrading from "Welcome back!" → main-content blob → footer text
• code-export-authored-test.py — the auto-generated Playwright export (baked-in role=heading locator)
• demo-site/ + README with exact repro steps and the two mutations