docs(ruby-gem): sync gem 0.27.0 contracts (MCP, capture, Botasaurus)

src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx

@@ -12,8 +12,9 @@ import { Code } from "@astrojs/starlight/components";
 Without MCP, agents must invoke shell commands, manage output buffers, and parse unstructured text. With MCP:
 
 - The agent automatically discovers available tools, arguments, and return types.
-- Responses are structured (JSON Feed objects, configuration schemas, and RSS XML).
+- Responses are structured envelopes (`ok`, `next_step`, `guidance`, `payload`) — articles in `payload.items`, YAML in `payload.yaml`, RSS in `payload.rss`.
 - Agents can inspect, capture, validate, and verify feeds iteratively in a closed feedback loop.
+- `html2rss://runtime` reports whether Botasaurus is configured in this MCP process (boolean only).
 
 ## Client Setup
 
@@ -41,7 +42,10 @@ Add `html2rss` to your Cursor MCP settings (`~/.cursor/mcp.json` or `.cursor/mcp
     "mcpServers": {
       "html2rss": {
         "command": "mise",
-        "args": ["exec", "--", "html2rss", "mcp"]
+        "args": ["exec", "--", "html2rss", "mcp"],
+        "env": {
+          "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010"
+        }
       }
     }
   }
@@ -61,7 +65,10 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp
     "mcpServers": {
       "html2rss": {
         "command": "mise",
-        "args": ["exec", "--", "html2rss", "mcp"]
+        "args": ["exec", "--", "html2rss", "mcp"],
+        "env": {
+          "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010"
+        }
       }
     }
   }
@@ -71,23 +78,25 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp
 
 ## Autonomous Workflow Patterns
 
+**Budget:** `scrape_url` is one call. Durable config is `capture_config` → `validate_config` → `apply_config`. Call `inspect_url` only when the result is weak or you need recon (final URL, status, https→http, native RSS/Atom).
+
 ### Pattern A: One-Shot Content Scraping
 
 When an agent needs articles immediately without saving a feed configuration:
 
-1. The agent calls `scrape_url` with the target URL.
-2. `html2rss` runs auto-source extraction (Schema.org, JSON state, semantic HTML) and returns a JSON Feed items array.
-3. If the page is protected or rendered with JavaScript, the agent calls `inspect_url` to diagnose the structure, then retries `scrape_url` with `strategy: "botasaurus"`.
+1. Call `scrape_url` with `strategy: "auto"`. Auto runs Faraday then Botasaurus when `BOTASAURUS_SCRAPER_URL` is set on the MCP process.
+2. An empty item list is still success for articles-now. Read `payload.items`. Follow `next_step` (including `read_runtime` / `html2rss://runtime` when Botasaurus is unset).
+3. Call `inspect_url` when the result is weak or you need recon (`final_url`, `status`, `scheme_downgrade`, native RSS/Atom).
 
 ### Pattern B: Iterative Feed Config Authoring
 
 When an agent is tasked with creating a durable YAML feed configuration:
 
-1. **Inspect:** The agent calls `inspect_url` to check content type, SST node counts, and eligible scrapers.
-2. **Capture:** The agent runs `capture_config` to derive CSS selectors for items, title, link, and description.
-3. **Refine:** The agent reviews the derived selectors or asks the human user for domain-specific adjustments.
-4. **Validate:** The agent passes the configuration to `validate_config` to verify schema conformance.
-5. **Apply:** The agent tests the final configuration with `apply_config` to produce and inspect live RSS XML.
+1. **Capture:** Call `capture_config`. YAML is `payload.yaml` (`items` + `enhance: true`). Check `payload.articles_count` and `has_selectors`.
+2. **Recon (optional):** Call `inspect_url` for `final_url`, `status`, `scheme_downgrade`, and `alternate_feeds` if the draft is weak.
+3. **Rewrite:** If the destination is html2rss-configs, add `directory.topics` and explicit channel `title`/`url`. Strive to keep `enhance: true` (set `false` only when chrome leaks into items).
+4. **Validate:** Pass `yaml` (or `config`) to `validate_config`. Exactly one of those arguments.
+5. **Apply:** Call `apply_config`. `isError` plus `payload.item_count: 0` means the config is not shippable — channel title in the XML is not success.
 
 ## JavaScript-Rendered Sites (Botasaurus)
 
@@ -100,4 +109,8 @@ For dynamic JavaScript single-page applications or sites protected by anti-bot m
   lang="bash"
 />
 
-Ensure `BOTASAURUS_SCRAPER_URL` is accessible (typically `http://127.0.0.1:4010`) in the environment where the MCP server runs. Agents can then pass `strategy: "botasaurus"` to `scrape_url`, `inspect_url`, and `capture_config`.
+Put `BOTASAURUS_SCRAPER_URL` (typically `http://127.0.0.1:4010`) in the MCP server `env` block above — a shell export does not reach the Cursor/Claude MCP process. Confirm with `html2rss://runtime` (`botasaurus_configured: true`). AutoFallback then hops without a second tool call.
+
+## Daemon logs (stderr)
+
+`html2rss mcp` uses stdout for JSON-RPC, so the daemon logs to **stderr**. Unless you set `LOG_LEVEL`, the MCP process uses `info` (the gem library default stays `warn`). A foreground watcher sees the start banner, each tool call, and pipeline fallbacks. Use `LOG_LEVEL=debug` for more detail or `LOG_LEVEL=warn` to quiet it.

src/content/docs/ruby-gem/guides/backward-compatibility.mdx

@@ -5,7 +5,22 @@ description: "Migration guide and backward-compatibility reference for html2rss
 
 import { Code } from "@astrojs/starlight/components";
 
-This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations.
+This page outlines breaking changes, purged legacy aliases, and migration steps when upgrading the gem.
+
+## 0.27.0
+
+Upgrade notes for configs and integrators targeting gem **0.27.0** (current `master` after `0.26.0`):
+
+1. **MCP result envelope** — Every MCP tool returns `{ ok, next_step, guidance, payload }` in text and `structuredContent`. Read `payload.items`, `payload.yaml`, and `payload.rss`; scrape text is not a raw item array.
+2. **MCP AutoFallback** — `scrape_url` and `capture_config` with `strategy: "auto"` run Faraday → Botasaurus in one call. `inspect_url` with `auto` stays on Faraday (pin `botasaurus` when you need browser rendering for inspect).
+3. **Capture output** — `html2rss capture` and MCP `capture_config` emit an items selector plus `enhance: true` only (no per-field title/url/description selector soup). Capture stamps `strategy:` when AutoFallback selects a concrete transport.
+4. **CLI `--explain`** — `html2rss auto --explain` and `html2rss capture --explain` print telemetry JSON on stderr; stdout stays the feed or YAML. Status includes `admission_drops` when Cleanup dropped items.
+5. **`Html2rss.auto_feed_result`** — URL-only auto discovery returns `FeedResult` with the same `status` telemetry as config builds.
+6. **`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..20`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus).
+7. **Botasaurus wire format** — Client expects OpenAPI 2.0 `ScrapeSuccess` / `ScrapeError` envelopes from the scrape API.
+8. **`--limit` default** — Auto discovery default is `25` (`AutoSource::DEFAULT_LIMIT`).
+
+Run `html2rss validate config.yml` after editing Botasaurus options.
 
 ## 0.26.0
 
@@ -19,7 +34,7 @@ Upgrade notes for configs and integrators targeting gem **0.26.0**:
 Upgrade notes for configs and integrators targeting gem **0.25.0**:
 
 1. **Removed `link` selector alias** — Custom selector configurations must use the canonical `url` key. The legacy `link` selector alias and exclusivity checks have been removed.
-2. **`auto` article limit (`--limit N`)** — Discovery now supports a configurable article limit (`--limit N` CLI flag, default `20`) and `Html2rss.auto_source(url, limit: 20)` in Ruby.
+2. **`auto` article limit (`--limit N`)** — Discovery supports a configurable limit (`--limit N` CLI flag). Default was `20` in 0.25.0; **0.27.0** default is `25` (`Html2rss.auto_source(url, limit: 25)`).
 3. **Channel metadata in `auto`** — `auto_source` automatically extracts channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.
 4. **Actionable `NoFeedItemsExtracted` error** — `auto` mode fails loud with diagnostic surface classifications (blocked surface, app-shell, unsupported extraction surface) when zero items are found.
 
@@ -89,12 +104,17 @@ Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` bl
 
 ## Migration Checklist
 
-When upgrading to modern `html2rss` releases:
+When upgrading to **0.27.0** or later:
+
+1. **MCP integrators**: Read tool results from the envelope `payload`; configure `BOTASAURUS_SCRAPER_URL` on the MCP process env.
+2. **Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..20` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)).
+3. **Validate configurations**: Run `html2rss validate config.yml`.
+
+When upgrading from releases before **0.26.0**:
 
-1. **Migrate Browserless to Botasaurus**: Update any feeds using `strategy: browserless` or `request.browserless` to `strategy: botasaurus` or `strategy: auto` with `BOTASAURUS_SCRAPER_URL`.
-2. **Rename URL selectors**: Ensure article URL selectors use `url` rather than `link`.
-3. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`.
-4. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file.
-5. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS `<enclosure>`; select podcast/media URLs with `enclosure`.
-6. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`.
-7. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema.
+4. **Migrate Browserless to Botasaurus**: Use `strategy: botasaurus` or `strategy: auto` with `BOTASAURUS_SCRAPER_URL`.
+5. **Rename URL selectors**: Use `url` rather than `link`.
+6. **Rename date selectors**: Use `published_at` rather than `updated` or `pubDate`.
+7. **Move channel-level transport keys**: Put `strategy` and `headers` at the top level of the YAML file.
+8. **Use `enclosure` for RSS media**: Select podcast/media URLs with `enclosure`; `image` does not become RSS `<enclosure>`.
+9. **Prefer `feed_result` for dual-format / cache**: Use `Html2rss.feed_result` / `Html2rss.auto_feed_result` and `status.to_h`.

src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx

@@ -5,11 +5,11 @@ description: Derive a reusable YAML feed config from a URL with html2rss capture
 
 import { Aside, Code } from "@astrojs/starlight/components";
 
-`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable feed config with derived CSS selectors. Use it when you want a first draft faster than hand-writing selectors from scratch.
+`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable YAML draft: an **items selector plus `enhance: true`**. At feed-build time, `enhance: true` fills missing article fields from each matched card.
 
 <Aside type="tip" title="Treat capture as a draft">
-  Selector quality depends on page structure. Complex layouts often need a manual `items_selector` hint or
-  small edits before you ship the config.
+  Selector quality depends on page structure. Catalog feeds still need `directory.topics` and explicit channel
+  `title`/`url`. Strive to keep `enhance: true`; set it `false` only when page chrome leaks into items.
 </Aside>
 
 ## When to Capture
@@ -36,6 +36,8 @@ Print YAML to stdout:
   html2rss capture https://example.com/articles --items_selector ".post-card"
   # Analyze a local HTML fixture
   html2rss capture https://example.com/articles --input ./page.html
+  # Quality JSON on stderr; YAML stays on stdout
+  html2rss capture https://example.com/articles --explain
   # Save the draft
   html2rss capture https://example.com/articles > my-feed.yml
 `}
@@ -46,9 +48,10 @@ Common options:
 
 - `--strategy` — `auto`, `faraday`, `botasaurus`, or `local_file` (default `auto`)
 - `--items_selector` — CSS selector hint for item cards
-- `--limit` — maximum articles kept while deriving selectors
+- `--limit` — maximum articles kept while deriving selectors (default `25`)
 - `--max-redirects` / `--max-requests` — request budget overrides
 - `--input` — local HTML file (sets `local_file` strategy)
+- `--explain` — print capture quality JSON on stderr (`articles_count`, `channel_title`, `has_selectors`, `segment_strategy`, `selected_strategy`, `admission_drops`); YAML stays on stdout
 
 See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full flag list.
 
@@ -57,56 +60,47 @@ See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full
 <Code
   code={`
   require 'html2rss'
-  require 'yaml'
-  # Derive a config hash (:channel and :selectors)
+  # Derive a config hash (channel + items selector with enhance: true)
   config = Html2rss.capture('https://example.com/articles')
   # Pin strategy or provide an items hint
   config = Html2rss.capture(
     'https://spa-site.com',
     strategy: :botasaurus,
     items_selector: '.article-card'
   )
-  # Serialize with string keys (same wire form as hand-written YAML)
-  File.write(
-    'my-feed.yml',
-    YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config))
-  )
-  # Use immediately
+  File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
   feed = Html2rss.feed(config)
 `}
   lang="ruby"
 />
 
 ## How It Works
 
-1. **Request** — fetches the page with the chosen strategy
-2. **Discover** — runs AutoSource to extract articles
-3. **Analyze** — normalizes the page into an SST document and maps segment positions back to articles
-4. **Derive** — builds CSS selectors from SST tag paths for items, title, link, and description
-5. **Assemble** — returns a config hash ready for YAML or `Html2rss.feed`
+1. **Request** — `FeedPipeline` (AutoFallback when `:auto`)
+2. **Discover** — AutoSource extracts admitted articles
+3. **Segment** — SST Segmenter strategies `:list` → `:cluster` → `:semantic`
+4. **Gate** — emit an items selector only when enough articles match
+5. **Assemble** — `{ items: { selector:, enhance: true } }` plus channel. When AutoFallback selects a concrete transport (or you pin one), Capture **stamps** `strategy:` into the YAML so later `html2rss feed` / `Html2rss.feed` replay the same transport.
 
-Capture segment discovery currently uses the list Segmenter strategy only (not AutoSource cluster/semantic heuristics). When the draft is weak, pass `--items_selector` or refine selectors by hand.
+When the quality gate fails, selectors are omitted (`has_selectors: false`) rather than inventing attribute selectors. Hint with `--items_selector` or refine by hand.
 
 ## What Capture Derives
 
 Capture focuses on:
 
 - `channel.url` (and related channel defaults)
-- `selectors.items`
-- `selectors.title`
-- `selectors.url` (derived href selector)
-- `selectors.description` when a distinct description root exists
+- `selectors.items` with `enhance: true`
 
-It does not invent author, `published_at`, categories, or enclosure selectors. Add those manually when the page exposes them reliably.
+It does not invent per-field title/url/description selectors, author, `published_at`, categories, or enclosure selectors. `enhance: true` fills missing article fields via the HTML article extractor at feed-build time.
 
-Description is omitted when it would resolve to the invalid CSS selector `.` (item root equals description root).
+MCP `capture_config` returns that YAML in `payload.yaml`. `validate_config` / `apply_config` accept the YAML string (XOR a config hash). `apply_config` is `isError` when the feed has zero items (`payload.item_count`).
 
 ## Recommended Follow-Up
 
 1. Validate: `html2rss validate my-feed.yml`
 2. Render: `html2rss feed my-feed.yml`
-3. Tighten selectors, strategy, or `request.botasaurus` options if needed
-4. For Feed Directory contributions, add `directory.topics` (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config))
+3. Tighten the items selector, strategy, or `request.botasaurus` options if needed
+4. For Feed Directory contributions, add `directory.topics` and keep `enhance: true` unless chrome leaks (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config))
 
 ## Related
 

src/content/docs/ruby-gem/guides/custom-http-requests.mdx

@@ -73,7 +73,7 @@ Request budgets are configured under `request`, not as top-level keys:
 
 - `request.max_redirects` limits redirect hops
 - `request.max_requests` limits the total request budget for the feed build
-- `request.botasaurus.*` is reserved for Botasaurus-only behavior such as navigation mode, retries, header/cookie forwarding, and scroll options
+- `request.botasaurus.*` is reserved for Botasaurus-only behavior such as navigation mode, retries, header/cookie forwarding, and `scroll`
 
 ## Common Use Cases
 

src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx

@@ -22,7 +22,7 @@ When a site requires browser rendering or anti-bot bypass by default, you can ex
     botasaurus:
       navigation_mode: google_get_bypass
       wait_for_selector: ".article-card"
-      scroll_to_bottom: true
+      scroll: true
   channel:
     url: https://example.com/app
   selectors:
@@ -44,7 +44,7 @@ Botasaurus extraction is ideal when:
 - **Content loads after page load** - JavaScript fetches data dynamically
 - **Single Page Applications (SPAs)** - React, Vue, Angular apps
 - **Anti-bot interstitials** - Cloudflare, Google search referrers, or challenge pages
-- **Scroll-triggered content** - Infinite scroll lists (`scroll_to_bottom: true`)
+- **Scroll-triggered content** - Infinite scroll lists (`scroll: true`)
 
 ## Botasaurus Request Controls
 
@@ -70,7 +70,7 @@ Configure browser actions under `request.botasaurus`:
   strategy: botasaurus
   request:
     botasaurus:
-      scroll_to_bottom: true
+      scroll: true
   `}
   lang="yaml"
 />

src/content/docs/ruby-gem/guides/managing-feed-configs.mdx

@@ -76,7 +76,9 @@ Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON F
 - `to_rss` / `to_json_feed(feed_url:)` — render formats
 - `status` — scrape telemetry (`Html2rss::Status`)
 
-`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`).
+`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`), and `admission_drops` (Cleanup reason → count).
+
+For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same `FeedResult` (CLI `html2rss auto --explain` prints `status.to_h` on stderr).
 
 <Code
   code={`