Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 24 additions & 7 deletions src/content/docs/ruby-gem/guides/backward-compatibility.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ import { Code } from "@astrojs/starlight/components";

This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations.

## 0.25.0

Upgrade notes for configs and integrators targeting gem **0.25.0**:

1. **Removed `link` selector alias** — Custom selector configurations must use the canonical `url` key. The legacy `link` selector alias and exclusivity checks have been removed.
2. **`auto` article limit (`--limit N`)** — Discovery now supports a configurable article limit (`--limit N` CLI flag, default `20`) and `Html2rss.auto_source(url, limit: 20)` in Ruby.
3. **Channel metadata in `auto`** — `auto_source` automatically extracts channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.
4. **Actionable `NoFeedItemsExtracted` error** — `auto` mode fails loud with diagnostic surface classifications (blocked surface, app-shell, unsupported extraction surface) when zero items are found.

## 0.24.0

Upgrade notes for configs and integrators targeting gem **0.24.0**:
Expand All @@ -21,21 +30,28 @@ In previous versions, `html2rss` accepted legacy selector names with a deprecati

| Removed Name | Required Name | Description |
| :----------- | :------------- | :-------------------- |
| `link` | `url` | Item destination URL |
| `pubDate` | `published_at` | Item publication date |
| `updated` | `published_at` | Item publication date |

### Migration Example

Update any occurrences of `pubDate` or `updated` in your `selectors` block to `published_at`:
Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` block to `url` or `published_at`:

<Code
code={`
# Deprecated / Removed
selectors:
link:
selector: "a.title"
extractor: "href"
updated:
selector: ".date"
# Current & Required
selectors:
url:
selector: "a.title"
extractor: "href"
published_at:
selector: ".date"
`}
Expand Down Expand Up @@ -68,9 +84,10 @@ Update any occurrences of `pubDate` or `updated` in your `selectors` block to `p

When upgrading to modern `html2rss` releases:

1. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`.
2. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file.
3. **Pin Browserless when needed**: If you relied on `auto` falling through to Browserless, set `strategy: browserless` (or `--strategy browserless`).
4. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS `<enclosure>`; select podcast/media URLs with `enclosure`.
5. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`.
6. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema.
1. **Rename URL selectors**: Ensure article URL selectors use `url` rather than `link`.
2. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`.
3. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file.
4. **Pin Browserless when needed**: If you relied on `auto` falling through to Browserless, set `strategy: browserless` (or `--strategy browserless`).
5. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS `<enclosure>`; select podcast/media URLs with `enclosure`.
6. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`.
7. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema.
4 changes: 3 additions & 1 deletion src/content/docs/ruby-gem/reference/auto-source.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ To enable it, add `auto_source: {}` to your configuration:
such as `window.__NEXT_DATA__`, `window.__NUXT__`, or `window.STATE`. The JSON-state scraper walks those blobs, finds arrays with
`title`/`url` pairs, and converts them into feed items.

`auto_source` also automatically extracts and prepends kicker/teaser labels to article titles when present.
`auto_source` also automatically extracts and prepends kicker/teaser labels to article titles when present, and discovers channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.

You can also restrict the maximum number of discovered articles via the `limit:` keyword argument in Ruby (`Html2rss.auto_source(url, limit: 10)`) or the `--limit` CLI flag.

**`json_state` Limitations:** the scraper requires discoverable arrays of hashes containing clear `title` and `url` fields. Minified or
obfuscated state objects, heavily encoded values, or blobs that require executing embedded functions are ignored.
Expand Down
2 changes: 2 additions & 0 deletions src/content/docs/ruby-gem/reference/cli-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Automatically discovers items from a page and prints the generated RSS or JSONFe
code={`
html2rss auto https://example.com/articles ; \
html2rss auto https://example.com/articles --format jsonfeed ; \
html2rss auto https://example.com/articles --limit 10 ; \
html2rss auto https://example.com/app --strategy browserless --max-redirects 5 --max-requests 6 ; \
BOTASAURUS_SCRAPER_URL="http://localhost:4010" html2rss auto https://example.com/protected --strategy botasaurus ; \
html2rss auto https://example.com/articles --items_selector ".post-card" ; \
Expand All @@ -38,6 +39,7 @@ Available options:

- `--strategy`: Optional request strategy (`auto`, `faraday`, `browserless`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus`. Pin `browserless` explicitly when you need headless Chrome (preload/interaction).
- `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`.
- `--limit`: Maximum number of articles to extract during discovery (defaults to `20`).
- `--items_selector`: Optional CSS selector hint for item extraction.
- `--max-redirects`: Maximum redirects to follow per request.
- `--max-requests`: Maximum requests to allow for this feed build (defaults to `4` for auto discovery).
Expand Down
Loading