Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 22 additions & 12 deletions src/content/docs/creating-custom-feeds.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Advanced Feeds (Escape Hatch)"
description: "Write YAML configs when auto-source is not enough; mount feeds.yml; contribute to html2rss-configs."
description: "Write YAML configs when auto-source is not enough; add them with registry add; contribute to html2rss-configs."
sidebar:
order: 2
---
Expand All @@ -12,7 +12,7 @@ Prefer [Getting Started](/getting-started/) (URL paste) and the [Feed Directory]
<Aside type="tip" title="Escape hatch">
Agents can draft configs via MCP (`capture` → `test` → `apply`) or the
[`html2rss-config`](https://github.com/html2rss/html2rss-configs/tree/master/.agents/skills/html2rss-config)
skill. You still mount or publish the YAML yourself.
skill. You still add the YAML with `registry add` or publish it yourself.
</Aside>

## Recommended workflow
Expand All @@ -21,15 +21,19 @@ Prefer [Getting Started](/getting-started/) (URL paste) and the [Feed Directory]
2. Optionally draft: `html2rss capture https://example.com/articles > your-config.yml`
3. Validate: `html2rss validate your-config.yml`
4. Live-check: `html2rss test your-config.yml`, then ship with `html2rss apply your-config.yml`
5. Mount into `html2rss-web` or contribute to html2rss-configs
5. Add into `html2rss-web` with `registry add`, or contribute to html2rss-configs
6. Keep `strategy: auto`; if results are incomplete, confirm the companion scraper is configured before adding site-specific controls

`html2rss feed` is a Thor alias for `apply`. `html2rss auto` aliases `scrape` (one-shot, no YAML).

## Minimal config

One YAML file is one feed. Include `registry.id` when you will add the file to `html2rss-web`:

<Code
code={`
registry:
id: example.com/blog
channel:
url: https://example.com/blog
title: My Blog
Expand Down Expand Up @@ -60,22 +64,28 @@ Details: [Selectors](/ruby-gem/reference/selectors/), [Strategy](/ruby-gem/refer

Raise `--max-redirects` / `--max-requests` only when the site needs more budget.

## Mount on html2rss-web
## Add to html2rss-web

Bind-mount a feeds file (see production Compose comments):
Add the YAML into the built-in `local` registry bundle (not a bind-mounted multi-feed file):

<Code
code={`
volumes:
- type: bind
source: ./config/feeds.yml
target: /app/config/feeds.yml
read_only: true
html2rss-web registry add path/to/your-config.yml
`}
lang="yaml"
lang="bash"
/>

On Docker Compose, copy the file into the container first, then add:

<Code
code={`
docker compose cp path/to/your-config.yml html2rss-web:/tmp/your-config.yml
docker compose exec html2rss-web html2rss-web registry add /tmp/your-config.yml
`}
lang="bash"
/>

Feeds are served as `.rss` / `.json` paths on your instance (for example `/example.com/blog.rss`).
The feed is served as `/{registry.id}.rss` / `.json` on your instance (for example `/example.com/blog.rss`).

## Contribute to the Feed Directory

Expand Down
69 changes: 26 additions & 43 deletions src/content/docs/ruby-gem/guides/managing-feed-configs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,62 +5,46 @@ description: "Learn how to manage feed configurations with YAML files. Organize,

import { Code } from "@astrojs/starlight/components";

For easier management, especially when using the CLI or `html2rss-web`, you can store your feed configurations in a YAML file.
For easier management, especially when using the CLI or `html2rss-web`, store each feed configuration in its own YAML file.

## Global and Feed-Specific Configurations
## One file, one feed

You can define global settings that apply to all feeds, and then define individual feed configurations under the `feeds` key.
Each YAML file is a single feed. Put channel settings, headers, selectors, and other options in that file — there is no multi-feed map.

<Code
code={
"# Global settings\n" +
"headers:\n" +
' "User-Agent": "Mozilla/5.0 (compatible; html2rss/1.0; Mobile)"\n' +
' "Accept": "text/html"\n' +
"\n" +
"# Feed-specific settings\n" +
"feeds:\n" +
" my-first-feed:\n" +
" channel:\n" +
' url: "https://example.com/blog"\n' +
" selectors:\n" +
" items:\n" +
' selector: ".post"\n' +
" title:\n" +
' selector: "h2"\n' +
" url:\n" +
' selector: "a"\n' +
' extractor: "href"\n' +
" my-second-feed:\n" +
" channel:\n" +
' url: "https://example.com/news"\n' +
" selectors:\n" +
" items:\n" +
' selector: ".news-item"\n' +
" title:\n" +
' selector: "h2"\n' +
" url:\n" +
' selector: "a"\n' +
' extractor: "href"'
}
code={`
# Optional request headers for this feed
headers:
"User-Agent": "Mozilla/5.0 (compatible; html2rss/1.0; Mobile)"
"Accept": "text/html"
# Feed definition
channel:
url: "https://example.com/blog"
selectors:
items:
selector: ".post"
title:
selector: "h2"
url:
selector: "a"
extractor: "href"
`}
lang="yml"
/>

Include a `registry.id` when you will add the file to `html2rss-web` with `registry add`. See [Advanced Feeds](/creating-custom-feeds/).

## Building Feeds from a YAML File

### Ruby

<Code
code={`
require 'html2rss'
# Build a specific feed from the YAML file
my_feed_config = Html2rss.config_from_yaml_file('feeds.yml', 'my-first-feed')
# One YAML file is one feed
my_feed_config = Html2rss.config_from_yaml_file('my-first-feed.yml')
rss = Html2rss.feed(my_feed_config)
puts rss
# If the YAML file contains only one feed, you can omit the feed name
single_feed_config = Html2rss.config_from_yaml_file('single.yml')
rss = Html2rss.feed(single_feed_config)
puts rss
`}
lang="ruby"
/>
Expand Down Expand Up @@ -88,7 +72,7 @@ For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same `
<Code
code={`
require 'html2rss'
config = Html2rss.config_from_yaml_file('feeds.yml', 'my-first-feed')
config = Html2rss.config_from_yaml_file('my-first-feed.yml')
result = Html2rss.feed_result(config)
result.to_rss
result.to_json_feed(feed_url: 'https://example.com/feeds/my-first-feed.json')
Expand All @@ -104,8 +88,7 @@ Depth and method contracts: [YARD for `html2rss`](https://www.rubydoc.info/gems/

<Code
code={`
html2rss apply feeds.yml my-first-feed ; \
html2rss apply single.yml
html2rss apply my-first-feed.yml
`}
lang="sh"
/>
6 changes: 2 additions & 4 deletions src/content/docs/ruby-gem/reference/cli-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ Loads a YAML config, builds the feed, and prints the RSS XML to stdout. (`html2r
<Code
code={`
html2rss apply single.yml ; \\
html2rss apply feeds.yml my-first-feed ; \\
html2rss apply single.yml --strategy auto ; \\
BOTASAURUS_SCRAPER_URL="http://localhost:4010" html2rss apply single.yml --strategy botasaurus ; \\
html2rss apply single.yml --max-redirects 5 --max-requests 6 ; \\
Expand All @@ -119,7 +118,7 @@ Loads a YAML config, builds the feed, and prints the RSS XML to stdout. (`html2r
lang="bash"
/>

Command: `html2rss apply YAML_FILE [feed_name]`
Command: `html2rss apply YAML_FILE`

Available options:

Expand Down Expand Up @@ -209,13 +208,12 @@ Validates a config with the runtime validator without generating a feed.
<Code
code={`
html2rss validate single.yml ; \\
html2rss validate feeds.yml my-first-feed ; \\
html2rss validate single.yml --params id:42
`}
lang="bash"
/>

Command: `html2rss validate YAML_FILE [feed_name]`
Command: `html2rss validate YAML_FILE`

Available options:

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/troubleshooting/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ If you are getting a "command not found" error, try the following:

- Try a more specific listing, newsroom, changelog, or updates URL before changing infrastructure
- If your first-run stack includes Botasaurus, ensure the `botasaurus` service is running
- Check the feed configuration in `feeds.yml` for typos or invalid selectors
- Check your feed YAML (and `registry.id` if you used `registry add`) for typos or invalid selectors
- Look for parsing errors in the logs:
<Code code={`docker compose logs html2rss-web`} lang="bash" />

Expand Down
15 changes: 7 additions & 8 deletions src/content/docs/web-application/deployment.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,21 +79,20 @@ Example `.env` beside Compose:

After `docker compose up -d`, check `docker compose logs caddy --tail 20` for certificate issuance.

## Custom feeds mount
## Custom feeds

Auto-source is the default. To add curated YAML feeds, bind-mount `feeds.yml` (see [Advanced Feeds](/creating-custom-feeds/)):
Auto-source is the default. To add curated YAML feeds, use `html2rss-web registry add` into the built-in `local` registry (see [Advanced Feeds](/creating-custom-feeds/)):

<Code
code={`
volumes:
- type: bind
source: ./config/feeds.yml
target: /app/config/feeds.yml
read_only: true
docker compose cp path/to/your-config.yml html2rss-web:/tmp/your-config.yml
docker compose exec html2rss-web html2rss-web registry add /tmp/your-config.yml
`}
lang="yaml"
lang="bash"
/>

Each YAML file is one feed and must include `registry.id`. Feeds are served at `/{registry.id}.rss` on your instance.

## Operate

- `GET /api/v1/health/ready` for readiness
Expand Down
29 changes: 29 additions & 0 deletions src/content/docs/web-application/guides/refine-selectors.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: "Refine a generated feed"
description: "Recover a generated feed when it finds the wrong list, misses fields, or starts from the wrong page."
---

Use **Refine selectors** after automatic feed generation when the result is empty, follows the wrong list of links, or misses the title, link, or published date. Refinement is a recovery step for one generated feed, not a YAML authoring workflow.

## Before You Start

Your `html2rss-web` instance must have `STUDIO_ENABLED=true`, and you must have entered a valid access token. Live suggestions and previews also require `AUTO_SOURCE_ENABLED=true`.

If refinement is unavailable, the result page does not show the action. Opening an old refinement link on an instance where the studio is disabled returns you to feed creation.

## Refine the Result

1. Generate a feed from a listing page and open **Refine selectors** from the result.
2. Check the page URL at the top. If you started from the wrong listing, edit it first; the studio requests a new suggestion for that page.
3. Check **Items selector**. It should identify the repeated article or entry container, such as `article` or `.post`.
4. Use the suggested selector when it identifies the correct list. Enable enhanced extraction only when the page needs additional semantic field detection.
5. Open a missing field to set a selector for its title, link, or published date.
6. Check the sample items, then choose **Save and generate**.

The result is successful when the sample contains the intended entries and the generated feed preserves their useful title and link.

## Keep the Recovery Focused

Prefer changing the listing URL before adding more selectors. A category, newsroom, changelog, or archive page usually gives better results than a generic homepage.

**View as YAML** shows the current selectors fragment for inspection. It is not a complete custom-feed config and does not publish a Feed Directory entry. For a reviewable, version-controlled scraping rule, use [Creating Custom Feeds](/creating-custom-feeds/).
2 changes: 1 addition & 1 deletion src/content/docs/web-application/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ sidebar:
- Web UI for page-URL auto-source
- Access-controlled create API (`HTML2RSS_ACCESS_TOKEN`)
- Feed Directory catalog from your instance (`GET /api/v1/configs`)
- Optional custom YAML via `feeds.yml` mount
- Optional custom YAML via `html2rss-web registry add` (built-in local registry)
- OpenAPI at `/openapi.yaml`

The extraction engine is the [`html2rss`](https://github.com/html2rss/html2rss) gem.
Expand Down
Loading
Loading