Skip to content

Remove farfetch-scraper leftovers from templates, TROUBLESHOOTING, SECURITY and diff_runs - #18

Merged
jehrr merged 2 commits into
mainfrom
fix/donor-leftovers
Sep 28, 2026
Merged

jehrr merged 2 commits into
mainfrom
fix/donor-leftovers

Conversation

@jehrr

@jehrr jehrr commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Leftovers from farfetch-scraper (the donor) that describe Farfetch as if it were Amazon. Amazon publishes no JSON-LD, so every "JSON-LD first" instruction here pointed a reader at a path that does not exist.

Fixed

  • .github/ISSUE_TEMPLATE/bug_report.yml — "Zero products? … not the bare /shopping/kids/items.aspx hub — the hub carries no product JSON-LD" (Farfetch's hub), "See 'Geo-redirect' in the README" (no such section), --out girls, and "96 products, as the README says a category page yields". Now: open the {out}_page1_debug.html dump (TROUBLESHOOTING's own advice), the README's "currency follows the exit IP" trap (amazon.co.uk/.co.jp quoting EUR from a European exit), --out headphones, and the README's measured "16–30 organic tiles per search page".
  • site_changed.yml — "the parser tries JSON-LD first, then a CSS + URL-pattern fallback" and a JSON-LD extraction one-liner / "paste one Product node from the ItemList". Now the data-attribute anchors from README "How it parses" and the /dp/{ASIN} fallback; evidence asks for --dump-html and whether the fallback warning fired.
  • TROUBLESHOOTING.md — 0-rows table: "The JSON-LD path found nothing… Check that product links still match -item-<digits>.aspx" and "The hub URL. /shopping/kids/items.aspx…". First row rewritten to Amazon's anchors and fallback; the hub row removed (no Amazon equivalent measured in this repo).
  • SECURITY.md — "how Akamai or reCAPTCHA behave" → "how AWS WAF or Amazon's own captcha behave" (README: AWS WAF, not Akamai).
  • diff_runs.py docstring — source_changed explained as "the DOM-corrected figure and the raw JSON-LD one" → the offscreen/split/detail nodes this repo's price_source records; examples used Farfetch's girls_clothing → headphones.
  • CONTRIBUTING.md — "the registration form" (Farfetch's reCAPTCHA target) → "a registration or login form".

Kept, deliberately

  • product_parser.py "Every other scraper in this family reads schema.org JSON-LD first…" — family provenance explaining why this repo differs. (Borderline: no longer true of every sibling, but it is not a donor description of this site.)
  • farfetch notes in captcha_solver.py, craigslist note in .gitignore — explicit sibling provenance.
  • README badges: already 7, in family order; "runs without an account" is honest (README "What works with nothing at all": 77 products over 3 pages with no key, proxy or account, 2026-09-08).
  • Repo description/topics: no donor words, unchanged. Released CHANGELOG sections untouched; the bullet is added to the existing [Unreleased] → Fixed.
  • The main checkout's untracked .claude/ and the audit-stage1 worktree were not touched.

Verification

python3 smoke_test.py exit 0 before and after (same two engine groups skipped locally); ci_checks.py --secret-check exit 0; issue templates parse as YAML.

🤖 Generated with Claude Code

jehrr and others added 2 commits September 28, 2026 19:48
…CURITY, diff_runs

- bug_report.yml: Farfetch /shopping/kids/items.aspx hub tip, --out girls
  and "96 products" replaced with Amazon's debug-dump advice, the README's
  currency-follows-exit-IP trap and its 16-30 tiles per search page.
- site_changed.yml: "JSON-LD first" and a JSON-LD evidence one-liner
  replaced with the data-attribute anchors and /dp/{ASIN} fallback.
- TROUBLESHOOTING.md: 0-rows table no longer names -item-<digits>.aspx
  links or Farfetch's hub.
- SECURITY.md: Akamai/reCAPTCHA -> AWS WAF / Amazon's own captcha.
- diff_runs.py: source_changed explained with offscreen/split/detail, not
  DOM-corrected vs raw JSON-LD; examples no longer use girls_clothing.
- CONTRIBUTING.md: "the registration form" generalised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SECURITY.md said the project has no releases or version tags; it has both
  (gh release list). Supported versions now names the latest release and main,
  in the wording bbb-scraper and foodpanda-scraper already use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr force-pushed the fix/donor-leftovers branch from 7b63aec to 0c8e6ca Compare September 28, 2026 19:48
@jehrr
jehrr merged commit 62a92f8 into main Sep 28, 2026
7 checks passed
@jehrr
jehrr deleted the fix/donor-leftovers branch September 28, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant