Hi @unclecode,
A note of genuine thanks. I maintain an academic-research skill bundle for Claude (business/finance/social-science methods). When documenting how researchers should handle JS-heavy government/disclosure pages, crawl4ai's core idea — turning messy web pages into clean, LLM-readable structure — shaped how I framed the 'escalation ladder' guidance (try the underlying API first, then official batch files, then a headless-browser tool like crawl4ai only when needed).
Two honest notes, in the spirit of good citation:
- I adopted the concept, not your code or dependency — the guidance is documentation, and I attribute crawl4ai (Apache 2.0) in our NOTICE.
- For academic-integrity reasons, my guidance explicitly tells researchers not to use the stealth / anti-detection features — scholarly scraping must respect robots.txt and site ToS. That's a constraint on the researcher, not a criticism of the tool; crawl4ai is excellent and your work is clearly a huge contribution (76k+ stars well earned).
Thank you for building and open-sourcing it.
Hi @unclecode,
A note of genuine thanks. I maintain an academic-research skill bundle for Claude (business/finance/social-science methods). When documenting how researchers should handle JS-heavy government/disclosure pages, crawl4ai's core idea — turning messy web pages into clean, LLM-readable structure — shaped how I framed the 'escalation ladder' guidance (try the underlying API first, then official batch files, then a headless-browser tool like crawl4ai only when needed).
Two honest notes, in the spirit of good citation:
Thank you for building and open-sourcing it.