Config option to disable is_blocked() post-crawl content veto #2058

Activity

1. nightcityblade commented on Jul 19, 2026

   

   nightcityblade

   on Jul 19, 2026

   Contributor

   More actions

   Hi, I'd like to work on this. I'll submit a PR shortly.
2. nightcityblade commented on Aug 27, 2026

   

   nightcityblade

   on Aug 27, 2026

   Last edited by nightcityblade

   Contributor

   More actions

   I’m withdrawing my earlier claim so I don’t block anyone. This needs maintainer agreement on the public API shape (CrawlerRunConfig flag versus exposing the verdict) before a focused implementation would be appropriate.
3. talelboussetta commented on Nov 15, 2026

   

   talelboussetta

   on Nov 15, 2026

   Contributor

   More actions

   Another case where this check vetoes a page that was fetched fine, reproduced on develop @ 2c0b9e9.

   A page that fills its body with JS 300 ms after load (the usual SPA pattern) comes back with the default config as:

   default            success=False status=200 error='Blocked by anti-bot protection: Structural: minimal_text on small page (270 bytes, 10 chars visible)' html_len=270
   wait_for=css:#t    success=True status=200 error='' html_len=336
   

   Nothing is blocking the crawl: the render just hasn't finished when the HTML is captured, and the same URL succeeds with wait_for. The "anti-bot" label points people at proxies and stealth settings when the fix is a wait.

   Repro (self-contained, serves its own page on 127.0.0.1:8799)

   import asyncio, threading, http.server, socketserver
   from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
   
   HTML = b"""<!doctype html><html><head><title>app</title></head><body><div id='app'>Loading...</div>
   <script>setTimeout(function(){document.getElementById('app').innerHTML='<h1>Hello</h1><p id="t">Rendered content that arrives 300 ms after load.</p>';},300);</script>
   </body></html>"""
   
   class H(http.server.BaseHTTPRequestHandler):
       def do_GET(self):
           self.send_response(200); self.send_header("Content-Type", "text/html")
           self.send_header("Content-Length", str(len(HTML))); self.end_headers(); self.wfile.write(HTML)
       def log_message(self, *a): pass
   
   srv = socketserver.TCPServer(("127.0.0.1", 8799), H)
   threading.Thread(target=srv.serve_forever, daemon=True).start()
   
   async def main():
       url = "http://127.0.0.1:8799/"
       async with AsyncWebCrawler(config=BrowserConfig(headless=True, verbose=False)) as c:
           for label, cfg in [
               ("default", CrawlerRunConfig(cache_mode=CacheMode.BYPASS)),
               ("wait_for=css:#t", CrawlerRunConfig(cache_mode=CacheMode.BYPASS, wait_for="css:#t")),
           ]:
               r = await c.arun(url, config=cfg)
               print(f"{label:18} success={r.success} status={r.status_code} error={r.error_message!r} html_len={len(r.html or '')}")
   
   asyncio.run(main())
   srv.shutdown()

   The opt-out flag in #2088 would let people turn the check off, but a default crawl would still report this. A smaller change could help on its own: when the verdict comes only from the structural tier on a 200 response, say so in the message, for example Possibly incomplete render (minimal_text, 270 bytes); if the page renders with JavaScript, try wait_for or delay_before_return_html. I can open a PR for that if it's wanted.