Skip to content

fix(config): make body visibility timeout configurable - #2131

Open
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2129
Open

fix(config): make body visibility timeout configurable#2131
nightcityblade wants to merge 1 commit into
unclecode:developfrom
nightcityblade:fix/issue-2129

Conversation

@nightcityblade

Copy link
Copy Markdown
Contributor

Summary

Fixes #2129.

Add CrawlerRunConfig.body_visibility_timeout, defaulting to the existing 30-second behavior, and use it for the body visibility check. This lets callers lower the ceiling for pages whose body remains hidden without changing the default crawl behavior. The value is included in config serialization and the untrusted-config allowlist.

List of files changed and why

  • crawl4ai/async_configs.py - define, document, serialize, and allow the new configuration field.
  • crawl4ai/async_crawler_strategy.py - pass the configured timeout to the body visibility wait.
  • tests/test_config_defaults.py - cover the unchanged default and custom-value dump/load round trip.

How Has This Been Tested?

  • uv run --python 3.12 --with pytest python -m pytest tests/test_config_defaults.py -q — 33 passed, 1 existing deprecation warning.
  • uv run --python 3.12 --with ruff ruff check --select E9,F63,F7,F82 --ignore F821 crawl4ai/async_configs.py crawl4ai/async_crawler_strategy.py tests/test_config_defaults.py — passed. (F821 is excluded because of the existing VirtualScrollConfig forward-reference finding in async_crawler_strategy.py.)
  • python3 -m compileall -q crawl4ai/async_configs.py crawl4ai/async_crawler_strategy.py tests/test_config_defaults.py — passed.
  • git diff --check — passed.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas — N/A; no complex logic was introduced.
  • I have made corresponding changes to the documentation — the public configuration docstring documents the new field.
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant