Skip to content

fix(content-filter): honor preserve whitelist for excluded tags - #2126

Open
bong-u wants to merge 1 commit into
unclecode:developfrom
bong-u:develop
Open

fix(content-filter): honor preserve whitelist for excluded tags#2126
bong-u wants to merge 1 commit into
unclecode:developfrom
bong-u:develop

Conversation

@bong-u

@bong-u bong-u commented Aug 6, 2026

Copy link
Copy Markdown

Summary

Fixes #2125

_remove_unwanted_tags decomposed by tag name, so preserve_tags and preserve_classes never reached elements whose tag is in excluded_tags.
It now iterates elements and skips the ones _is_preserved() matches, so both whitelists go through the guard that already existed.

This changes semantics for script and style, which are in excluded_tags too: preserve_tags=["script"] now really keeps them.
A user has to name the tag to get that, so I did not special-case it.

List of files changed and why

  • crawl4ai/content_filter_strategy.py - filter by element instead of tag name in _remove_unwanted_tags so the whitelist is consulted; docstring updated to say the whitelist also overrides excluded_tags.
  • tests/test_pruning_preserve_whitelist_1900.py - the old test_whitelist_does_not_override_excluded_tags asserted nothing and documented the bug as expected, so I inverted it, and added a case for a whitelisted class on an excluded tag.

How Has This Been Tested?

pytest tests/test_pruning_preserve_whitelist_1900.py - 21 passed.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant