fix(scraping): a nested <noscript> discards the entire page body - #2114
fix(scraping): a nested <noscript> discards the entire page body#2114Aitosoft wants to merge 1 commit into
Conversation
`<noscript>` may not nest. With scripting enabled its content is raw text, so
the parser never sees an inner `<noscript>` as an element and the OUTER element
is left unclosed — libxml2 then swallows every following node into it. Since
`LXMLWebScrapingStrategy._scrap` hands the raw serialised DOM straight to
`lhtml.document_fromstring`, the damage is done before there is a tree to fix.
A single WordPress lazy-load plugin that wraps the Google Tag Manager block in
a second `<noscript>` is enough to lose a whole site:
312,628 B rendered HTML -> 97 B cleaned_html -> 1 B markdown
with `status_code: 200` and `success: True`, so nothing downstream can tell.
`cleaned_html` is literally `<html><head><title>…</title></head></html>` — the
`<body>` is gone. I found this across 70 hosts in one corpus; it reproduced
byte-identically on the same URLs 3.5 months apart, because the plugin markup is
stable.
Six-line reproduction on unmodified v0.9.2:
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
MIN = '''<html><head><title>T</title></head><body>
<noscript><iframe src="about:blank"></iframe><noscript><iframe src="about:blank"></iframe></noscript>
<h1>Contact</h1><p>Phone 010 123 4567</p>
<div><p>A second paragraph.</p></div>
</body></html>'''
LXMLWebScrapingStrategy().scrap('https://x/', MIN, word_count_threshold=1).cleaned_html
# nested: 42 B -> '<html><head><title>T</title></head></html>'
# single <noscript>: 187 B (well-formed noscript is harmless)
# no <noscript>: 188 B
Fix: `strip_noscript()`, applied at the top of `_scrap()` immediately before
parsing. Removal rather than unwrapping, because Crawl4AI renders with
JavaScript enabled — by definition `<noscript>` content is not what the page
displayed, and its scripted equivalent is already in the DOM. Two passes:
well-formed elements first (a non-greedy match ends at the first `</noscript>`,
which is where the browser's parser ends the outer element too), then any
unpaired tag left by truncated markup, so an unclosed `<noscript>` cannot
swallow a body either. Pages without the substring return the same object
unparsed, so the cost is one `in` check.
`grep noscript crawl4ai/content_scraping_strategy.py crawl4ai/utils.py` returns
nothing on develop — there is no existing `<noscript>` handling to conflict with.
Tests: tests/general/test_noscript_body_loss.py, 11 passed. Covers nested,
well-formed, unclosed, uppercase and attributed shapes, asserts nested and
well-formed become indistinguishable, and pins that a page whose only content
is inside `<noscript>` correctly stays empty.
pytest tests/general/test_noscript_body_loss.py -q # 11 passed
pytest tests/test_pr_1435_redirected_status_code.py -q # 7 passed
eeshsaxena
left a comment
eeshsaxena
left a comment
There was a problem hiding this comment.
Good docstring, and the two-pass approach actually matches how a browser handles unnestable <noscript>. Pass 1's non-greedy ...</noscript\s*> stops at the first close tag, which is exactly where the browser ends the outer element when scripting is enabled, so <noscript>...<noscript>...</noscript> tail</noscript> gets the first block removed and tail (which the browser would actually render) is preserved. Pass 2 then strips the stray unpaired open/close tags that truncated or malformed markup leaves behind, unwrapping rather than deleting, which is again what the browser does with a leftover </noscript>. So an unclosed noscript can no longer swallow the body, and removing (not unwrapping) the paired content is justified since Crawl4AI renders with JS on. Reads correct.
The one inherent caveat with pre-parse regex stripping is a literal <noscript sitting inside a <script> string or an HTML comment, but that is an unavoidable tradeoff of fixing this before lxml sees the tree, and the alternative (letting lxml eat the page) is clearly worse.
2 participants
Summary
<noscript>may not nest. With scripting enabled its content is raw text, so aparser never sees an inner
<noscript>as an element and the outer elementis left unclosed — libxml2 then swallows every following node into it.
LXMLWebScrapingStrategy._scraphands the raw serialised DOM straight tolhtml.document_fromstring, so the damage is done before there is a tree thatcould be repaired downstream.
A single WordPress lazy-load plugin that wraps the Google Tag Manager block in a
second
<noscript>is enough to lose an entire page:with
status_code: 200andsuccess: True, so nothing downstream can tell thatanything went wrong.
cleaned_htmlis literally<html><head><title>…</title></head></html>— the whole<body>is gone. Themarkdown generator is innocent; it faithfully renders nothing.
I hit this across 70 hosts in one corpus, and it reproduced byte-identically
on the same URLs 3.5 months apart, because the plugin markup is stable.
Reproduction (6 lines, unmodified
develop)Confirmed by excision on the real captured DOM:
cleaned_html<noscript>tags<script>onlyFix
strip_noscript(), applied at the top of_scrap()immediately before parsing.Removal rather than unwrapping, because Crawl4AI renders with JavaScript
enabled: by definition
<noscript>content is not what the page displayed,and its scripted equivalent is already in the DOM. Dropping it loses nothing and
removes the whole malformed-nesting failure class.
Two passes, so this handles the tag rather than the specific nesting:
</noscript>,which is exactly where the browser's own parser ends the outer element too,
so the nested shape collapses correctly.
<noscript>cannot swallow a body either.Pages without the substring return the same object unparsed, so the cost on the
overwhelming majority of pages is one
incheck.grep noscript crawl4ai/content_scraping_strategy.py crawl4ai/utils.pyreturnsnothing on
develop— there is no existing<noscript>handling to conflictwith, which is also why this went unnoticed.
Tests
tests/general/test_noscript_body_loss.py— 11 tests. Covers nested,well-formed, unclosed, uppercase and attributed shapes; asserts that nested and
well-formed become indistinguishable (the point of the fix, as opposed to
special-casing the nested shape); and pins that a page whose only content is
inside
<noscript>correctly stays empty, since JavaScript was enabled.Known limitation
A
<noscript>literal inside a<script>string could in principle start amatch that ends at a later real
</noscript>. Any pre-parse repair carriesthis, and the alternative — a spec-compliant HTML5 parse of every document — is
a much larger change than this bug warrants. Not observed in the corpus above.
Happy to adjust the approach if you would rather see this handled elsewhere in
the pipeline.