fix: only treat the base domain and its subdomains as internal

[eeshsaxena]

Summary

is_external_url decided whether a link belongs to the site being crawled with a bare suffix check:

return not url_domain.endswith(base)

Any host whose name merely ends with the base domain string passed that check and was classified as internal:

is_external_url("http://malicious-example.com", "example.com")  # False (treated as internal)
is_external_url("http://notexample.com/path", "example.com")    # False (treated as internal)

base comes from get_base_domain(), so it is a bare registrable domain and these are genuinely different sites an attacker can register. The result feeds link classification in content_scraping_strategy.py and the deep-crawl scope check in domain_mapper.py, so a crawl scoped to one domain could follow links onto look-alike domains.

The fix matches what the rest of the codebase already does (deep_crawling/filters.py, domain_mapper.py): internal means equal to the base domain, or a dot-separated subdomain of it (domain == base or domain.endswith(f".{base}")). Real subdomains (blog.example.com) and relative URLs are unchanged. The www. strip also moves from .replace("www.", "") to .removeprefix("www."), so a host containing www. somewhere other than the front is not silently rewritten.

No existing issue filed — happy to open one first if you'd prefer it tracked separately.

List of files changed and why

• crawl4ai/utils.py — rewrite is_external_url's membership test from endswith(base) to == base or endswith("." + base), and switch the www. strip to removeprefix.
• tests/regression/test_reg_utils.py — add TestIsExternalUrl covering the base domain, a subdomain, www., the two look-alike cases (notexample.com, malicious-example.com), the base domain as a prefix, an unrelated domain, a relative URL, and a mailto: link.

How Has This Been Tested?

Ran pytest tests/regression/test_reg_utils.py. Against the current code, the two look-alike cases in the new TestIsExternalUrl fail (demonstrating the bug); with the fix applied the full file passes.

Note: tests/test_link_extractor.py::test_link_extractor fails on a clean checkout as well (the async test isn't picked up by the configured asyncio plugin), so it's unrelated to this change.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[eeshsaxena]

This closes a real hole. The old url_domain.endswith(base) treated any look-alike suffix as internal, so malicious-example.com and example.com.evil.com both counted as internal to example.com. Switching to url_domain == base or url_domain.endswith(f".{base}") requires either an exact match or a dot-boundary subdomain, which is the correct rule, and moving from .replace("www.", "") (which strips www. anywhere) to .removeprefix("www.") avoids mangling hosts that merely contain www. The test matrix covers the important cases, including the example.com.evil.com prefix trick.

One robustness nit, not a blocker: parsed.netloc.lower().split(":")[0] keeps userinfo, so http://user@evil.com leaves user@evil.com as the compared host. parsed.hostname would strip userinfo and port and lowercase for you, sidestepping that whole class of edge case.