SiteMapParserBolt: strict parsing, opt-in sniffing, non-terminal parse failure - #2125
Open
abhinav-phi wants to merge 1 commit into
Open
SiteMapParserBolt: strict parsing, opt-in sniffing, non-terminal parse failure#2125abhinav-phi wants to merge 1 commit into
abhinav-phi wants to merge 1 commit into
Conversation
…e failure (apache#2083) - the parser is now built in strict mode (sitemap.strict, default true): crawler-commons then discards URLs a sitemap lists on hosts other than its own, so a sitemap cannot enrol URLs on hosts it has nothing to do with, and an HTML page mentioning the sitemap namespace is not parsed leniently into half a sitemap - content sniffing moves behind sitemap.sniffContent (default false, like feed.sniffContent for feeds): a page carrying the namespace string in its first bytes was reclassified as a sitemap, never reached the parser bolt and was never indexed. The key the existing test already set but the bolt ignored is now honoured; a content type which rules a sitemap out (a page served as HTML) stops the sniffing - a document marked as a sitemap through persisted metadata whose body does not parse is emitted as FETCH_ERROR with the isSitemap key dropped, instead of a terminal ERROR: with the archetype's fetchInterval.error of -1, an ERROR removed the URL from the crawl for good, letting whoever controls the content decide what stays in the corpus. On its next fetch the document goes to the parser bolt like any other page The image and all-extensions test sitemaps listed their first URL on www.example.com, which strict mode now correctly excludes; that entry moved under the sitemap's own host.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2083.
sitemap.strict, default true): crawler-commons then discards URLs a sitemap lists on hosts other than its own — previously a sitemap could enrol URLs on any host, and those entries skippedparser.emitOutlinks.max.per.pageand the robots meta tags of the HTML path.sitemap.sniffContent, default false, likefeed.sniffContentfor feeds): any HTML page carrying the sitemaps.org namespace string in its first bytes was reclassified as a sitemap, never reached the parser bolt and was never indexed. The keySiteMapParserBoltTestalready set but the bolt ignored is now honoured; a content type which rules a sitemap out (a page served as HTML) stops the sniffing when it is enabled.FETCH_ERRORwith theisSitemapkey dropped, instead of a terminalERROR. With the archetype'sfetchInterval.error: -1an ERROR removed the URL from the crawl for good — in an open crawl that lets a third party remove other people's URLs from the corpus. On its next fetch the document goes to the parser bolt like any other page.Release note needed: strict parsing and the sniffing default both change what an existing crawl discovers.
Test resources: the image / all-extensions sitemaps listed their first URL on
www.example.com, which strict mode now correctly excludes; that entry moved under the sitemap's own host.