Canonicalise hosts for politeness queues and the robots.txt cache - #2122
Open
abhinav-phi wants to merge 1 commit into
Open
Canonicalise hosts for politeness queues and the robots.txt cache#2122abhinav-phi wants to merge 1 commit into
abhinav-phi wants to merge 1 commit into
Conversation
…ache#2082) okhttp percent-decodes and lowercases the host when it parses the URL, but FetchItem.create, SimpleFetcherBolt.getPolitenessKey and HttpRobotRulesParser.getCacheKey keyed on the raw host string, so http://exampl%65.org/ and http://example.org/ were one origin at connect time and two queue ids and two robots.txt cache entries in the bolt. One server could end up with several delay clocks running in parallel and its robots.txt fetched once per spelling. URLUtil.getCanonicalHost percent-decodes the host, strips a trailing dot and lowercases it, and all three call sites now use it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2082.
okhttp percent-decodes and lowercases the host when it parses the URL, so
http://exampl%65.org/andhttp://example.org/are one origin at connect time — butFetchItem.create,SimpleFetcherBolt.getPolitenessKeyandHttpRobotRulesParser.getCacheKeykeyed on the raw host string (only lowercased). Each spelling got its own politeness queue and its own robots.txt cache entry, so the per-host delay was applied several times in parallel to the same server (bounded byfetcher.threads.number, not by the number of aliases) and robots.txt was downloaded once per spelling.Adds
URLUtil.getCanonicalHost, which percent-decodes the host, strips a trailing dot and lowercases it, and uses it at all three call sites. This changes only how URLs are grouped, not which URLs are crawled.