Robots.txt fetch no longer replaces the configured content limit - #2121
Open
abhinav-phi wants to merge 1 commit into
Open
Robots.txt fetch no longer replaces the configured content limit#2121abhinav-phi wants to merge 1 commit into
abhinav-phi wants to merge 1 commit into
Conversation
…he#2087) HttpRobotRulesParser wrote http.robots.content.limit (-1 by default) into the fetch metadata as http.content.limit, which HttpProtocol reads unconditionally: a robots.txt fetch ran with no limit at all, even when the operator had configured a finite http.content.limit, despite the shipped comment saying the default was 'same as http.content.limit'. - a http.robots.content.limit of -1 now means 'inherit the global limit' and the key is only written into the fetch metadata when a robots specific limit is configured - the shipped default of http.robots.content.limit is 524288 (512 kiB), matching the minimum the robots.txt RFC draft asks crawlers to parse - HttpProtocol ignores a metadata http.content.limit of -1 when the global limit is finite, so per-URL metadata can tighten the limit but not remove it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2087.
HttpRobotRulesParser.setConfputhttp.robots.content.limit(-1 by default) into the fetch metadata ashttp.content.limit, andHttpProtocol.getProtocolOutputapplies that value unconditionally — so every robots.txt fetch ran with no limit at all, even when the operator configured a finitehttp.content.limit, although the shipped comment said the default was 'same as http.content.limit'. With no limit, the decompressed body is buffered untilConstants.MAX_ARRAY_SIZE, just under 2 GB.http.robots.content.limit: -1now means 'inherit the global limit'; the key is only written into the fetch metadata when a robots-specific limit is configuredhttp.robots.content.limitis 524288 (512 kiB), the minimum the robots.txt RFC draft asks crawlers to fetch and parseHttpProtocolignores a metadatahttp.content.limitof -1 when the global limit is finite, so per-URL metadata can tighten the limit but not remove it