File scheme off by default; reads confined to file.protocol.root - #2124
Open
abhinav-phi wants to merge 1 commit into
Open
File scheme off by default; reads confined to file.protocol.root#2124abhinav-phi wants to merge 1 commit into
abhinav-phi wants to merge 1 commit into
Conversation
…che#2081) crawler-default.yaml enabled the file scheme in every topology built on the library defaults, and FileProtocol read whatever path the worker user could read: a fetched page can put a file:// URL into the frontier and the response body - including the topology configuration - went on to be parsed and indexed. The shipped protocols list is now http,https, and the file scheme has to be enabled deliberately. FileProtocol gains a file.protocol.root key: when it is set, only paths canonicalising below that directory are served (symlinks included), everything else returns 403; when it is not set, the file scheme serves nothing. An operator crawling a local corpus sets both keys.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2081.
crawler-default.yamlshippedprotocols: "http,https,file", andFileProtocolread whatever path the worker user could read: a fetched page can put afile://URL into the frontier (absolutefile://hrefs resolve to themselves in the parser), and the response body — including the topology configuration and the backend credentials in it — went on to be parsed and indexed. The only gate was URL filtering, and the library default ships no URL filters at all.protocolslist is nowhttp,https; enabling the file scheme is a deliberate actFileProtocolgainsfile.protocol.root: when set, only paths canonicalising below that directory are served (symlinks included), everything else returns 403Release note needed: topologies crawling local corpora on the old default stop working until they add
filetoprotocolsand setfile.protocol.root.