Keep the fetcher off private address space by default - #2127
Open
abhinav-phi wants to merge 1 commit into
Open
Conversation
The IP address filter was the only check that looked at the address the fetcher actually connects to, and both of its keys shipped commented out, so the interceptor was never installed at library defaults. The regex exclusion list in the archetypes matched literal localhost and four dotted-quad ranges and nothing else: link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 forms the resolver maps to 127.0.0.1 all passed it. - crawler-default.yaml ships http.filter.ipaddress.exclude enabled (loopback, RFC1918, link-local, CGNAT, IPv6 unique-local); a fetched page decides which hosts the fetcher connects to, and these ranges host unauthenticated services a public index must not leak into. Crawling an intranet needs an explicit opt-out, documented in the comment next to the key - the archetype default-regex-filters.txt (and its opensearch/solr copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8, IPv6 fc00::/7 and fe80::/10, the abbreviated loopback forms (127.1) and integer IPv4 hosts (2130706433), and says in its comment that a regex list judges the bytes of the URL only, so the IP filter stays the authoritative check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2080.
The connection-time IP filter was the only check that looked at the address the fetcher actually connects to, and both of its keys shipped commented out in
crawler-default.yaml, so at defaults the interceptor was never installed. The regex exclusion list in the archetypes matched literallocalhostand four dotted-quad ranges and nothing else — link-local, CGNAT, IPv6 unique-local and the abbreviated and integer IPv4 forms the JVM resolver maps to 127.0.0.1 all passed it, andInetAddress.getByNamemaps127.1and2130706433to loopback without any DNS control.crawler-default.yamlshipshttp.filter.ipaddress.excludeenabled:localhost,sitelocal,linklocal,100.64.0.0/10,fd00::/8. A fetched page decides which hosts the fetcher connects to, and these ranges host unauthenticated services (e.g. cloud instance metadata at 169.254.169.254, reachable whatever DNS name a link used, since the check runs on the resolved socket address) which a public index must not leak intodefault-regex-filters.txt(and its opensearch/solr copies) gains rules for 169.254.0.0/16, 100.64.0.0/10, 0.0.0.0/8, IPv6fc00::/7andfe80::/10, abbreviated loopback (127.1) and integer IPv4 hosts (2130706433), and states that a regex list judges the bytes of the URL only — it can never enforce where a host name resolves, so the IP filter stays the authoritative checkRelease note needed: enabling the exclude list by default changes behaviour for anyone crawling an intranet or a loopback service; the way to opt out is documented next to the key.