Run redirect hops through the URL filters - #2128
Open
abhinav-phi wants to merge 1 commit into
Open
Conversation
With http.allow.redirects enabled, okhttp followed the whole redirect chain inside the client and the crawler only ever saw the final response, returned under the original URL: no intermediate target went through the URL filters, so the scheme exclusions, host confinement and depth rules configured for the crawl did not apply to the hops a fetched page steered the fetcher to. Redirects are now followed manually in getProtocolOutput: each target is resolved, run through the URL filter chain and only fetched when accepted; a rejected target ends the chain and the redirect response is returned as is, which the caller handles like it does when immediate follows are off. The chain is capped at 5 hops, and the final URL is recorded in the response metadata under _redirTo so callers can tell the content is not from the URL they asked for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2088.
With
http.allow.redirectsenabled, okhttp followed the whole redirect chain inside the client and the crawler only ever saw the final response, returned under the original URL. No intermediate target went throughURLFilters, so the scheme exclusions, host confinement and depth rules configured for the crawl did not apply to the hops a fetched page steered the fetcher to — an operator who turned the setting on to keep redirect chains out of the status store also turned off URL policy for those hops, two unrelated concerns.Redirects are now followed manually in
getProtocolOutput(the client's internal following is disabled):_redirTo, so callers can tell that the content is not from the URL they asked for; the cookie origin is scoped per hop to the URL that actually answeredThe
IPFilterRulesinterceptor already ran per hop as a network interceptor; it keeps doing so. Docs updated (configuration.adoc, internals.adoc) — the previous wording said immediate follows bypass URL filtering, which is no longer true for OkHttp.Note: chains longer than 5 hops stop being followed in-client; their last redirect is handed to the status store, so no URL is lost.