Skip to content

Run redirect hops through the URL filters - #2128

Open
abhinav-phi wants to merge 1 commit into
apache:mainfrom
abhinav-phi:issue-2088-redirect-filtering
Open

Run redirect hops through the URL filters#2128
abhinav-phi wants to merge 1 commit into
apache:mainfrom
abhinav-phi:issue-2088-redirect-filtering

Conversation

@abhinav-phi

Copy link
Copy Markdown
Contributor

Fixes #2088.

With http.allow.redirects enabled, okhttp followed the whole redirect chain inside the client and the crawler only ever saw the final response, returned under the original URL. No intermediate target went through URLFilters, so the scheme exclusions, host confinement and depth rules configured for the crawl did not apply to the hops a fetched page steered the fetcher to — an operator who turned the setting on to keep redirect chains out of the status store also turned off URL policy for those hops, two unrelated concerns.

Redirects are now followed manually in getProtocolOutput (the client's internal following is disabled):

  • each target is resolved against the URL it was reached from and run through the URL filter chain; only accepted targets are fetched
  • a rejected target ends the chain: the redirect response itself is returned, which the caller handles exactly like it does when immediate follows are off (emits the target to the status stream, where the full policy applies)
  • the chain is capped at 5 hops (a chain which does not end within that returns its last redirect response)
  • the final URL is recorded in the response metadata under _redirTo, so callers can tell that the content is not from the URL they asked for; the cookie origin is scoped per hop to the URL that actually answered

The IPFilterRules interceptor already ran per hop as a network interceptor; it keeps doing so. Docs updated (configuration.adoc, internals.adoc) — the previous wording said immediate follows bypass URL filtering, which is no longer true for OkHttp.

Note: chains longer than 5 hops stop being followed in-client; their last redirect is handed to the status store, so no URL is lost.

With http.allow.redirects enabled, okhttp followed the whole redirect
chain inside the client and the crawler only ever saw the final
response, returned under the original URL: no intermediate target went
through the URL filters, so the scheme exclusions, host confinement
and depth rules configured for the crawl did not apply to the hops a
fetched page steered the fetcher to.

Redirects are now followed manually in getProtocolOutput: each target
is resolved, run through the URL filter chain and only fetched when
accepted; a rejected target ends the chain and the redirect response
is returned as is, which the caller handles like it does when
immediate follows are off. The chain is capped at 5 hops, and the
final URL is recorded in the response metadata under _redirTo so
callers can tell the content is not from the URL they asked for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redirect hops taken inside okhttp are not run through the URL filters

1 participant