OKHttp: authenticated TLS by default, credentials withheld from unauthenticated servers

[abhinav-phi]

Fixes #2079

The problem

HttpProtocol.configure() read http.trust.everything with a default of true and, when set (which was the default), installed two things at once:

1. a no-op X509TrustManager built on a static SSLContext.getInstance("SSL"), accepting any certificate chain — self-signed, expired, revoked, or issued for a completely different name; and
2. a HostnameVerifier that returned true for every host name.

Because these two travelled together, a single undocumented key silently disabled both halves of server authentication, with three consequences:

• Credentials leaked to unauthenticated servers. The Authorization header built from http.basicauth.user/http.basicauth.password, credential headers configured through http.custom.headers, headers set per request via metadata, and cookies replayed from http.use.cookies were all sent on connections where nobody had verified who was on the other end. Anybody able to answer for the host name — a DNS hijack, a hostile network on path, an intercepted robots.txt redirect — received them. This was verified in the issue: a crawl with basic auth pointed at an HTTPS server presenting a self-signed certificate showed the Authorization header arriving.
• The host name was never checked, independently of chain validation. Even a perfectly valid certificate issued for a different name was accepted: an additional loss of server identity beyond the trust-all manager.
• The context used the protocol string "SSL" rather than "TLS", a relic that limits the enabled protocol families.

And as the issue notes, the key was set in no shipped YAML and nothing was logged: operators inherited all of this rather than choosing it.

The fix

• http.trust.everything now defaults to false. Certificate chains are validated and servers are authenticated, as a normal TLS client would do. Crawling hosts with unvalidatable certificates (e.g. self-signed intranet ones) remains possible — through an explicit, documented opt-in.
• The trust-all context is built from TLS instead of SSL.
• Hostname verification is a separate decision. The new http.verify.hostnames key (default true) controls it independently: trusting any certificate no longer implies accepting any name. Disabling it is a separate, separately-logged decision.
• Credentials are withheld from unauthenticated servers. When the trust-all manager is active, the protocol stops sending basic auth, credential headers (authorization, proxy-authorization, cookie, x-api-key — case-insensitive) from http.custom.headers, credential headers set per request, and cookies, unless the operator explicitly sets http.credentials.allow.insecure: true. Ordinary headers (user agent, accept, custom non-credential ones, …) are unaffected.
• Every insecure state logs a WARN naming the exact configuration keys involved, so the behaviour is visible in the logs rather than silent.
• The keys are surfaced: http.trust.everything, http.verify.hostnames and http.credentials.allow.insecure are documented in crawler-default.yaml, the archetype crawler-conf.yaml files (core, opensearch, solr) and configuration.adoc; the SSL/TLS and Authentication sections of extending.adoc now describe the interaction.

Behaviour notes

• http.trust.everything: true + http.verify.hostnames: false restores the old permissive combination, but each half now needs its own explicit key and each logs a WARN.
• Self-signed certificates do not require disabling hostname verification when the certificate matches the host name; the two concerns are orthogonal.
• Proxy authentication (Proxy-Authorization built by the proxy authenticator) is untouched: it goes to the configured proxy, not to crawled servers.

Testing

The new OkHttpTrustEverythingTest (11 tests) serves HTTPS locally with two self-signed PKCS12 keystores — one issued for localhost, one for another host name — and covers:

• trustAllContextUsesTls — the static context is a TLS context;
• selfSignedCertificateRejectedByDefault — the new default rejects the handshake;
• trustAllFetchesServerWithSelfSignedCertificate — the opt-in still crawls such hosts;
• hostnameIsStillVerified — under trust-all, a certificate for another name is still rejected (this is the regression test from the issue);
• hostnameVerificationCanBeDisabledSeparately — the separate key works;
• basic auth, credential custom headers and cookies — each withheld from unauthenticated servers and sent again when http.credentials.allow.insecure is set, verified against the server's request journal.

mvn -pl core verify passes: 430 tests, checkstyle, forbiddenapis and google-java-format clean.

Release note (required, behaviour change)

The default of http.trust.everything changed from true to false in this major release. Crawls relying on self-signed or otherwise unvalidatable certificates must set:

http.trust.everything: true

and, if they authenticate with basic auth, custom credential headers or cookies, additionally:

http.credentials.allow.insecure: true

Hostname verification stays enabled unless http.verify.hostnames: false is set explicitly.