Skip to content

fix(ssl): keep failed and pending Let's Encrypt sites on HTTP, wait for DNS-01 propagation and keep the www redirect - #521

Open
mrrobot47 wants to merge 10 commits into
EasyEngine:developfrom
mrrobot47:fix/ssl-le-dns-fail-closed
Open

mrrobot47 wants to merge 10 commits into
EasyEngine:developfrom
mrrobot47:fix/ssl-le-dns-fail-closed

Conversation

@mrrobot47

@mrrobot47 mrrobot47 commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Pairs with EasyEngine/site-type-wp#247, which keeps the WordPress URLs on http while the site is served over HTTP; both should ship in the same release.

Problem

A site created with --ssl=le using the DNS-01 challenge through Cloudflare, in a zone that has a wildcard record (*.<zone> CNAME <zone>), could end up half on HTTPS and later lose its www redirect.

  1. Validation raced DNS propagation. EasyEngine added the TXT records and waited a fixed 10 seconds. Until _acme-challenge.www.<site> is served everywhere, that name matches the wildcard and is answered with the target's TXT records (the apex's), so Let's Encrypt validated the site name but failed on www with "Incorrect TXT record ... found at _acme-challenge.www...".
  2. A failed create did not fail closed. init_le() cleared site_ssl and returned, but www_ssl_wrapper() still wrote the compose file without HTTPS_METHOD=nohttps and restarted the site. The site was saved with SSL "Not Enabled", while nginx-proxy served it over HTTPS with the closest matching certificate of another name, so browsers showed a name-mismatch error. update --ssl=le already rolls back through disable_ssl(); create had no such path.
  3. The www redirect was lost on the next attempt. A later ee site update <site> --ssl=le ran the www check, which follows redirects: http://www.<site> → http://<site> → https://<site>. The last hop failed certificate verification on the wrong certificate, so the check decided www wasn't pointed at the server, removed <site>-redirect.conf, and the new certificate was issued with www but without the HTTPS www redirect.
  4. The "Cleaned up DNS record" message printed _acme-challenge. twice.
  5. Manual DNS-01 left HTTPS on. Without a Cloudflare API key, a DNS-01 create or update --ssl=le saved the site as Let's Encrypt and returned before any certificate existed, but still wrote the compose file with HTTPS on, so nginx-proxy redirected the site to HTTPS with the closest matching certificate until ee site ssl-verify succeeded.
  6. Wrong retry hints. After a failed create, EasyEngine suggested ee site ssl-verify, which can't work then (the site has no SSL), and a failed update --ssl=le printed overlapping messages.

Changes

  • Fail closed (www_ssl_wrapper()): when the certificate step leaves site_ssl empty on create or update, the compose file is written with nohttps, the containers are recreated, the HTTP-only www redirect is kept and the HTTPS redirect is skipped. On create, a warning says the site is served over HTTP only and gives the ee site update <site> --ssl=le command to run once the issue is fixed. On update, update_ssl() does the rollback (disable_ssl()) and reports the failure once, and the DB keeps the site without SSL. Renewals (ssl-renew, alias changes) are unchanged: their caller restores the previous certificate, which keeps serving over HTTPS.
  • DNS-01 propagation: the fixed wait is replaced by polling. For every TXT record the Cloudflare solver actually created, the zone's authoritative nameservers are found by walking up from the record name (NS records whose owner is the name itself, so a wildcard match doesn't count), and each IPv4 address of each nameserver is queried directly without recursion, so no resolver cache is involved. EasyEngine proceeds once every nameserver serves every expected value (a wildcard certificate has two values on the same name), with backoff (2, 4, 8, 16 s) and a 120 s limit. The old 10 seconds are kept as a minimum after that: anycast nameservers answer from the site nearest to this server, while Let's Encrypt validates from several regions. The queries set RD, which authoritative servers ignore, so a network that redirects port 53 to its own resolver still gets answers. On timeout it names the records and nameservers still missing and tries the validation anyway, as before, so the existing ssl-verify hints still apply. Rebuilding a stale order in check() waits the same way. The queries use a small DNS client in site-utils.php (UDP with an EDNS0 buffer, TCP retry on a truncated answer, ID and response checks), so there's no new dependency, and nothing runs when no record was created.
  • www check: redirects to HTTPS are followed without certificate verification, as Let's Encrypt's HTTP-01 validator does. The random file content still has to come back, so a name served by another server still counts as not pointed. The anti-symlink handling and the exclusive file creation are unchanged.
  • The cleanup messages print the record name once.
  • Pending manual DNS-01 certificate: site_ssl stays le, so ssl-verify, ssl-info and delete keep working, and a missing certificate marks the site as pending (is_le_ssl_pending()). www_ssl_wrapper() keeps such a site on HTTP (compose nohttps, HTTP-only www redirect) and says it is served over HTTP until ee site ssl-verify <site> succeeds. ee site refresh/enable --refresh, update --php and alias domain updates keep nohttps while it is pending. A standalone ssl-verify that issues the certificate turns HTTPS on (compose file, nginx container, HTTPS www redirect when www points here) and calls the new site_scheme_changed() hook, which site types that store the site URL override (site-type-wp switches the WordPress URLs). A renewal or alias update that issues the first certificate (e.g. after a Cloudflare key is set) calls it too. A failed renewal of a pending site stays on HTTP (there is no certificate to keep serving), and if HTTPS activation fails after ssl-verify issued the certificate, the redirect and the hook still run, so ee site refresh completes it. update --ssl=le on a pending site points to ssl-verify, a pending update no longer says SSL is enabled, and a site can't inherit the certificate of a pending parent.
  • Messages: the ee site ssl-verify hints are gone from init_le() and from the order/validation failures in Site_Letsencrypt; the caller says how to retry (create: ee site update <site> --ssl=le; update: re-run the update; renewal: the current certificate is kept). A standalone ssl-verify still prints its own retry hint, and the manual DNS-01 "run ssl-verify once the DNS changes have propagated" message is unchanged (now followed by a newline).

Testing

  • Unit tests (PHP 7.4 and 8.5, throwaway containers): pending state (create and renewal stay HTTP, a renewal waiting for DNS keeps the current certificate, a pending update saves le and stays HTTP, update on a pending site points to ssl-verify, a failed update rolls back and reports once without ssl-verify hints, ssl-verify turns HTTPS on with or without the www redirect and calls the hook, a renewal that issues the first certificate calls the hook, a failed renewal of a pending site stays HTTP, a failed HTTPS activation still switches the URLs and the redirect, refresh keeps a pending site on HTTP, a pending inherit parent is refused, authorization failures print no ssl-verify hint); fail-closed matrix (failed create with and without www, wildcard hint, successful create, failed renewal keeps HTTPS, enable untouched, still-valid certificate kept, failed update ends without SSL and without any HTTPS compose write); the www check through http → http → https with a self-signed certificate for another name (pointed when the file comes back, not pointed when it doesn't); the DNS poll against a fake authoritative server (wildcard answer first and the record after 3 s, the 10 s minimum, the RD flag, never appearing, a lagging second nameserver, two values on one name, truncated UDP answer retried over TCP, NXDOMAIN, an unreachable server, malformed responses); the Cloudflare solver's created records and cleanup message. The fail-closed, www and cleanup tests fail on develop.
  • Against two CoreDNS servers serving a zone with * CNAME <apex>: nameserver discovery finds both servers, the wildcard answer is rejected, the wait returns once the second server has the record (and not before 10 s), times out when the record never appears or only one server has it, and a 3.6 KB answer falls back to TCP.
  • Behat on a hosted runner with Pebble (Let's Encrypt's test server; the ACME server's HTTP-01 requests for the www name are rejected to make www validation fail):
    • create with a failing www validation: the site stays HTTP-only, the compose file has nohttps, the redirect conf has no HTTPS block, https://<site> gets nginx-proxy's default error instead of the site; then update --ssl=le issues a certificate covering www and the HTTPS www redirect works;
    • a site already redirected to HTTPS with another name's certificate keeps its www redirect on update --ssl=le;
    • a WordPress create with a failing www validation stays HTTP-only with http URLs and serves the site over HTTP; update --ssl=le then issues the certificate, switches the URLs to https and redirects http to https (with site-type-wp#247);
    • a manual DNS-01 WordPress create stays HTTP-only with http URLs, update --ssl=le points to ssl-verify, and once the TXT records are published ssl-verify issues the certificate, turns HTTPS on with the www redirect and switches the URLs to https;
    • the existing Let's Encrypt scenarios (normal create, alias add/remove, update, renew) pass unchanged, as do the main and WordPress lanes (PHP 7.4 and 8.5: main 190/190, WordPress 10/10, Let's Encrypt 16/16).
    • On develop the same three new scenarios fail: the failed create isn't kept on HTTP, and the update of the site already redirected to HTTPS issues the certificate without www.

… issued

When Let's Encrypt registration, authorization or validation failed during `ee site create --ssl=le`, init_le() cleared site_ssl, but www_ssl_wrapper() still wrote the compose file without HTTPS_METHOD=nohttps. nginx-proxy then served the site over HTTPS with the closest matching certificate of another site (a name-mismatch error in browsers), while the site was saved with SSL not enabled.

The wrapper now writes the compose file with nohttps, keeps the HTTP-only www redirect and skips the HTTPS redirect when no certificate was issued, and says how to enable SSL once the issue is fixed. Renewals are unchanged: their caller restores the previous certificate, which keeps serving.
…eservers

With the Cloudflare DNS solver, EasyEngine waited a fixed 10 seconds after adding the TXT records and then asked Let's Encrypt to validate. Until a new `_acme-challenge.www.<site>` record is served everywhere, a zone with a wildcard record answers that name with the wildcard's target, e.g. the apex's TXT records, so validation of the www name failed with "Incorrect TXT record".

The fixed wait is replaced by polling: the nameservers of the zone that holds each record are looked up, and each of their IPv4 addresses is queried directly, without recursion, until all of them serve the expected value, with backoff and a 120 second limit. On timeout the validation is tried anyway, as before, and the records and nameservers still missing are named. Rebuilding a stale order in check() waits the same way. The queries use a small UDP client with an EDNS0 buffer and a TCP retry for truncated answers, so no new dependency is needed.
…eck fails

The www check fetches a random file through the www name and follows redirects. On a site that is already redirected to HTTPS but has no matching certificate yet, the chain http://www → http://<site> → https://<site> failed certificate verification, so the check decided www was not pointed at the server, removed the site's www redirect and issued the certificate without it.

The check now follows a redirect to HTTPS without verifying the certificate, as Let's Encrypt's HTTP-01 validator does. The random file content still has to come back, which is what proves the name reaches this site.
…in the nameserver queries

Every authoritative nameserver of a zone can answer from the anycast site nearest to this server, while Let's Encrypt validates from several regions, so the time the fixed wait used to give is kept as a minimum once the values are served. The queries set RD: authoritative servers ignore it, and a network that redirects port 53 to its own resolver would otherwise only answer from its cache. Pending signal handlers now run after each query.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The custom DNS protocol implementation and production SSL failure and recovery paths warrant final human review.

No open issues 🎉

What changed in this PR

This PR improves EasyEngine’s Let’s Encrypt failure recovery and DNS validation, keeping failed certificate setup on HTTP while preserving the www redirect.

Changes:

  • Keeps sites HTTP-only after failed certificate setup on create or update, without changing renewal behavior.
  • Replaces the fixed DNS wait with TXT propagation polling and a DNS client supporting UDP and TCP fallback.
  • Allows www checks through HTTPS redirects with invalid certificates and corrects DNS cleanup messages.
File Description
src/​helper/​site-utils.php Adds authoritative nameserver discovery, TXT polling, and DNS transport and parsing.
src/​helper/​Site_Letsencrypt.php Tracks published DNS records and waits after rebuilding stale orders.
src/​helper/​SimpleDnsCloudflareSolver.php Exposes created TXT records and corrects cleanup messages.
src/​helper/​class-ee-site.php Adds HTTP-only failure recovery, integrates DNS polling, and adjusts www checks.

Review effort: Balanced

…ts certificate

Without a Cloudflare API key, a DNS-01 create or `update --ssl=le` saves the site as Let's Encrypt and returns before any certificate exists, but still wrote the compose file with HTTPS on. nginx-proxy then redirected the site to HTTPS and served it with the closest matching certificate of another site until `ee site ssl-verify` succeeded.

Such a site now stays on plain HTTP: `site_ssl` stays `le`, so ssl-verify, ssl-info and delete keep working on it, and a missing certificate marks it as pending (`is_le_ssl_pending()`). www_ssl_wrapper() writes the compose file with `nohttps` for it, and site refresh/enable, the PHP version update and alias domain updates keep `nohttps` while it is pending. Once a standalone `ee site ssl-verify` issues the certificate, it turns HTTPS on (compose file, nginx container, HTTPS www redirect when www points here) and calls the new `site_scheme_changed()` hook, so site types that store the site URL can switch it.

`update --ssl=le` on a pending site points to ssl-verify instead of saying SSL is already enabled, and a pending update no longer reports SSL as enabled.
After a failed create, init_le() and the ACME client suggested `ee site ssl-verify`, which can't work there: the site has no SSL then. The site-level messages now come from the caller: a create says to run `ee site update <site> --ssl=le`, `update --ssl=le` says to re-run itself, a renewal keeps the current certificate. A standalone ssl-verify still prints its own retry hint, and the manual DNS-01 "run ssl-verify once the DNS changes have propagated" message is unchanged.

A failed `update --ssl=le` printed the create-time "served over HTTP only" warning and then update_ssl()'s own error, and rolled the site back twice. www_ssl_wrapper() now leaves both to update_ssl().
…ificate, and refuse to inherit from it

A pending manual DNS-01 site can also get its first certificate from `ee site ssl-renew` or an alias domain update, e.g. once a Cloudflare API key is set. www_ssl_wrapper() then turns HTTPS on, but only a standalone ssl-verify told the site type, so WordPress kept its http URLs. reissue_le_certificate() now calls site_scheme_changed() too when the site was pending before and is issued now.

A site can't inherit the certificate of a pending parent: it would get HTTPS without any certificate to serve.
@mrrobot47 mrrobot47 changed the title fix(ssl): keep failed Let's Encrypt sites on HTTP, wait for DNS-01 propagation and keep the www redirect fix(ssl): keep failed and pending Let's Encrypt sites on HTTP, wait for DNS-01 propagation and keep the www redirect Oct 23, 2026
@rtBot
rtBot requested a balanced review from Copilot October 23, 2026 18:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failed pending renewals can enable HTTPS without a certificate, and activation recovery can leave stored site URLs on HTTP.

2 open issues

Review effort: Balanced

Comment thread src/helper/class-ee-site.php Outdated
Comment thread src/helper/class-ee-site.php Outdated
…tch its URLs when HTTPS activation fails after issuance

A pending site has no previous certificate for a failed `ssl-renew --force` to keep serving, but www_ssl_wrapper() still turned HTTPS on for it, as for any renewal. It now fails closed when no certificate exists.

When ssl-verify issued the certificate but the compose file or the nginx start failed, the HTTPS redirect and site_scheme_changed() were skipped. ssl-verify won't see the site as pending again and the suggested refresh doesn't call the hook, so WordPress kept its http URLs. Both now run before the error.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved rollback and concurrency issues, alongside the new DNS transport and cross-command certificate transitions, require fixes and final human validation.

3 open issues
2 resolved since last review

Review effort: Balanced

Comment thread src/helper/class-ee-site.php
Comment thread src/helper/class-ee-site.php
Comment thread src/helper/class-ee-site.php
…issuing its first certificate

If the container start fails after www_ssl_wrapper() issued a pending site's first certificate during a renewal or alias update, reissue_le_certificate() restores the certificate files, which removes that certificate, but the compose file already served HTTPS. The rollback now writes the HTTP-only compose file again.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The custom DNS client, cross-command SSL activation and rollback paths, and coordinated WordPress release warrant final human validation.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants