Repository navigation
Open
Labels
DevOpsSecuritySecurity, access control, and data exposureSecurity, access control, and data exposure
Milestone
Description
Summary
On 2026-11-19 at 17:49Z every docs.plus site failed its health checks for about 30 s. A burst of outside traffic slowed the edge proxy (Traefik). The apps, the host and its network stayed healthy: Prometheus scraped the apps directly through the dip, and no container restarted. Found in the #747 24 h read.
Traefik serves every site, so a slow edge stops all of them at once. This issue adds three limits so one client cannot slow the edge for everyone.
Benchmark
| Field | Value |
|---|---|
| Metric | During a burst from one client: the other routers' /health time (max), Traefik open connections and goroutines, and throttled CPU seconds per second |
| Before | Production, 2026-11-19 17:49Z, from production Prometheus (the numbers stay in the private notes). Every router failed its health check from 17:49:13Z to 17:49:43Z |
| After | Not measured yet |
| Condition | Local burst test against the Traefik config of this repo, the same request shape, n = 5. Production read: no edge-wide dip in 7 days after the deploy |
| Pass rule | The other routers keep answering /health under 1 s during the burst, in 5 of 5 runs |
Steps
| # | Change | Where | Owner |
|---|---|---|---|
| E1 | A per-client inFlightReq middleware on the public routers, starting with the TinyDocy WebSocket router. It caps how many open requests one client can hold |
scripts/traefik/dynamic/middlewares.yml and the router labels |
agent in the repo |
| E2 | A CPU and memory budget for Traefik, sized from its 7-day peak | docker-compose.prod.yml |
agent in the repo |
| E3 | The TinyDocy WebSocket host answers a request that is not a WebSocket upgrade at once with 404 |
TinyDocy (outside this repo), or a Traefik route rule | maintainer |
| E4 | The local burst test, before and after, n = 5 | local | agent in the repo |
E1 and E2 change static and dynamic Traefik config. If #746 has not shipped yet, ship them in its one Traefik restart. Otherwise they need their own restart window.
Risk
- A cap set too low refuses real users behind one shared address, such as an office network. Size it from the 7-day peak of open connections per client, and log refusals.
- E2 changes a resource limit on the one container that serves every site. Roll it back by restoring the old values in
docker-compose.prod.yml.
Acceptance criteria
- E1-E3 shipped.
- E4 passes 5 of 5, posted as a Measured comment.
- 7 days after the deploy: no edge-wide dip in the blackbox probes.
Related
- Move production to a new Debian 13 droplet #747: the 24 h read that found the dip.
- Upgrade production Traefik to v3.7.18, with every static edge change in one restart #746: the Traefik upgrade, which restarts Traefik once for every static edge change.
- Turn off SSH password login on the production host, and add fail2ban #744: SSH hardening on the same host.
Metadata
Metadata
Assignees
Labels
DevOpsSecuritySecurity, access control, and data exposureSecurity, access control, and data exposure
Activity