Skip to content

Keep one client from slowing the edge proxy for every site #802

Open

Description

@HMarzban

Summary

On 2026-11-19 at 17:49Z every docs.plus site failed its health checks for about 30 s. A burst of outside traffic slowed the edge proxy (Traefik). The apps, the host and its network stayed healthy: Prometheus scraped the apps directly through the dip, and no container restarted. Found in the #747 24 h read.

Traefik serves every site, so a slow edge stops all of them at once. This issue adds three limits so one client cannot slow the edge for everyone.

Benchmark

Field Value
Metric During a burst from one client: the other routers' /health time (max), Traefik open connections and goroutines, and throttled CPU seconds per second
Before Production, 2026-11-19 17:49Z, from production Prometheus (the numbers stay in the private notes). Every router failed its health check from 17:49:13Z to 17:49:43Z
After Not measured yet
Condition Local burst test against the Traefik config of this repo, the same request shape, n = 5. Production read: no edge-wide dip in 7 days after the deploy
Pass rule The other routers keep answering /health under 1 s during the burst, in 5 of 5 runs

Steps

# Change Where Owner
E1 A per-client inFlightReq middleware on the public routers, starting with the TinyDocy WebSocket router. It caps how many open requests one client can hold scripts/traefik/dynamic/middlewares.yml and the router labels agent in the repo
E2 A CPU and memory budget for Traefik, sized from its 7-day peak docker-compose.prod.yml agent in the repo
E3 The TinyDocy WebSocket host answers a request that is not a WebSocket upgrade at once with 404 TinyDocy (outside this repo), or a Traefik route rule maintainer
E4 The local burst test, before and after, n = 5 local agent in the repo

E1 and E2 change static and dynamic Traefik config. If #746 has not shipped yet, ship them in its one Traefik restart. Otherwise they need their own restart window.

Risk

  • A cap set too low refuses real users behind one shared address, such as an office network. Size it from the 7-day peak of open connections per client, and log refusals.
  • E2 changes a resource limit on the one container that serves every site. Roll it back by restoring the old values in docker-compose.prod.yml.

Acceptance criteria

  • E1-E3 shipped.
  • E4 passes 5 of 5, posted as a Measured comment.
  • 7 days after the deploy: no edge-wide dip in the blackbox probes.

Related

Activity

  1. added this to the v2.3.7 milestone on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    DevOpsSecuritySecurity, access control, and data exposure

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions