Skip to content

🐛Origin dial fails instantly with i/o timeout when originRequest.connectTimeout is set via remote-managed tunnel config #1702

Description

@darkedges

Describe the bug
When a tunnel's ingress config is managed remotely (config_src: cloudflare, i.e. pushed via the API / Terraform's cloudflare_zero_trust_tunnel_cloudflared_config / the dashboard) and an ingress rule's originRequest.connectTi meout is set to any explicit value, every dial to that origin fails instantly with dial tcp: lookup <host>: i/o timeout — even though the origin is up and reachable the whole time.

The failure happens in well under 200ms, nowhere near any configured timeout, and strace -f -e trace=network on the cloudflared process shows zero socket()/connect() syscalls for the origin during the failed request —
the dial never reaches the OS network stack. This points to the dial's context.Context already being expired/cancelled before net.Dialer is invoked, consistent with a unit or sign error converting connectTimeout (documente
d as nanoseconds) into a deadline.

Removing connectTimeout from originRequest entirely (so cloudflared falls back to its own default) fixes it immediately, with no other change.

To Reproduce
Steps to reproduce the behavior:

  1. Configure a named tunnel with config_src: cloudflare and an ingress rule with an explicit originRequest.connectTimeout, e.g.:
    {
      "hostname": "app.example.com",
      "service": "http://origin-host:8080",
      "originRequest": {
        "connectTimeout": 10000000000,
        "httpHostHeader": "app.example.com"
      }
    }
    (10000000000 = 10s expressed in nanoseconds, per the documented unit for this field.)
  2. Run cloudflared tunnel run with the origin actually up and reachable.
  3. Request the hostname through the tunnel — see dial tcp: lookup <host>: i/o timeout instantly, every time.
  4. Tunnel ID: fbc8884a-aa1e-468a-b3bd-c93782598912
  5. cloudflared config (as pushed via the API, redirect/hostname redacted for this writeup but reproducible with any hostname/origin pair):
    {
      "ingress": [
        {
          "hostname": "echo-basic.zt.example.com",
          "originRequest": {
            "connectTimeout": 10000000000,
            "httpHostHeader": "echo-basic.zt.example.com"
          },
          "service": "http://echo-basic:8080"
        },
        { "service": "http_status:404" }
      ],
      "warp-routing": { "enabled": false }
    }

Expected behavior
The origin is dialed with (approximately) a 10-second connect timeout, and the request succeeds since the origin responds immediately.

Environment and versions

  • OS: Linux (WSL2, Ubuntu), also reproduced with cloudflared running as a bare host process outside any container
  • Architecture: AMD64
  • Version: reproduced on both 2026.7.3 (go1.26.4) and 2026.2.0 (go1.24.13) — not a recent regression

Logs and errors

INF Registered tunnel connection connIndex=0 connection=... protocol=quic
INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"echo-basic.zt.example.com\",\"originRequest\":{\"connectTimeout\":10000000000,\"httpHostHeader\":\"echo-basic.zt.example.com\"},\"service\":\"http://echo-basi
c:8080\"},...],\"warp-routing\":{\"enabled\":false}}" version=5
ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup echo-basic: i/o timeout" connIndex=0 event=1 ingressRule=0 originService=http://ech
o-basic:8080
ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup echo-basic: i/o timeout" connIndex=0 dest=https://echo-basic.zt.examp
le.com/health event=0 ip=198.41.192.167 type=http

strace -f -tt -e trace=network over the same window captures zero socket()/connect() calls referencing the origin hostname/IP/port at all — the process-level rejection happens before any syscall.

Additional context

  • Reproduced in three different contexts, ruling out Docker/network-driver causes:
    1. cloudflared inside a docker compose service, origin containers on the same user-defined bridge network, addressed by container hostname.
    2. Same setup, origin addressed by raw container IP instead of hostname (same failure, so not DNS-specific).
    3. cloudflared run as a bare process directly on the host (no Docker at all), origin addressed via localhost:<published-port> — same instant failure, including for localhost itself.
  • A separate nicolaka/netshoot container/process on the identical network resolves and connects to the same origin (ping, nslookup, nc -vz, curl) instantly and reliably throughout every failed cloudflared attempt, rul
    ing out the origin or network path.
  • The same cloudflared binary, copied into that working container and run standalone against the same tunnel/config, fails identically — ruling out the container image/base OS as a factor.
  • --max-active-flows <n> (appears to gate WARP private-network routing flows, not ordinary ingress) does not affect this — tested explicitly, no effect.
  • Origins specified locally rather than through remote config (cloudflared tunnel --url http://origin:port, i.e. quick tunnels / local config, which never set connectTimeout this way) were not observed to have this problem —
    the bug appears specific to how connectTimeout is parsed/applied when it arrives via the remotely-fetched ingress config.
  • Workaround: omit originRequest.connectTimeout from the ingress rule entirely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Priority: NormalMinor issue impacting one or more usersType: BugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions