proxysocketendpoint: graceful shutdown unlinks the socket path even when another instance now owns it

[jpe0824]

Setup: unix-socket mode (-proxysocketendpoint), one instance per node as a Docker Swarm global service, socket in a shared tmpfs volume consumed by a collector.

Problem: when two instances' lifetimes overlap on the same path, the exiting one deletes the socket the new one just bound.

1. New instance starts, logs already exists, removing existing file, binds, logs running and listening...
2. Old instance finishes its -shutdowngracetime drain and exits 0
3. srv.Shutdown() closes the listener, and Go's net.UnixListener.Close() calls syscall.Unlink(path) unconditionally — no check that the inode at that path is still the one it created (the stdlib documents this race in a comment in unixsock_posix.go)

The surviving instance keeps serving an orphaned inode and logs nothing wrong; the socket path simply no longer exists, so every client fails. Observed twice in production on a swarm cluster (leader election and node rejoin, both of which replace global tasks with overlapping lifetimes).

Suggested fix: l.(*net.UnixListener).SetUnlinkOnClose(false) after net.Listen, then remove the file explicitly on shutdown only when it is still the instance's own socket — e.g. compare os.Stat inode (Sys().(*syscall.Stat_t).Ino) against the value recorded right after bind, and skip the unlink on mismatch.

Also worth noting: neither -watchdoginterval nor -allowhealthcheck can detect this state, because startSocketWatchdog and healthCheckServer are both called with cfg.SocketPath (the upstream docker socket) rather than cfg.ProxySocketEndpoint. A watchdog/health option covering the proxy's own endpoint would let an orchestrator restart a stranded instance.

Verified against main (1.13.1); reproduced on 1.12.3.

[wollomatic]

Hi @jpe0824

thanks for the detailed report. I'll look into it at the beginning of September.

Regards,
Wolfgang