fix(net): pass through connected AF_UNIX sendmsg under a destination policy

[dzerik]

Problem

With any --net-allow/--net-deny rule active, a sendmsg() on an already-connected AF_UNIX stream socket fails with ECONNREFUSED. connect() and recvmsg() on the same socket succeed — only the outbound sendmsg() is rejected. Since libwayland and D-Bus write their protocol via sendmsg (for the fd-passing capability), the client connects, sends its request, and then hangs waiting for a reply the peer never received. This is the GUI breakage reported in #121 (native Wayland clients such as rofi).

send()/sendto were unaffected — they Continue connected sockets unconditionally — which is why the breakage looked selective.

Minimal repro (any live pathname unix socket):

sandlock run --net-allow 127.0.0.1 -w / -- python3 -c \
 'import socket; s=socket.socket(socket.AF_UNIX,socket.SOCK_STREAM); s.connect("/run/user/1000/wayland-0"); print("send",s.send(b"x")); print("sendmsg",s.sendmsg([b"x"]))'
# send 1
# ConnectionRefusedError: [Errno 111]   <-- sendmsg only

Root cause

In sendmsg_on_behalf, the connected / non-IP-family "Continue" fast path is gated behind if !dest_policy. With a destination policy active, a connected AF_UNIX send skips that path, is dup'd, and reaches query_socket_protocol, which returns None for a unix socket (no IP protocol) — treated as an unknown/forbidden protocol and refused with ECONNREFUSED. sendmmsg_on_behalf has the same gap.

Fix

The IP destination policy governs IP sockets only; a --net-allow rule must not change how AF_UNIX sockets are handled. An AF_UNIX send is already fully decided by the named-unix fs gate (a named target is taken on-behalf; a connected/abstract send has no destination to validate). Probe SO_DOMAIN and Continue for unix sockets before the IP path, in both sendmsg_on_behalf and sendmmsg_on_behalf. This mirrors sendto, which only takes IP-family destinations on-behalf.

IP enforcement is untouched: non-unix sockets still resolve their protocol and validate every destination. Verified end-to-end that connected AF_UNIX sendmsg passes through under --net-allow (Wayland registry roundtrip restored) while denied IP destinations are still blocked.

Test

Adds an integration test: a connected AF_UNIX stream sendmsg under an active net_allow must pass through and return the byte count.

[congwang-mk]

TOCTOU: this Continue under an active destination policy is racy and, I think, can't be made safe. On RESP_CONTINUE the kernel re-resolves arg0 (and msg_name) against the live child after this check. A malicious child can race dup2(inet_sock, sockfd) right after the SO_DOMAIN probe, so the kernel runs the send on an IP socket to a denied destination. That's exactly the "never Continue under dest_policy" invariant (lines 1314-1318 / 1609-1614) being broken; no Continue-based fix closes it, since both fd and msg_name are child-controlled at execution time.

The actual defect is narrow: query_socket_protocol is required eagerly and hard-fails ECONNREFUSED just below, but protocol is only consumed in the non-connected IP branch of send_msghdr_on_behalf. Suggested redirection, staying on the TOCTOU-safe on-behalf path:

1. Make protocol: Option<Protocol>, require it only in the !connected branch.
2. Translate SCM_RIGHTS: parse the cmsgs, pidfd_getfd each child fd into the supervisor, rewrite with supervisor fd numbers, close after send.

Step 2 is needed either way: on-behalf sends copy the control buffer raw, so fd-passing is already silently corrupted here and in send_named_unix_msghdr. Doing it on-behalf closes the race and fixes that gap in one place. Heads up that the repro and the new test only exercise the no-fd case, so neither approach is verified against an actual fd-passing sendmsg.

[dzerik]

You're right on the TOCTOU — Continue under a destination policy can't be made safe, since the kernel re-resolves both sockfd and msg_name against the live child after our check, so a racing dup2(inet_sock, sockfd) redirects the send. Reworked to stay on the on-behalf path exactly as you suggested:

• send_msghdr_on_behalf now takes protocol: Option<Protocol> and consumes it only in the !connected IP branch. A connected send (every AF_UNIX send that reaches here — its connection was gated at connect time) needs no protocol and goes out on the immune dup_fd; a non-connected send whose protocol can't be resolved still fails closed with ECONNREFUSED. No Continue under a destination policy in either sendmsg_on_behalf or sendmmsg_on_behalf.
• SCM_RIGHTS translation: translate_scm_rights pidfd_getfds each fd into the supervisor and rewrites its number before the send, on both the IP and named-unix on-behalf paths. As you noted, the control buffer was being copied verbatim, so fd-passing was silently corrupted — this closes that too.

Your heads-up that the repro/test only exercised the no-fd case was the useful nudge: I verified end-to-end that a real SCM_RIGHTS send now delivers a working fd (the receiver reads the passed file's exact bytes), and added that as an integration test.

Hardened the parser against a hostile child while I was in there:

• cmsg_len is bounds-checked against the remaining buffer (no off + cmsg_len overflow); a malformed header fails closed with EINVAL instead of panicking / scanning out of bounds. Unit-tested with a usize::MAX-ish cmsg_len.
• SCM_CREDENTIALS is rejected with EPERM rather than forwarded — on the on-behalf path the supervisor is the sender, so passing the child's crafted pid/uid/gid would let a privileged supervisor forge credentials the child can't assert. Failing closed is the safe choice.
• An oversized control buffer (> 16 KiB) fails closed with EMSGSIZE rather than being truncated into a partial cmsg chain.

IP enforcement is unchanged — non-unix sockets still resolve their protocol and validate every destination (verified: a denied UDP sendmsg is still blocked). Force-pushed; CI green.

[congwang-mk]

Thanks for your quick followup. Here are some additional issues:

[congwang-mk]

Should this be return Err(EIO)? Otherwise it would silently send the message unexpectedly?

[congwang-mk]

Should we check it against AF_UNIX so that translate_scm_rights() would be called only for Unix socket?

[congwang-mk]

This code is duplicated with the above in send_named_unix_msghdr(), make it a helper?

[congwang-mk]

NOT caused by this PR, but a related issue found by AI is:

A blocking on-behalf send can stall the entire supervisor loop

Seccomp notifications are processed sequentially on a single loop
(notif.rs: "Notifications are processed sequentially (not spawned)").
handle_net never returns NotifAction::Defer, so the on-behalf sends in
network.rs (libc::sendmsg in send_msghdr_on_behalf and
send_named_unix_msghdr, libc::sendto in sendto_on_behalf) execute
inline on that loop.

dup_fd_from_pid (pidfd_getfd) shares the open file description, so the
supervisor inherits the child's blocking mode. If the child's socket is
blocking and the send buffer is full (peer not draining), the
supervisor blocks indefinitely: no timeout applies (DEFER_TIMEOUT only
covers deferred work), and every trapped syscall from every sandboxed
process hangs until the peer reads or closes.

This predates PR #125 (reachable via connected TCP sendmsg under a
destination policy), but #125 amplifies it: connected AF_UNIX stream
sendmsg (Wayland, D-Bus) now routes through this path, so a stalled
peer freezes the whole sandbox, and a child can trigger it with no
network permission at all (fill one end of a socketpair via untrapped
write(), then sendmsg).

Nothing is bypassed; the impact is denial of service of enforcement.

Suggested fix: run the on-behalf send as NotifAction::Defer (bounds a
wedge to one defer slot and 30s). Faithful version: force MSG_DONTWAIT;
on EAGAIN with a blocking child, defer and await writability on the
dup'd fd (AsyncFd) before retrying, preserving blocking semantics
without occupying the loop.

Please let me know if you want to fix it together here. (If not, I can fix it separately.)

Thanks!

[dzerik]

Thanks — addressed the inline comments and took the supervisor-loop issue on here (happy to do it together).

Inline:

• Silent send on unreadable control → now returns EIO. Folded the duplicated control handling of send_msghdr_on_behalf and send_named_unix_msghdr into one materialize_control(), which fails closed (EIO on an unreadable buffer, EMSGSIZE on an oversized one) instead of sending without the cmsgs.
• Translate only for unix sockets → materialize_control now translates SCM_RIGHTS / rejects SCM_CREDENTIALS only when the socket is AF_UNIX (probed via SO_DOMAIN); an IP socket's control (e.g. IP_PKTINFO) passes through untouched.
• Dedup → done, via that shared helper.

Supervisor loop (blocking on-behalf send): you're right, and it's a real DoS — a blocking child sending to a peer that stops draining wedged the whole sequential loop, reachable with no net permission at all, and #125 widened it to connected AF_UNIX (Wayland/D-Bus). Went with the faithful version:

• Each send is materialized into an owned message (flattened payload + translated control + inode pin + sockaddr), so it can be replayed off the loop.
• The first attempt is MSG_DONTWAIT on the loop, so it never blocks there. For a blocking child whose message didn't fully fit, it returns NotifAction::Defer — the deferred future awaits writability via AsyncFd (epoll, never blocking a worker thread) and pushes the rest, advancing a byte offset past each partial send until the full length is delivered, so a blocking send(N) still returns N. Control/address are attached only at offset 0, so SCM_RIGHTS transmits once. A non-blocking child gets the single-attempt result (short count / EAGAIN), as the kernel would give it.
• sendmmsg goes non-blocking per entry and stops the batch on a would-block or partial-stream entry (kernel non-blocking-sendmmsg semantics), rather than deferring mid-batch.

Verified end-to-end: a single blocking send() of 2 MiB to a peer that stalls 2 s then drains returns the full 2 MiB in ~2 s (not truncated to one SO_SNDBUF), and a concurrent thread's sends stay at ~1 ms latency while that send is stalled — the loop is free. Integration test added for the full-delivery case.

One heads-up on tradeoffs I landed on: a deferred send is bounded by the supervisor's deferred-work timeout, so a child blocked on a genuinely-stuck peer past that bound gets EIO rather than blocking forever — bounding the DoS. And SCM_CREDENTIALS on the on-behalf path is rejected with EPERM (from the earlier round) rather than forwarded, since the supervisor is the sender and can't faithfully assert the child's credentials. Happy to adjust either if you'd prefer different semantics.

[congwang-mk]

This batch (and the named-unix one) breaks blocking-socket semantics in two ways: if entry 0 would block, a blocking child gets EAGAIN with sent==0, which the kernel never returns on a blocking socket; and a partially-sent stream entry is counted as sent with a short msg_len, i.e. silent truncation, since blocking callers do not check per-entry msg_len.

A would-block at entry i>0 with nothing sent needs no fix: returning the short count i is contract-legal and callers retry. Only two cases must leave the loop, and they are the same operation ("complete entry i off-loop, then report i+1"):

1. entry 0 would block entirely
2. entry i partially sent (stream)

Suggested shape: split defer_send into a byte-level core, e.g. async fn push_until_done(afd, m, offset) -> Result<usize, i32> (resolve_send wraps it as today). Then, for a blocking child hitting case 1 or 2, return NotifAction::defer of: push_until_done, write the full msg_len back, return i+1. On error: Errno if i==0 and offset==0, else ReturnValue(i). Everything captured (notif copy, notif_fd, the MaterializedMsg) is already static-friendly. All three batch loops can share that tail.

Please do not materialize the whole batch up front to make it replayable: vlen x MAX_SEND_BUF is the supervisor OOM the 64MB cap exists to prevent.