Skip to content

core: replace the ptrace argv freeze with an exec relay - #225

Closed
congwang-mk wants to merge 7 commits into
mainfrom
exec-relay
Closed

core: replace the ptrace argv freeze with an exec relay#225
congwang-mk wants to merge 7 commits into
mainfrom
exec-relay

Conversation

@congwang-mk

@congwang-mk congwang-mk commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

When a policy_fn or an execve-bound handler inspects argv, the supervisor reads it from the child, judges it, and continues the execve; the kernel then copies argv from the same memory, which sibling threads and CLONE_VM peers can rewrite in between. Until now that window was closed by freezing every sandbox task with ptrace, which in turn needed ptrace fork-event tracking so the freeze knew every task. This PR removes both and keeps ptrace only in checkpoint capture, which has no other way to read a task's registers.

How it works now. An approved execve is redirected to a small freestanding static program, the relay (src/exec_relay/relay.c, x86_64/aarch64/riscv64, embedded with include_bytes!). The relay's image and the judged argv/envp go into one sealed memfd. The supervisor installs it at a free fd K just below the child's soft RLIMIT_NOFILE with SECCOMP_ADDFD_FLAG_SETFD, sets the soft limit to K so no dup2, open, F_DUPFD, SCM_RIGHTS or pidfd_getfd in the sandbox can put another file there, verifies the entry, and rewrites the child's path to /dev/fd/K. The relay, single-threaded with a fresh address space, reads the trailer with pread(K) and execs the target with exactly the argv the policy saw: execveat(dirfd, base) against a directory whose identity it checks for ELF targets in plain mode, and by path for #! scripts (so $0 is right) and under COW and chroot, where the existing exec handlers resolve the target against the now single-threaded relay. The relay's own execve is recognised by the caller's exe inode matching the memfd, restores the soft limit, and passes through without a second policy event.

What the caller sees. Targets the kernel would refuse fail the caller's execve with the same errno first (ENOENT, ENOTDIR, EACCES for DAC or a directory, E2BIG), so execvp's PATH walk still works. argv[0] and comm are preserved. The program that finally runs sees its original soft NOFILE limit. The relay's own syscalls between its two execs are dispatched but never emitted as policy events, so observers such as sandlock learn keep seeing the real program rather than the memfd. Costs and edges: one extra execve per policy-checked spawn; exec failures the supervisor cannot predict (a Landlock exec denial, ENOEXEC) surface as the child exiting 127 with a one-line stderr message; execve-bound extra handlers also see the relay's re-exec, flagged by HandlerCtx::relay_exec; a clone sharing the fd table without CLONE_THREAD is refused with EINVAL under argv policy, since its own rlimit would not pin the fd.

Why not the obvious alternatives. /proc/<supervisor>/fd/N is unreachable from a Landlocked child because Landlock scopes ptrace-mode access and procfs uses it for foreign fd directories (verified). A plain ADDFD'd fd is swappable by a sibling in the microseconds before the kernel opens it; the rlimit pin closes that. The in-place path rewrite has to stay within 16 bytes because CPython's subprocess puts the exec path 16 bytes before the argv pointer array, which cannot move, hence /dev/fd/K rather than /proc/self/fd/K.

Tests. tests/integration/test_exec_relay.rs: a fixture whose sibling thread flips argv between an allowed and a denied word while the process execs, run 40 times, asserting the denied word never runs and whatever runs equals what the policy judged (this test fails on the first attempt if the relay is bypassed); $0 for scripts; argv[0] and comm; ENOENT and EACCES parity; exactly one policy event per exec; 80 subprocess spawns from 8 threads; the restored NOFILE limit; the CLONE_FILES refusal; a chroot rootfs exec; a script written into the COW branch and executed. The existing policy_fn, fork and resource suites pass unchanged. The rewrite-scan fix from #224 is included because the relay path exercises it.

Cross builds of the relay for aarch64 and riscv64 could not be run locally (no cross toolchains here); CI covers both.

🤖 Generated with Claude Code

Policy-checked execs need an argv the sandbox cannot change after the
supervisor has judged it. Freezing every task with ptrace closes that
window today, but it needs ptrace fork tracking to know every task,
and ptrace is the one mechanism this project wants to keep out of the
exec path. The relay is a freestanding static program with a fresh
private address space: the supervisor will exec it in place of the
approved program, and it execs the target with the argv carried in a
trailer on its own image. This commit adds the program, its wire
format with a Rust encoder and test decoder, and the build wiring;
nothing uses it yet.

Signed-off-by: Cong Wang <cwang@multikernel.io>
With a policy_fn or an execve extra handler active, the argv the
supervisor judged could be rewritten by a sibling thread or CLONE_VM
peer before the kernel copied it. Instead of freezing every sandbox
task with ptrace, an approved execve now runs the relay: its image and
the judged argv go into a sealed memfd installed at a free fd just
below the child's soft RLIMIT_NOFILE, the soft limit is pinned to that
number so nothing in the sandbox can put another file there, and the
child's path is rewritten to /dev/fd/K. The relay execs the target
with exactly the judged argv, pinning the parent directory for ELF
targets and going by path for scripts and under COW or chroot, where
the existing exec handlers resolve the target against the now
single-threaded relay. Targets the kernel would refuse fail the
caller's execve with the same errno first, so execvp's PATH walk still
works. The relay's own execve is recognised by the caller's exe inode
and passes through without a second policy event; a clone sharing the
fd table without CLONE_THREAD is refused because its own limit would
not pin the fd.

Signed-off-by: Cong Wang <cwang@multikernel.io>
The exec relay carries the judged argv itself, so nothing needs to stop
sandbox tasks around an execve, and nothing needs to know every child
at creation time. Remove freeze.rs, the one-shot ptrace fork-event
tracking in resource.rs, and the fork(2) interception that existed only
to feed it. Fork counting for the process limit stays. ptrace now
appears only in checkpoint capture, which has no other way to read a
task's registers.

Signed-off-by: Cong Wang <cwang@multikernel.io>
Under chroot and COW the relay execs by the child's own path and leaves
resolution to the existing exec handlers, which now run against the
single-threaded relay. Pin both with tests: a chroot rootfs whose
argv reaches the policy and whose program runs, and a script the
sandbox wrote into the COW branch and then executed.

Signed-off-by: Cong Wang <cwang@multikernel.io>
The freeze and fork-event tracking are gone; say what now keeps argv
TOCTOU-safe and which deferral rule remains.

Signed-off-by: Cong Wang <cwang@multikernel.io>
Between its two execs the relay opens the target's directory, and that
open reached policy_fn as an event from the application's pid. sandlock
learn reads /proc/<pid>/exe and maps on the first event after an exec
to pin the real binary, so it recorded the memfd instead and produced
profiles that could not run the program. A running relay is mechanism,
not application behaviour: its notifications are still dispatched but
never emitted as events.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
congwang-mk force-pushed the exec-relay branch 2 times, most recently from 4b1792c to f75b1a3 Compare September 13, 2026 02:28
The relay pins its memfd at a fd K just below the child's soft
RLIMIT_NOFILE and rewrites the exec to /dev/fd/K. A sibling thread or a
CLONE_VM peer could replace K with another file between the supervisor
verifying it and the kernel opening it, running an unapproved program
with an attacker-chosen argv. The earlier defense lowered the soft limit
to fence K, which a task could undo by raising the limit back, so it also
needed a prlimit/setrlimit trap and a racy clone gate for CLONE_FILES
peers that carry their own limit.

Of every fd-creating syscall, only dup2 and dup3 can force a file onto an
already-occupied descriptor; open, F_DUPFD, SCM_RIGHTS and pidfd_getfd
all take the lowest free number and cannot land on an occupied K. So trap
dup2 and dup3 under argv safety and, while an exec is in flight, refuse
one whose newfd is a pinned K. newfd is a register argument, not child
memory, so the check cannot be raced, and keying on the fd rather than a
per-process limit also covers a CLONE_FILES peer sharing the fd table.
The hold that arms the refusal is recorded before the install, so K is
never live but unguarded.

This drops the soft-limit lowering and restore, the prlimit64/setrlimit
trap, and the racy clone3 CLONE_FILES gate. The program that runs keeps
its original NOFILE limits untouched, and dup2/dup3 outside an exec pass
through.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk

Copy link
Copy Markdown
Contributor Author

This is broken, it closes the argv race but opens the path race.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant