core: replace the ptrace argv freeze with an exec relay - #225
Closed
congwang-mk wants to merge 7 commits into
Closed
core: replace the ptrace argv freeze with an exec relay#225congwang-mk wants to merge 7 commits into
congwang-mk wants to merge 7 commits into
Conversation
congwang-mk
force-pushed
the
exec-relay
branch
from
September 12, 2026 04:42
e2eb0dc to
64359a7
Compare
Policy-checked execs need an argv the sandbox cannot change after the supervisor has judged it. Freezing every task with ptrace closes that window today, but it needs ptrace fork tracking to know every task, and ptrace is the one mechanism this project wants to keep out of the exec path. The relay is a freestanding static program with a fresh private address space: the supervisor will exec it in place of the approved program, and it execs the target with the argv carried in a trailer on its own image. This commit adds the program, its wire format with a Rust encoder and test decoder, and the build wiring; nothing uses it yet. Signed-off-by: Cong Wang <cwang@multikernel.io>
With a policy_fn or an execve extra handler active, the argv the supervisor judged could be rewritten by a sibling thread or CLONE_VM peer before the kernel copied it. Instead of freezing every sandbox task with ptrace, an approved execve now runs the relay: its image and the judged argv go into a sealed memfd installed at a free fd just below the child's soft RLIMIT_NOFILE, the soft limit is pinned to that number so nothing in the sandbox can put another file there, and the child's path is rewritten to /dev/fd/K. The relay execs the target with exactly the judged argv, pinning the parent directory for ELF targets and going by path for scripts and under COW or chroot, where the existing exec handlers resolve the target against the now single-threaded relay. Targets the kernel would refuse fail the caller's execve with the same errno first, so execvp's PATH walk still works. The relay's own execve is recognised by the caller's exe inode and passes through without a second policy event; a clone sharing the fd table without CLONE_THREAD is refused because its own limit would not pin the fd. Signed-off-by: Cong Wang <cwang@multikernel.io>
The exec relay carries the judged argv itself, so nothing needs to stop sandbox tasks around an execve, and nothing needs to know every child at creation time. Remove freeze.rs, the one-shot ptrace fork-event tracking in resource.rs, and the fork(2) interception that existed only to feed it. Fork counting for the process limit stays. ptrace now appears only in checkpoint capture, which has no other way to read a task's registers. Signed-off-by: Cong Wang <cwang@multikernel.io>
Under chroot and COW the relay execs by the child's own path and leaves resolution to the existing exec handlers, which now run against the single-threaded relay. Pin both with tests: a chroot rootfs whose argv reaches the policy and whose program runs, and a script the sandbox wrote into the COW branch and then executed. Signed-off-by: Cong Wang <cwang@multikernel.io>
The freeze and fork-event tracking are gone; say what now keeps argv TOCTOU-safe and which deferral rule remains. Signed-off-by: Cong Wang <cwang@multikernel.io>
Between its two execs the relay opens the target's directory, and that open reached policy_fn as an event from the application's pid. sandlock learn reads /proc/<pid>/exe and maps on the first event after an exec to pin the real binary, so it recorded the memfd instead and produced profiles that could not run the program. A running relay is mechanism, not application behaviour: its notifications are still dispatched but never emitted as events. Signed-off-by: Cong Wang <cwang@multikernel.io>
congwang-mk
force-pushed
the
exec-relay
branch
2 times, most recently
from
September 13, 2026 02:28
4b1792c to
f75b1a3
Compare
The relay pins its memfd at a fd K just below the child's soft RLIMIT_NOFILE and rewrites the exec to /dev/fd/K. A sibling thread or a CLONE_VM peer could replace K with another file between the supervisor verifying it and the kernel opening it, running an unapproved program with an attacker-chosen argv. The earlier defense lowered the soft limit to fence K, which a task could undo by raising the limit back, so it also needed a prlimit/setrlimit trap and a racy clone gate for CLONE_FILES peers that carry their own limit. Of every fd-creating syscall, only dup2 and dup3 can force a file onto an already-occupied descriptor; open, F_DUPFD, SCM_RIGHTS and pidfd_getfd all take the lowest free number and cannot land on an occupied K. So trap dup2 and dup3 under argv safety and, while an exec is in flight, refuse one whose newfd is a pinned K. newfd is a register argument, not child memory, so the check cannot be raced, and keying on the fd rather than a per-process limit also covers a CLONE_FILES peer sharing the fd table. The hold that arms the refusal is recorded before the install, so K is never live but unguarded. This drops the soft-limit lowering and restore, the prlimit64/setrlimit trap, and the racy clone3 CLONE_FILES gate. The program that runs keeps its original NOFILE limits untouched, and dup2/dup3 outside an exec pass through. Signed-off-by: Cong Wang <cwang@multikernel.io>
congwang-mk
force-pushed
the
exec-relay
branch
from
September 13, 2026 02:46
f75b1a3 to
33e083a
Compare
congwang-mk
marked this pull request as draft
September 13, 2026 03:11
Contributor
Author
|
This is broken, it closes the argv race but opens the path race. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a
policy_fnor an execve-bound handler inspects argv, the supervisor reads it from the child, judges it, and continues the execve; the kernel then copies argv from the same memory, which sibling threads and CLONE_VM peers can rewrite in between. Until now that window was closed by freezing every sandbox task with ptrace, which in turn needed ptrace fork-event tracking so the freeze knew every task. This PR removes both and keeps ptrace only in checkpoint capture, which has no other way to read a task's registers.How it works now. An approved execve is redirected to a small freestanding static program, the relay (
src/exec_relay/relay.c, x86_64/aarch64/riscv64, embedded withinclude_bytes!). The relay's image and the judged argv/envp go into one sealed memfd. The supervisor installs it at a free fd K just below the child's soft RLIMIT_NOFILE withSECCOMP_ADDFD_FLAG_SETFD, sets the soft limit to K so nodup2,open,F_DUPFD, SCM_RIGHTS orpidfd_getfdin the sandbox can put another file there, verifies the entry, and rewrites the child's path to/dev/fd/K. The relay, single-threaded with a fresh address space, reads the trailer withpread(K)and execs the target with exactly the argv the policy saw:execveat(dirfd, base)against a directory whose identity it checks for ELF targets in plain mode, and by path for#!scripts (so$0is right) and under COW and chroot, where the existing exec handlers resolve the target against the now single-threaded relay. The relay's own execve is recognised by the caller's exe inode matching the memfd, restores the soft limit, and passes through without a second policy event.What the caller sees. Targets the kernel would refuse fail the caller's execve with the same errno first (ENOENT, ENOTDIR, EACCES for DAC or a directory, E2BIG), so execvp's PATH walk still works. argv[0] and comm are preserved. The program that finally runs sees its original soft NOFILE limit. The relay's own syscalls between its two execs are dispatched but never emitted as policy events, so observers such as
sandlock learnkeep seeing the real program rather than the memfd. Costs and edges: one extra execve per policy-checked spawn; exec failures the supervisor cannot predict (a Landlock exec denial, ENOEXEC) surface as the child exiting 127 with a one-line stderr message; execve-bound extra handlers also see the relay's re-exec, flagged byHandlerCtx::relay_exec; a clone sharing the fd table without CLONE_THREAD is refused with EINVAL under argv policy, since its own rlimit would not pin the fd.Why not the obvious alternatives.
/proc/<supervisor>/fd/Nis unreachable from a Landlocked child because Landlock scopes ptrace-mode access and procfs uses it for foreign fd directories (verified). A plain ADDFD'd fd is swappable by a sibling in the microseconds before the kernel opens it; the rlimit pin closes that. The in-place path rewrite has to stay within 16 bytes because CPython's subprocess puts the exec path 16 bytes before the argv pointer array, which cannot move, hence/dev/fd/Krather than/proc/self/fd/K.Tests.
tests/integration/test_exec_relay.rs: a fixture whose sibling thread flips argv between an allowed and a denied word while the process execs, run 40 times, asserting the denied word never runs and whatever runs equals what the policy judged (this test fails on the first attempt if the relay is bypassed);$0for scripts; argv[0] and comm; ENOENT and EACCES parity; exactly one policy event per exec; 80 subprocess spawns from 8 threads; the restored NOFILE limit; the CLONE_FILES refusal; a chroot rootfs exec; a script written into the COW branch and executed. The existing policy_fn, fork and resource suites pass unchanged. The rewrite-scan fix from #224 is included because the relay path exercises it.Cross builds of the relay for aarch64 and riscv64 could not be run locally (no cross toolchains here); CI covers both.
🤖 Generated with Claude Code