diff --git a/.claude/skills/elfuse-guest-abi/SKILL.md b/.claude/skills/elfuse-guest-abi/SKILL.md index b347a33f..94d743ef 100644 --- a/.claude/skills/elfuse-guest-abi/SKILL.md +++ b/.claude/skills/elfuse-guest-abi/SKILL.md @@ -63,21 +63,25 @@ the block size the shim assumes. |----|------|-------------| | 0 | TLBI_NONE | skip flush | | 1 | TLBI_BROADCAST | TLBI VMALLE1IS + DSB ISH + ISB | -| 2 | drop-frame | host rebuilt EL0 state; discard saved frame on ERET | +| 2 | drop-frame | host rebuilt EL0 state; discard saved frame on ERET, restoring no register except X8, reloaded from the frame's own X8 slot (`[sp, #64]`) | | 3 | TLBI_RANGE | loop TLBI VAE1IS, X9=VA, X10=page count | | 4 | TLBI_RANGE_LARGE | single RVAE1IS, encoded operand in X9 | X11=1 is the icache-flush hint: set when a page transitions to executable, and the shim then issues an IC invalidate alongside whichever TLBI it picked. The -shim restores X11 from the saved frame before ERET, so EL0 never observes it. +restoring tails reload X11 from the saved frame before ERET, so EL0 never +observes it. The X8 = 2 tail restores nothing but X8, so a delivery that follows +a page-table-modifying syscall in the same epilogue hands X9 through X11 out to +EL0 in place of the guest's. X7 is the ptrace-stop request on the same return, and it obeys a rule the TLBI codes do not. The shim reads it only after restoring the saved frame, so the tracer snapshots the guest's architectural registers rather than shim scratch; non-zero means take HVC #13 before the ERET. That makes X7 unusable on the one -tail that never restores the frame, X8 = 2, where the live registers already -are the final EL0 state and a host write to X7 would land in EL0 as guest -state. The host takes that stop inline in the epilogue instead and leaves X7 +tail that never restores the frame, X8 = 2, where the live registers are the +final EL0 state with one exception -- X8 holds the marker, which is why that +tail reloads it from the frame -- and a host write to X7 would land in EL0 as +guest state. The host takes that stop inline in the epilogue instead and leaves X7 alone, and an `execve` re-entry, which has no tail at all, leaves the stop owed for the new image. @@ -123,7 +127,13 @@ your path goes through the dispatch epilogue at all: - Inside the epilogue, set X8=2. The shim reads it as the drop-frame marker and discards the saved GPR frame. Signal delivery on the syscall-return path works this way. It does not need the marker when EL0 was preempted rather - than returning from a syscall, because there is no shim frame to drop. + than returning from a syscall, because there is no shim frame to drop. If + the X8 you want EL0 to see differs from the one the frame was entered with, + publish it into the frame's X8 slot with the marker: the marker occupies the + register, and the tail reloads X8 from that slot alone. Bound that write at + the EL1 stack region (`thread_sp_el1_region`), not at the shim data block: + the block's low end is the shim-globals cache, and only its top + `MAX_THREADS` slots are stack. - Bypass the epilogue by returning `SYSCALL_EXEC_HAPPENED`. The epilogue returns early, before it writes X0 or X8. `sys_execve` works this way, and the normal X0 writeback is exactly what it needs to avoid. It must also skip diff --git a/docs/internals.md b/docs/internals.md index 2ada3b61..204af26f 100644 --- a/docs/internals.md +++ b/docs/internals.md @@ -285,16 +285,18 @@ X8 == 1 TLBI_BROADCAST TLBI VMALLE1IS + DSB ISH + ISB -> restore GPRs (keep X0); ERET X8 == 2 drop-frame discard the saved GPR frame (`add sp, sp, #256`) and ERET on the rebuilt - EL0 register state. Set by `execve` and - `rt_sigreturn` (which write the whole frame - directly into the vCPU) and by - `signal_deliver()` on the syscall-return + EL0 register state, except for `X8` itself, + which the shim reloads from the frame's own + `X8` slot (`[sp, #64]`) because the marker + arrived in that register. Set by + `rt_sigreturn` (which writes the whole + register set directly into the vCPU, and + publishes its `X8` into that slot) and by + `deliver_signal_locked` on the syscall-return path (so handler PC/SP/LR/args installed by the host are not overwritten by the stale - shim frame on ERET). `execve` additionally - issues `IC IALLU` because the new program - text may live in pages that previously held - the old text. + shim frame on ERET). The flush is + unconditional, `IC IALLU` included. X8 == 3 TLBI_RANGE loop TLBI VAE1IS over `X9` (start VA), `X10` (page count); 4 KiB granule. Used for up to `TLBI_SELECTIVE_MAX_PAGES = 16` pages. @@ -317,25 +319,125 @@ separate broadcast after the split lands. `X8 == 2` is the generic drop-saved-frame marker: the host has rebuilt EL0 register state directly into the vCPU and the saved syscall frame on the EL1 stack is stale, so the shim drops the frame -and `ERET`s without restoring GPRs. Three call sites use it: - -- `sys_execve` (`src/syscall/exec.c:785, 1093`) after the ELF reload. -- `signal_rt_sigreturn` (`src/syscall/signal.c:1710`) after restoring - the saved sigframe. -- `signal_deliver` (`src/syscall/signal.c:1594`) when a signal is - delivered on the syscall-return path; without the marker the shim - would overwrite the handler PC, SP, LR, and arg-register state with - the stale syscall frame on `ERET`. - -`X8` (the syscall-number register) and `X9`/`X10` are already considered -clobbered by the Linux syscall ABI, so callers never expect them to be -preserved across SVC. - -Important: the first two paths (`sys_execve` and -`signal_rt_sigreturn`) return `SYSCALL_EXEC_HAPPENED` to bypass the -normal syscall dispatch epilogue. `signal_deliver` runs from inside -the epilogue. Any future code path that rebuilds EL0 register state -on the syscall-return path must write `X8 = 2` the same way. +and `ERET`s without restoring GPRs, except `X8`, which it reloads from +the frame's own `X8` slot because the marker arrived in that register. +Two call sites write it, both in `src/syscall/signal.c`: + +- `signal_rt_sigreturn`, after restoring the saved sigframe. +- `deliver_signal_locked`, when a signal is delivered on the + syscall-return path; without the marker the shim would overwrite the + handler PC, SP, LR, and arg-register state with the stale syscall + frame on `ERET`. + +`sys_execve` rebuilds EL0 state too and writes no marker: it re-enters +through the shim's MMU-off `_start`, which never pops a frame. + +Three shim tails branch to `exec_drop_frame`, not one. Besides the +`HVC #5` epilogue, both `HVC #9` W^X tails test `X8` against 2 and +branch there (`handle_inst_abort` and `handle_data_abort` in +`src/core/shim.S`), because the host answers a flip request with a +`SIGSEGV` delivery when the region never had the permission asked for. +Nothing publishes an `X8` on that route, so the reload hands EL0 the +value the exception was taken with, which is the guest's own. Measured +with a guest holding `0xa5` in `X8` across a branch to a page with no +`PROT_EXEC` and across a store to a page with no `PROT_WRITE`: the +handler enters with `X8 = 0x2` on the tree before the reload and with +`0xa5` after it, on both tails. `tests/test-shim-sigreturn-x8` covers +the pair. + +A fourth tail carries the same reload in its own body. `handle_brk` +forwards a `BRK` from EL0 to the host through `HVC #10`, and the host +delivers `SIGTRAP` there the way it delivers on any other path, +marker included. The tail used to pop its frame before the `HVC` and +`ERET` bare, so the marker was the `X8` the handler entered with: the +one marker consumer that never looked at it. It now takes the +`handle_el0_fault` shape, loading the GPRs without popping so the +frame outlives the `HVC`, then reloading `X8` from `[sp, #64]` and +popping after it. Nothing publishes on this route either, so what the +handler gets is the `X8` the `BRK` was taken with. JIT translators use +`BRK` as a patching trampoline and read that register, which is what +makes it worth the two instructions. Measured with a guest holding +`0xc3` in `X8` across a `BRK #0` with a `SIGTRAP` handler installed: +handler entry reads `X8 = 0x2` before the change and `0xc3` after, +while the value the guest resumes with once its `rt_sigreturn` has run +is `0xc3` on both, since that half goes through `HVC #5` like any +other. `tests/test-shim-sigreturn-x8` asserts both halves and +`scripts/check-svc-tails.py` holds both tails to the reload. + +Linux preserves `X1`-`X30` across `SVC #0` and modifies only `X0`, so +none of these registers may carry a host-to-shim value out to EL0. `X8` +is the one the marker occupies, which is why the drop tail reloads it +from the frame slot the host publishes into; a resumed `SVC` that has +not executed yet takes its syscall number from that register, so a +marker left there runs the call as syscall 2 +(sysprog21/elfuse#379). The ordinary syscall-return tail is a different +matter and this reload does not reach it. `X8` carries the TLBI kind +there and `X9`-`X11` its operands and the icache hint, so a signal +delivered in the epilogue of a page-table syscall records those wire +values as the guest's registers and its own `rt_sigreturn` hands them +back. The saved PC being past the `SVC` does not make that free: the +instruction after an `SVC` can be another `SVC`. Measured with two +adjacent `SVC`s under a 200 us itimer over 60000 rounds, the first of +them `mprotect`: all 707 deliveries left the second running as syscall +0 or 3 rather than the 226 the guest held, and the handler's +`ucontext` read `X8 = 0x3` at the same time. Pre-existing, and not the +register this reload moves: interleaved on the same machine, afdcfce +read 703 of its 704 as syscall 2, the marker rather than the wire +value. It is left for its own change, which has to substitute the +saved frame's `X8` on that path the way `deliver_signal_locked` takes +the parked value on this one. Since #379 closes here and this does +not, it is tracked as its own issue, #384. + +The frame slot covers the return itself. It does not cover a signal +delivered after the `rt_sigreturn` but before the vCPU is resumed: +that delivery snapshots the live registers, where `X8` is still the +marker, and the frame it builds would hand the marker back on its own +return. `signal_rt_sigreturn` therefore also parks the value in a +per-vCPU record that `deliver_signal_locked` reads, and the run loop +calls `signal_forget_sigreturn_x8()` before every `hv_vcpu_run()`, so +the record cannot be read by anything but a delivery in the epilogue +that wrote it. A record that outlived the resume would be handed to a +later delivery that merely lands on the same PC, a fault on an +instruction the guest returned to among them. + +One path can move the guest inside the epilogue, and the record +follows it there. `PTRACE_INTERRUPT` is consumed inline on this tail +rather than deferred to `HVC #13`, because the live registers there +are already the architectural EL0 set. A tracer that writes a new PC +and resumes with an injected signal makes the delivery's `ELR_EL1` +differ from the ELR the record was parked for, so the identity check +would miss the one delivery the record is genuinely for and the frame +would record the marker as the guest's `X8`. +`signal_repark_sigreturn_x8()` re-keys the record on the PC the stop +left behind, which is all that changed: the guest, its epilogue and +the `X8` owed to it are the same. Measured on a self-signaling tracee +kicked 4000 times, interleaved so the binaries met the same machine: +312 of 542, 494 of 986 and 431 of 776 stub landings came back with +`X8 = 2` on afdcfce, 1 of 251, 1 of 612 and 1 of 378 with the frame +slot and the park but no re-key, and 0 of 672, 0 of 134 and 0 of 291 +with it. + +Two halves of that stop are pre-existing and stay. `X8` holds the +marker while the tracer reads it, so `PTRACE_GETREGSET` reports 2 +where the guest's `X8` belongs, on both trees. And a tracer that +writes some other `X8` back does not get it honored: the shim +dispatches on that register after `HVC #5`, so a value that is neither +the marker nor a TLBI kind takes the conservative tail, which restores +the stale frame and then tests the `X7` this tail leaves holding guest +state, and the vCPU dies on the `HVC #13` that follows with no stop +armed. +Measured with the same tracee editing `X8` to `0x99` at every stop: +`FATAL ... HVC #13 with no ptrace stop armed` on this tree and on the +tree before the reload alike. Honoring such an edit means taking it +out of the live register as well as into the frame slot, which is a +change to the dispatch and not to this record. + +Important: `signal_rt_sigreturn` returns `SYSCALL_EXEC_HAPPENED` to +bypass the normal syscall dispatch epilogue, as `sys_execve` does. +`deliver_signal_locked` runs from inside the epilogue. Any future code +path that rebuilds EL0 register state on the syscall-return path must +write `X8 = 2` the same way, and publish the guest's `X8` with it if +that value differs from the one the frame was entered with. ## EL1 Shim And HVC Protocol @@ -348,10 +450,10 @@ aligned address from the `Rt` register); HVF traps DC ZVA via `HCR_EL2.TDZ=1`. | #0 | Normal exit | `X0` = exit code | | #2 | Bad exception | `X0`=ESR, `X1`=FAR, `X2`=ELR, `X3`=SPSR, `X5`=vector | | #4 | Set boot system register | `X0` = reg ID (0–8), `X1` = value (used by the shim during boot to install RES1 bits and enable the MMU) | -| #5 | Syscall forward | `X0`–`X5` = args, `X8` = syscall number on entry; on return `X8` carries the TLBI kind (`0` = none, `1` = broadcast, `3` = selective range with `X9` = VA + `X10` = page count, `4` = single-shot `TLBI RVAE1IS` with encoded operand in `X9`). `X8 = 2` is the generic drop-saved-frame marker -- set when the host has rebuilt EL0 state directly (by `execve`, `rt_sigreturn`, and `signal_deliver()` on the syscall-return path) so the shim discards the saved syscall frame on ERET. `X11` is the icache-flush hint (set to `1` when the request transitions a page to executable, so the shim issues `IC` alongside the chosen TLBI) | +| #5 | Syscall forward | `X0`–`X5` = args, `X8` = syscall number on entry; on return `X8` carries the TLBI kind (`0` = none, `1` = broadcast, `3` = selective range with `X9` = VA + `X10` = page count, `4` = single-shot `TLBI RVAE1IS` with encoded operand in `X9`). `X8 = 2` is the generic drop-saved-frame marker -- set when the host has rebuilt EL0 state directly, by two writers both in `src/syscall/signal.c` (`signal_rt_sigreturn`, and `deliver_signal_locked` on the syscall-return path; `sys_execve` rebuilds EL0 state too and writes no marker), so the shim discards the saved syscall frame on ERET and restores no register except `X8`, which it reloads from the frame's own `X8` slot (`[sp, #64]`) because the marker arrived in that register. `X11` is the icache-flush hint (set to `1` when the request transitions a page to executable, so the shim issues `IC` alongside the chosen TLBI) | | #6 | Embedder extension | `X8` = call number, `X0`–`X7` = args; routed to `g->hvc6_handler` if set, no-op otherwise. Handler may request a vCPU yield via `proc_request_hvc6_yield()` | | #7 | MRS trap (read sysreg) | host reads register from ESR ISS; returns value in `X0` | -| #9 | W^X toggle | `X0` = FAR, `X1` = type (0 = exec→RX, 1 = write→RW) | +| #9 | W^X toggle | in: `X0` = FAR, `X1` = type (0 = exec→RX, 1 = write→RW); out: `X8` = 2 when the host answered with a `SIGSEGV` delivery instead of a flip, which sends the shim to `exec_drop_frame`, and 0 on a completed flip. No kind is dispatched on this route: both tails test `cmp x8, #2` and otherwise branch to `tlbi_restore_eret`, which issues a single-page `TLBI VAE1IS` on `FAR_EL1` plus `IC IALLU` whatever `X8` held. The frame that delivery builds records `X0` and `X1` as the shim left them, so a handler reading `regs[0]`/`regs[1]` out of its `ucontext` sees `FAR_EL1` and the W^X type rather than the guest's values; pre-existing, and the same family as the `X8` marker above | | #10 | BRK from EL0 | SIGTRAP delivery / ptrace-stop; GPRs in frame | | #11 | EL0 fault | SIGSEGV/SIGILL delivery; GPRs in frame | | #12 | EL0 system-instruction trap | cache maintenance logging (DC CVAU, IC IVAU, …) and `MSR TPIDR_EL0` emulation | @@ -796,7 +898,8 @@ In `src/syscall/proc.c`: `hv_vcpus_exit()`. A stop taken on a syscall return whose tail restores the saved SVC frame goes through HVC #13, so ptrace snapshots the architectural GPR set rather than shim scratch. The tails that rebuild EL0 state instead - (`X8 = 2`) already hold that set live, so the host stops on them directly; + (`X8 = 2`) already hold that set live, bar the `X8` the shim reloads from + the saved frame, so the host stops on them directly; an `execve` re-entry leaves the stop owed for the new image's first syscall. - `PTRACE_GETREGSET` / `PTRACE_SETREGSET` (`NT_PRSTATUS`) -- read or write diff --git a/scripts/check-svc-tails.py b/scripts/check-svc-tails.py index ee4608fd..0618cf73 100644 --- a/scripts/check-svc-tails.py +++ b/scripts/check-svc-tails.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Hold every HVC #5 return tail to the X7 ptrace test. +"""Hold every HVC #5 return tail to the X7 ptrace test, and the X8 tails to X8. The host encodes a ptrace-stop request in X7 on the HVC #5 return and the shim reads it in svc_hvc_restore_eret. A tail that reaches EL0 without passing @@ -11,8 +11,9 @@ comment: exec_drop_frame never tested X7. That one is now deliberate (the host leaves - X7 alone on the X8 == 2 tail, whose live registers are already the final EL0 - state) and is the single allowed exception below. + X7 alone on the X8 == 2 tail, whose live registers are the final EL0 state + except for X8, which holds the marker and is reloaded from the frame by the + second rule below) and is the single allowed exception below. tlbi_selective's defensive zero-count exit was "cbz x10, 1f", and 1f resolved to a numeric label sitting inside svc_restore_eret, past the test. A numeric @@ -23,6 +24,40 @@ svc_restore_eret. Every tail the host can select lives there. Numeric labels inside that region are fine, and the selective-TLBI loop uses one; what is not fine is a reference that resolves past the end of it. + +The same tail carries a second rule, for the same reason: it restores nothing, +so whatever is in a register when it runs reaches EL0. X8 is where the marker +that selected the tail arrived, so the host publishes the guest's X8 in the +frame's own X8 slot and exec_drop_frame reloads it from there before the pop. +Deleting that one load is invisible in review and turns an SVC the guest has +not executed yet into syscall 2 (sysprog21/elfuse#379), so the offset is +checked here against the C side that writes it rather than left to match by +eye. + +handle_brk is held to the same rule and checked the same way. It is not an +HVC #5 tail, but it ends the same: the host delivers a signal out of HVC #10 +and leaves the marker in X8, and the tail reloads the guest's value from the +frame slot before dropping the frame. A SIGTRAP handler reading X8 is the +ordinary case there, since JIT translators use BRK as a trampoline. + +The host half of the same hand-off is checked too. A signal delivered after an +rt_sigreturn but before the vCPU is resumed cannot read the guest's X8 out of +the register either, so rt_sigreturn parks it and that delivery takes the parked +value. The park is correct only for that one epilogue, and what keeps it from +reaching an unrelated later delivery is that the run loop forgets it before +every resume. A resume added later somewhere else would not forget, and no test +driving the resume that exists today can see that, which is why the call sites +are enumerated here. + +What that check establishes is textual and narrower than the rule it stands for: +a call to signal_forget_sigreturn_x8() is written above every hv_vcpu_run(). +It does not establish that the call runs. Narrowing it in place, by wrapping the +existing call in a condition, leaves this gate at rc=0 with its summary line +unchanged while the record outlives the resume. Only +tests/test-shim-sigreturn-x8 answers that, and it does: its stale phase fails on +a narrowed forget, reporting the X8 the earlier rt_sigreturn returned with. +Read this gate as the answer to a resume added without a forget, and the test as +the answer to a forget that does not run. """ import argparse @@ -60,6 +95,28 @@ GATE = "svc_hvc_restore_eret" FORBIDDEN = "svc_restore_eret" +DROP_TAIL = "exec_drop_frame" +BRK_TAIL = "handle_brk" + +# Each X8-reloading tail's one restore, and the pop it has to precede. +DROP_RELOAD = re.compile(r"^\s*ldr\s+x8,\s*\[sp,\s*#(\d+)\]") +DROP_POP = re.compile(r"^\s*add\s+sp,\s*sp,\s*#(\d+)\b") +ERET = re.compile(r"^\s*eret\b") + +# The C side of the same frame slot. +SIGNAL_C = ROOT / "src" / "syscall" / "signal.c" +C_DEFINE = re.compile(r"^#define\s+(SHIM_FRAME_OFF_X8|SHIM_FRAME_BYTES)\s+(\d+)\s*$") + +# The host side of the hand-off: the resume, and the call that has to precede +# it. RESUME_LOOKBACK is generous on purpose -- a forget separated from its +# resume by a few lines of bookkeeping is the shape the run loop already has, +# and demanding the line above would reject it. What the window buys is +# tolerance, not proof: see the module docstring for what this does not answer. +SRC = ROOT / "src" +RESUME = re.compile(r"\bhv_vcpu_run\s*\(") +FORGET = re.compile(r"\bsignal_forget_sigreturn_x8\s*\(\s*\)") +COMMENT = re.compile(r"^\s*(?:/\*|\*|//)") +RESUME_LOOKBACK = 12 # The one tail allowed to skip the test, and why. Keep the reason with the name: # an exception added later without one is the bug this gate exists to stop. @@ -74,7 +131,121 @@ def branch_target(operands): return tail.split()[0] if tail else "" -def check(lines, path, counted=None): +def frame_constants(lines, path): + """The C-side SHIM_FRAME_* values, or a problem list.""" + found = {} + for line in lines: + if m := C_DEFINE.match(line): + found[m.group(1)] = int(m.group(2)) + missing = {"SHIM_FRAME_OFF_X8", "SHIM_FRAME_BYTES"} - set(found) + if missing: + return None, [ + f"{path}: no #define for {', '.join(sorted(missing))}; the host no " + f"longer names the frame slot the drop tail reloads X8 from" + ] + return found, [] + + +def check_reload_tail(lines, path, tail, gone, leaks, frame=None): + """A frame-dropping tail must put the frame's X8 back before it pops it. + + @tail is the label, @gone what a missing label means, and @leaks what EL0 + is handed when the load is not there. Both callers below restore no other + register, so the load is the whole of what stands between the host's X8 and + the guest's. + """ + at = next( + (i for i, l in enumerate(lines) if LABEL.match(l) and l.startswith(tail + ":")), + None, + ) + if at is None: + return [f"{path}: no '{tail}:' label; {gone}"] + + reload_at = reload_off = pop_at = pop_size = None + for i in range(at + 1, len(lines)): + if reload_at is None and (m := DROP_RELOAD.match(lines[i])): + reload_at, reload_off = i, int(m.group(1)) + if pop_at is None and (m := DROP_POP.match(lines[i])): + pop_at, pop_size = i, int(m.group(1)) + if ERET.match(lines[i]): + break + + problems = [] + if reload_at is None: + problems.append( + f"{path}:{at + 1}: '{tail}' does not reload X8 from the frame. " + f"It restores no register, so {leaks}" + ) + elif pop_at is not None and reload_at > pop_at: + problems.append( + f"{path}:{reload_at + 1}: the X8 reload is below the pop, so it " + f"reads past the frame it was meant to read." + ) + if pop_at is None: + problems.append( + f"{path}:{at + 1}: '{tail}' ERETs without popping the frame " + f"it was told to drop. SAVE_GPRS subtracted those bytes from " + f"SP_EL1 and nothing else gives them back, so the exception stack " + f"walks down one frame per drop until it leaves the shim block." + ) + if frame: + if reload_off is not None and reload_off != frame["SHIM_FRAME_OFF_X8"]: + problems.append( + f"{path}:{reload_at + 1}: reloads X8 from [sp, #{reload_off}], " + f"but the host publishes it at " + f"{frame['SHIM_FRAME_OFF_X8']} (SHIM_FRAME_OFF_X8)." + ) + if pop_size is not None and pop_size != frame["SHIM_FRAME_BYTES"]: + problems.append( + f"{path}:{pop_at + 1}: pops {pop_size} bytes, but the host " + f"bounds-checks a {frame['SHIM_FRAME_BYTES']}-byte frame " + f"(SHIM_FRAME_BYTES)." + ) + return problems + + +def check_resume_barrier(lines, path): + """Every vCPU resume must have a forget written above it. + + Textual, and deliberately reported as such: this finds a resume that no + forget precedes, which is the way the call goes missing when a resume is + added elsewhere. A forget that is present but does not run reads the same + here; tests/test-shim-sigreturn-x8 is what catches that. + """ + problems = [] + sites = 0 + for i, line in enumerate(lines): + if COMMENT.match(line) or not RESUME.search(line): + continue + sites += 1 + lo = max(0, i - RESUME_LOOKBACK) + covered = False + for j in range(i - 1, lo - 1, -1): + # Prose is not code on either side of this walk. A commented-out + # forget is exactly the deletion this rule exists to catch, and + # counting it as coverage would hide it. + if COMMENT.match(lines[j]): + continue + # A second resume between the two is the one that is uncovered: + # the forget above it belongs to the first. + if RESUME.search(lines[j]): + break + if FORGET.search(lines[j]): + covered = True + break + if covered: + continue + problems.append( + f"{path}:{i + 1}: resumes the vCPU without calling " + f"signal_forget_sigreturn_x8() first. The X8 an rt_sigreturn " + f"parked for a delivery in its own epilogue then outlives the " + f"guest running, and a later delivery that lands on the same PC " + f"is handed it in place of the live register." + ) + return problems, sites + + +def check(lines, path, counted=None, frame=None): """Return a list of problem strings. Empty means the tails are sound. @counted, when a list, receives the number of tails reaching the test, so @@ -92,12 +263,31 @@ def check(lines, path, counted=None): if end is None: return [f"{path}: no '{FORBIDDEN}:' after the HVC #5; the tail moved"] + problems_drop = check_reload_tail( + lines, + path, + DROP_TAIL, + "the X8 == 2 tail is gone", + "the drop-frame marker itself reaches EL0 in X8, and a frame whose " + "saved PC is on an SVC issues that SVC as syscall 2.", + frame, + ) + problems_drop += check_reload_tail( + lines, + path, + BRK_TAIL, + "the BRK tail is gone", + "the marker a SIGTRAP delivery leaves behind reaches the handler in " + "X8, in place of the X8 the guest hit the BRK with.", + frame, + ) + named = {m.group(1): i for i, l in enumerate(lines) if (m := LABEL.match(l))} numeric = [ (int(m.group(1)), i) for i, l in enumerate(lines) if (m := NUM_LABEL.match(l)) ] - problems = [] + problems = list(problems_drop) # The whole gate rests on this label testing X7. Renaming or emptying it # would leave every tail branching somewhere that no longer checks. @@ -234,8 +424,31 @@ def check(lines, path, counted=None): return problems -def _shim(tail): - """Wrap a dispatch tail in the minimum surrounding shape.""" +# A sound BRK tail, and a sound drop tail. Both are appended to every fixture +# below: each rule runs against the whole file, so a fixture aimed at one of +# them has to keep the other intact or every case inherits its failure. +GOOD_BRK = [ + "handle_brk:", + " LOAD_GPRS", + " hvc #10", + " ldr x8, [sp, #64]", + " add sp, sp, #256", + " eret", +] +GOOD_DROP = [ + "exec_drop_frame:", + " ldr x8, [sp, #64]", + " add sp, sp, #256", + " eret", +] + + +def _shim(tail, brk=None): + """Wrap a dispatch tail in the minimum surrounding shape. + + The drop tail goes last so a case aiming at it can slice its body off with + [:-3] and write its own. + """ return ( ["handle_svc_0:", " hvc #5", " cbz x8, svc_hvc_restore_eret"] + tail @@ -246,9 +459,9 @@ def _shim(tail): "svc_hvc_restore_eret:", " cbz x7, svc_restore_eret", " b svc_restore_eret", - "exec_drop_frame:", - " eret", ] + + (GOOD_BRK if brk is None else brk) + + GOOD_DROP ) @@ -272,7 +485,9 @@ def _shim(tail): "svc_hvc_restore_eret:", " cbz x7, svc_restore_eret", " eret", - ], + ] + + GOOD_BRK + + GOOD_DROP, 2, ), ( @@ -313,7 +528,9 @@ def _shim(tail): "svc_hvc_restore_eret:", " cbz x7, svc_restore_eret", " eret", - ], + ] + + GOOD_BRK + + GOOD_DROP, 1, ), ( @@ -323,7 +540,8 @@ def _shim(tail): ), ( "branch out to an unrelated handler", - _shim(["tlbi_full:", " b handle_brk"]) + ["handle_brk:", " eret"], + _shim(["tlbi_full:", " b handle_unrelated"]) + + ["handle_unrelated:", " eret"], 1, ), ("no dispatch at all", ["_start:", " ret"], 1), @@ -360,8 +578,134 @@ def _shim(tail): " eret", "svc_hvc_restore_eret:", " b svc_restore_eret", + ] + + GOOD_BRK + + GOOD_DROP, + 1, + ), + ( + "drop tail no longer reloads X8", + _shim(["tlbi_full:", " b svc_hvc_restore_eret"])[:-3] + + [" add sp, sp, #256", " eret"], + 1, + ), + ( + "drop tail reloads X8 after the pop", + _shim(["tlbi_full:", " b svc_hvc_restore_eret"])[:-3] + + [" add sp, sp, #256", " ldr x8, [sp, #64]", " eret"], + 1, + ), + ( + "drop tail reloads X8 from the wrong slot", + _shim(["tlbi_full:", " b svc_hvc_restore_eret"])[:-3] + + [" ldr x8, [sp, #72]", " add sp, sp, #256", " eret"], + 1, + ), + ( + "drop tail no longer pops the frame", + _shim(["tlbi_full:", " b svc_hvc_restore_eret"])[:-3] + + [" ldr x8, [sp, #64]", " eret"], + 1, + ), + ( + "BRK tail no longer reloads X8", + _shim( + ["tlbi_full:", " b svc_hvc_restore_eret"], + brk=["handle_brk:", " LOAD_GPRS", " hvc #10", + " add sp, sp, #256", " eret"], + ), + 1, + ), + ( + "BRK tail reloads X8 after the pop", + _shim( + ["tlbi_full:", " b svc_hvc_restore_eret"], + brk=["handle_brk:", " LOAD_GPRS", " hvc #10", + " add sp, sp, #256", " ldr x8, [sp, #64]", " eret"], + ), + 1, + ), + ( + "BRK tail reloads X8 from the wrong slot", + _shim( + ["tlbi_full:", " b svc_hvc_restore_eret"], + brk=["handle_brk:", " LOAD_GPRS", " hvc #10", + " ldr x8, [sp, #72]", " add sp, sp, #256", " eret"], + ), + 1, + ), + ( + "BRK tail popped its frame before the HVC again", + _shim( + ["tlbi_full:", " b svc_hvc_restore_eret"], + brk=["handle_brk:", " RESTORE_GPRS", " hvc #10", " eret"], + ), + 2, + ), + ( + "BRK tail gone", + _shim(["tlbi_full:", " b svc_hvc_restore_eret"], brk=[]), + 1, + ), +] + + +SELF_TEST_FRAME = {"SHIM_FRAME_OFF_X8": 64, "SHIM_FRAME_BYTES": 256} + +RESUME_CASES = [ + ( + "resume behind the forget", + [ + " signal_forget_sigreturn_x8();", + "", + " HV_CHECK_CTX(hv_vcpu_run(vcpu), vcpu, g);", + ], + 1, + 0, + ), + ( + "forget deleted", + [" HV_CHECK_CTX(hv_vcpu_run(vcpu), vcpu, g);"], + 1, + 1, + ), + ( + "forget moved below the resume", + [ + " HV_CHECK_CTX(hv_vcpu_run(vcpu), vcpu, g);", + " signal_forget_sigreturn_x8();", + ], + 1, + 1, + ), + ( + "a second resume added without one", + [ + " signal_forget_sigreturn_x8();", + " HV_CHECK_CTX(hv_vcpu_run(vcpu), vcpu, g);", + " if (retry)", + " hv_vcpu_run(vcpu);", + ], + 2, + 1, + ), + ( + "prose naming the resume is not a call site", + [ + " /* kick the vCPUs out of hv_vcpu_run() before the unmap */", + " * ordered by the first hv_vcpu_run and the release here", + ], + 0, + 0, + ), + ( + "forget commented out above the resume", + [ + " // signal_forget_sigreturn_x8();", + " HV_CHECK_CTX(hv_vcpu_run(vcpu), vcpu, g);", ], 1, + 1, ), ] @@ -370,14 +714,23 @@ def self_test(): print(" SVCTAIL self-test", flush=True) failures = 0 for name, lines, expected in CASES: - got = len(check(lines, "")) + got = len(check(lines, "", frame=SELF_TEST_FRAME)) if got != expected: print(f" FAIL {name}: expected {expected} problem(s), got {got}") failures += 1 + for name, lines, want_sites, expected in RESUME_CASES: + got, sites = check_resume_barrier(lines, "") + if len(got) != expected or sites != want_sites: + print( + f" FAIL {name}: expected {expected} problem(s) over " + f"{want_sites} site(s), got {len(got)} over {sites}" + ) + failures += 1 + total = len(CASES) + len(RESUME_CASES) if failures: - print(f" self-test: {failures} of {len(CASES)} cases failed") + print(f" self-test: {failures} of {total} cases failed") return 1 - print(f" self-test: {len(CASES)} cases, all pass") + print(f" self-test: {total} cases, all pass") return 0 @@ -390,21 +743,42 @@ def main(): return self_test() counted = [] - problems = check(SHIM.read_text().splitlines(), str(SHIM), counted) + frame, problems = frame_constants( + SIGNAL_C.read_text().splitlines(), str(SIGNAL_C) + ) + problems += check(SHIM.read_text().splitlines(), str(SHIM), counted, frame) + + resumes = 0 + for src in sorted(SRC.rglob("*.c")): + found, sites = check_resume_barrier( + src.read_text().splitlines(), str(src.relative_to(ROOT)) + ) + problems += found + resumes += sites + if not resumes: + problems.append( + "src: no hv_vcpu_run() call site found; the vCPU resume moved, and " + "with it the point the parked X8 has to be forgotten at" + ) + print(f" SVCTAIL {SHIM.relative_to(ROOT)}", flush=True) if problems: for p in problems: print(f" {p}", file=sys.stderr) print( - f"\n {len(problems)} HVC #5 tail(s) can reach EL0 without the X7 " - f"ptrace test.", + f"\n {len(problems)} problem(s): a tail that can reach EL0 without " + f"the X7 ptrace test, a frame-dropping tail that hands EL0 the " + f"marker instead of the guest's X8 or leaves the frame it was told " + f"to drop, or a resume with no forget written above it.", file=sys.stderr, ) return 1 print( f" {counted[0]} HVC #5 tail(s) reach the X7 test, " - f"{len(ALLOWED_SKIP)} documented exception" + f"{len(ALLOWED_SKIP)} documented exception; {DROP_TAIL} and {BRK_TAIL} " + f"both reload X8 from [sp, #{frame['SHIM_FRAME_OFF_X8']}] before the " + f"pop; {resumes} vCPU resume(s) have a forget written above them" ) return 0 diff --git a/src/core/guest.h b/src/core/guest.h index f265fabd..a4e14a97 100644 --- a/src/core/guest.h +++ b/src/core/guest.h @@ -301,8 +301,8 @@ typedef struct { * TLBI_BROADCAST -> X8 = 1 (TLBI VMALLE1IS, broadest) * TLBI_RANGE -> X8 = 3, X9 = start VA, X10 = page count * (TLBI VAE1IS loop preserves unrelated TLB entries) - * X8 = 2 is reserved for the execve drop-frame marker the shim handles - * separately; it is never produced by the accumulator. + * X8 = 2 is reserved for the drop-frame marker the shim handles separately; it + * is never produced by the accumulator. */ typedef enum { TLBI_NONE = 0, diff --git a/src/core/shim.S b/src/core/shim.S index b473a2da..a7a1b525 100644 --- a/src/core/shim.S +++ b/src/core/shim.S @@ -35,9 +35,13 @@ * X8 on return from #5, the post-syscall request: * 0 no flush * 1 broadcast TLBI VMALLE1IS - * 2 host replaced the EL0 register state (execve, rt_sigreturn, signal - * delivery on the syscall-return path): full flush, drop the saved frame, - * ERET without restoring GPRs + * 2 host replaced the EL0 register state (rt_sigreturn, signal delivery on + * the syscall-return path): full flush, drop the saved frame, ERET + * without restoring GPRs, with one exception. X8 is the register the + * marker itself arrives in, so it cannot also carry the guest value the + * host wants EL0 to see. The host leaves that value in the frame's own X8 + * slot and exec_drop_frame reloads X8 from there; see the note on that + * label. * 3 TLBI VAE1IS over X10 pages from the page-aligned VA in X9 * 4 single-shot TLBI RVAE1IS (FEAT_TLBIRANGE), operand pre-encoded in X9 as * baddr | NUM<<39 | SCALE<<44 | TTL<<37 | ASID<<48 (SCALE, TTL and ASID @@ -48,15 +52,20 @@ * take HVC #13 after the restore, so the tracer sees the architectural GPR set * rather than shim scratch. The host leaves X7 alone on the X8 == 2 tail, which * never restores the frame and would carry the flag into EL0 as guest state; it - * takes that stop itself, where the live registers are already final. A tail + * takes that stop itself, where the live registers are already final bar X8, + * which holds the marker and is reloaded from the frame by the tail. A tail * that skips the X7 test drops a stop the host has already consumed, so new * exits from the dispatch below go through svc_hvc_restore_eret, never straight * to svc_restore_eret. * * X11 is the I-cache hint, read for X8 in {1, 3, 4}: 1 issues IC IALLU after * the TLBI because the change made new content executable, 0 skips it. X8 == 2 - * always flushes, since execve loads new code. The shim restores X11 from the - * saved frame before ERET, so EL0 never sees the hint. + * flushes unconditionally, IC IALLU included, because the host has rebuilt the + * EL0 state and the shim cannot tell what changed. The restoring tails reload + * X11 from the saved frame before ERET, so EL0 never sees the hint. The X8 == 2 + * tail restores nothing but X8, so a delivery that follows a page-table + * modifying syscall in the same epilogue hands X9 through X11 out to EL0 in + * place of the guest's; see the note on exec_drop_frame. * * Two constraints on everything below. macOS 'as' treats ';' on AArch64 as a * comment, not a statement separator, so every instruction needs its own line. @@ -1669,7 +1678,7 @@ restore_and_bad: /* handle_brk: BRK instruction from EL0 (EC=0x3C) * * JIT translators use BRK instructions as patching trampolines and debug hooks. - * The guest registers a SIGTRAP handler via rt_sigaction. The shim restores all + * The guest registers a SIGTRAP handler via rt_sigaction. The shim loads all * GPRs and forwards to the host via HVC #10 for signal delivery. * * Host reads ESR_EL1 (BRK immediate), ELR_EL1 (BRK PC), and current GPRs from @@ -1678,15 +1687,27 @@ restore_and_bad: * SIGTRAP action (terminate). */ handle_brk: - /* Restore all GPRs from stack frame (host needs current state) */ - RESTORE_GPRS + /* Load the EL0 GPRs from the saved frame WITHOUT popping it, the shape + * handle_el0_fault uses and for the same reason: the host needs the guest's + * register state, and the frame has to outlive the HVC so the tail below + * can read out of it. + */ + LOAD_GPRS /* Forward to host for SIGTRAP delivery */ hvc #10 /* Host has set up signal frame (if handler registered) or flagged for - * termination. Either way, ERET to new PC. - */ + * termination. A delivery leaves the drop-frame marker in X8, which is + * guest state on this path: the SIGTRAP handler would enter with 2 in place + * of whatever the guest was carrying, and a JIT translator using BRK as a + * trampoline is the caller most likely to read that register. Reload it + * from the frame's own X8 slot before the pop, the way exec_drop_frame + * does; nothing publishes there on this path, so the value is the one the + * BRK was taken with. Then drop the frame and ERET to the new PC. + */ + ldr x8, [sp, #64] + add sp, sp, #256 eret /* handle_svc_0: Linux syscall forwarding @@ -1710,7 +1731,9 @@ handle_svc_0: * 4. Set X8 to indicate the post-syscall request: * 0 = no TLB flush * 1 = broadcast TLBI VMALLE1IS - * 2 = execve replaced register state (drop frame + flush) + * 2 = host replaced register state (rt_sigreturn, signal + * delivery): drop frame + flush, and reload X8 from the + * frame's own slot; see exec_drop_frame * 3 = selective TLBI VAE1IS over X10 pages starting at X9 * 5. Resume vCPU (execution continues below) */ @@ -1844,6 +1867,48 @@ svc_hvc_restore_eret: eret exec_drop_frame: + /* This tail restores nothing from the frame because the host has written + * the EL0 state it wants straight into the vCPU. X8 is the one register + * that cannot arrive that way: it carries the drop-frame marker in, so the + * value the host wants EL0 to see has to travel out of band. + * + * It travels in the frame's own X8 slot, [sp+64] per SAVE_GPRS, filled by + * the host before it writes the marker. That slot cannot collide with the + * marker the way a second register would: it is EL1-only memory on this + * vCPU's exception stack, unreachable from EL0 and unshared with any other + * vCPU, and the pop below retires it. Putting it back here is what keeps a + * frame whose saved PC sits on an SVC from issuing that SVC as syscall 2. + * + * Three tails branch here and only the HVC #5 one has a publisher. The two + * W^X permission faults reach it when the host answers the flip request + * with the marker, which it does when the region never had the permission + * and the fault is a real SIGSEGV. Nothing publishes on that route, so the + * reload hands EL0 the X8 the exception was taken with, which is the + * guest's own and is what the handler then enters with. Before this reload + * existed those two entered their handler with the marker. handle_brk has + * the same problem and does not branch here, since it has its own flush to + * skip and no X8 dispatch to run; it carries its own copy of the reload. + * + * The ordinary syscall-return tail above is not covered, and this reload + * does not reach it. A signal delivered in the epilogue of a page-table + * syscall snapshots the live registers there, where X8 is the TLBI kind and + * X9 through X11 are its operands and the icache hint, so the frame that + * delivery builds records the wire values as the guest's and its own + * rt_sigreturn hands them back. The saved PC being past the SVC does not + * make that free: the instruction after an SVC can be another SVC, and the + * guest then issues the wire value as a syscall number. Measured with two + * adjacent SVCs under a 200us itimer over 60000 rounds, the first of them + * mprotect: all 707 deliveries left the second running as syscall 0 or 3 + * rather than the 226 the guest held. It is pre-existing and not the + * register this tail moves. The same probe on afdcfce, interleaved so both + * binaries met the same machine, read 703 of its 704 as syscall 2, the + * marker rather than the wire value. syscall/proc.h names the same + * shadowing on the frame a handler returns through. + * + * It survives the change that closes #379, so it is tracked as its own + * issue rather than living only here: #384. + */ + ldr x8, [sp, #64] tlbi vmalle1is dsb ish isb diff --git a/src/runtime/thread.c b/src/runtime/thread.c index 17ebf792..6b3ea185 100644 --- a/src/runtime/thread.c +++ b/src/runtime/thread.c @@ -518,10 +518,10 @@ uint64_t thread_alloc_sp_el1(const guest_t *g, thread_entry_t *t) int slot = bit_ctz64(free_mask); /* Main thread's SP_EL1 sits at the top of the shim data block. Each - * subsequent thread is 4KiB below. + * subsequent thread is one slot below. */ uint64_t top = sp_el1_top(g); - sp = top - (uint64_t) slot * 4096; + sp = top - (uint64_t) slot * SP_EL1_SLOT_BYTES; sp_el1_allocated |= BIT64(slot); t->sp_el1 = sp; t->sp_el1_slot = slot; @@ -532,6 +532,19 @@ uint64_t thread_alloc_sp_el1(const guest_t *g, thread_entry_t *t) return sp; } +void thread_sp_el1_region(const guest_t *g, uint64_t *lo, uint64_t *hi) +{ + /* Derived from what thread_alloc_sp_el1 above hands out, not from the block + * it hands it out of: slot 0 starts at the top and slot MAX_THREADS - 1 is + * the last one, so the region is the top MAX_THREADS slots and everything + * under it belongs to the shim-globals cache. + */ + uint64_t top = sp_el1_top(g); + + *hi = top; + *lo = top - (uint64_t) MAX_THREADS * SP_EL1_SLOT_BYTES; +} + void thread_for_each(void (*fn)(thread_entry_t *t, void *ctx), void *ctx) { pthread_mutex_lock(&thread_lock); diff --git a/src/runtime/thread.h b/src/runtime/thread.h index 2423ca76..92f153a8 100644 --- a/src/runtime/thread.h +++ b/src/runtime/thread.h @@ -29,6 +29,13 @@ /* Maximum number of concurrent guest threads in one VM. */ #define MAX_THREADS 64 + +/* One EL1 exception stack per thread. Named because the region these slots + * occupy is a bound as well as an allocation: the host writes into a live shim + * frame on that stack, and anything below the lowest slot is the shim-globals + * cache rather than stack. + */ +#define SP_EL1_SLOT_BYTES 4096 #define MAX_DEFERRED_STACK_UNMAPS 8 /* Per-thread state. One entry per guest thread (main + workers). Tagged (struct @@ -353,15 +360,28 @@ uint64_t thread_pending_union(void); int thread_is_single_active(void); /* Allocate a per-thread SP_EL1 stack and record both the IPA and the slot index - * into t. Thread N gets the Nth 4KiB slot counting down from the top of the - * shim data block (g->shim_data_base + 2MiB). The shim block lives at high IPA - * computed by guest_init, so callers must pass g; the slot index is stored in - * t->sp_el1_slot so the free path (which is reached from teardown contexts that - * lack g) can clear the bitmask directly. + * into t. Thread N gets the Nth SP_EL1_SLOT_BYTES slot counting down from the + * top of the shim data block (g->shim_data_base + 2MiB). The shim block lives + * at high IPA computed by guest_init, so callers must pass g; the slot index is + * stored in t->sp_el1_slot so the free path (which is reached from teardown + * contexts that lack g) can clear the bitmask directly. * Returns the SP_EL1 IPA, or 0 on slot exhaustion. */ uint64_t thread_alloc_sp_el1(const guest_t *g, thread_entry_t *t); +/* The IPA range the slots above are carved from: *lo is the bottom of the + * lowest slot, *hi one past the top of the highest, so a live SP_EL1 is in + * [*lo, *hi). + * + * The shim data block holds two unrelated things. These slots sit at the top of + * it, and the shim-globals cache -- identity slots, urandom ring, attention + * bitmask -- starts at the bottom. A host that validates an SP_EL1 against the + * whole block therefore accepts addresses that name cache and not stack, and + * writing a shim frame at one of those corrupts the cache instead of reporting + * the bad SP_EL1. Callers that write through an SP_EL1 bound it with this. + */ +void thread_sp_el1_region(const guest_t *g, uint64_t *lo, uint64_t *hi); + /* Iterate over all active threads, calling fn(entry, ctx) for each. Holds the * thread table lock during iteration. */ diff --git a/src/syscall/proc.c b/src/syscall/proc.c index fe8d1946..0014cea1 100644 --- a/src/syscall/proc.c +++ b/src/syscall/proc.c @@ -1915,9 +1915,10 @@ int64_t sys_ptrace(guest_t *g, * whatever the tracer writes back, since the shim restores its own * frame over it. The HVC #5 epilogue consumes the flag and then either * stops right there, on the tails whose live registers are already the - * final EL0 set, or asks the shim through X7 to restore the frame and - * come back at HVC #13. The canceled-exit handler consumes it once it - * has established the vCPU is at EL0. + * final EL0 set bar the X8 the tail reloads from the frame, or asks the + * shim through X7 to restore the frame and come back at HVC #13. The + * canceled-exit handler consumes it once it has established the vCPU is + * at EL0. * * Attention goes up before the kick, the same order * shim_globals_raise_attention uses and for the same reason: a fast @@ -4029,6 +4030,13 @@ static bool ptrace_take_stop(guest_t *g, hv_vcpu_t vcpu, int *exit_code) if (cont_sig > 0) signal_queue(cont_sig); + /* The stop above returns with whatever registers the tracer wrote, PC among + * them. An X8 parked by an rt_sigreturn earlier in this same epilogue is + * still the X8 the guest is owed at the PC it now resumes from, so the + * record follows it before the delivery below reads it. + */ + signal_repark_sigreturn_x8(vcpu); + /* One delivery covers both the injected resume signal and anything that * arrived while the tracee was stopped, so neither caller repeats it. */ @@ -4074,7 +4082,8 @@ static bool syscall_return_epilogue(guest_t *g, * host-only. The vector entry clobbers no GPR below X9, so the live set at * HVC #5 is still the guest's, and only that restore puts a host write to * X7 back. Two tails skip it: X8 == 2, where the host has rebuilt EL0 state - * and the live registers are already final, and an execve re-entry, which + * and the live registers are already final bar X8, which carries the marker + * and is reloaded from the frame by the tail, and an execve re-entry, which * goes through the MMU-off _start with no tail at all. * * Only the exec-happened return has to ask the vCPU which of those it is. @@ -4099,7 +4108,11 @@ static bool syscall_return_epilogue(guest_t *g, ptrace_consume_owed_stop(g)) { if (regs_final) { /* Live registers are already the architectural EL0 set, which is - * what the detour exists to produce. Stop here instead. + * what the detour exists to produce. Stop here instead. One + * register is not the guest's: X8 still holds the drop-frame + * marker, which the tail reloads from the frame after this stop, so + * PTRACE_GETREGSET here reports 2 where the guest's X8 belongs. + * Pre-existing and unchanged; docs/internals.md measures it. */ running = ptrace_take_stop(g, vcpu, exit_code); stop_taken = true; @@ -4675,8 +4688,8 @@ int vcpu_run_loop_with_hooks(hv_vcpu_t vcpu, * so it is the only thread that can survive one. Running it at the top * of the loop puts the rebuilt EL0 state in place before the vCPU is * resumed, whether this thread was preempted in guest code or is - * returning from its own syscall (sys_execve sets the X8=2 frame-drop - * marker either way). + * returning from its own syscall (sys_execve writes no frame-drop + * marker either way: its MMU-off _start re-entry never pops a frame). */ if (thread_current_is_leader() && thread_leader_work_pending()) exec_run_handoff(vcpu, g, verbose); @@ -4708,6 +4721,12 @@ int vcpu_run_loop_with_hooks(hv_vcpu_t vcpu, atomic_store_explicit(&g_vcpu_progress, iter * 2 + 1, memory_order_relaxed); + /* The guest owns X8 again from here, so the X8 an rt_sigreturn parked + * for a delivery in its own epilogue must not survive into the next + * exception. See signal_forget_sigreturn_x8 in syscall/signal.h. + */ + signal_forget_sigreturn_x8(); + HV_CHECK_CTX(hv_vcpu_run(vcpu), vcpu, g); if (is_main) diff --git a/src/syscall/proc.h b/src/syscall/proc.h index f71663ad..e1ae0c6c 100644 --- a/src/syscall/proc.h +++ b/src/syscall/proc.h @@ -368,8 +368,13 @@ const char *proc_resolve_sysroot_create_path(const char *path, * a signal really did arrive. And the frame the handler returns through carries * the live X8, which by then is the TLBI wire value the shim epilogue wrote * rather than the syscall number the shim would have restored from its own - * saved frame. That is harmless while the saved PC is past the SVC, but a - * rewound PC would make rt_sigreturn re-execute the SVC as the wrong call. + * saved frame. A rewound PC makes rt_sigreturn re-execute that SVC as the call + * the wire value names. Leaving the PC past the SVC narrows that rather than + * closing it, because the instruction after an SVC can be another SVC; that + * residue is pre-existing, measured beside exec_drop_frame in core/shim.S, and + * out of reach of this cancel. The drop-frame marker is the other value that + * displaces the guest's X8, and signal.c keeps that one out of both the frame + * and the ERET on its own. * * Cancel is a no-op unless ELR_EL1 still holds the value the arm wrote, so a * later delivery on an unrelated path cannot disturb a guest that has moved on. diff --git a/src/syscall/signal.c b/src/syscall/signal.c index cd5f2df9..45693fee 100644 --- a/src/syscall/signal.c +++ b/src/syscall/signal.c @@ -32,7 +32,10 @@ * * rt_sigreturn always has a frame to drop, having entered through one. * deliver_signal_locked and signal_rt_sigreturn each say why at the point they - * write the marker. sys_execve is a fourth rebuilder, in exec.c, and takes + * write the marker. The marker occupies X8, and the tail that consumes it + * restores no register, so the guest's own X8 has to reach EL0 another way: + * shim_publish_frame_x8 puts it in the shim frame's X8 slot, and the shim + * reloads it from there. sys_execve is a fourth rebuilder, in exec.c, and takes * neither shape: it re-enters through the shim's MMU-off _start with the GPRs * zeroed, so there is no frame to drop and no marker to write. * @@ -2203,6 +2206,121 @@ static void build_sigcontext_reserved(uint8_t *reserved, memset(reserved + off, 0, 8); } +/* The EL1 shim's exception frame: 256 bytes, saved X8 at byte 64. SAVE_GPRS in + * src/core/shim.S lays it out and exec_drop_frame reloads X8 from that slot; + * scripts/check-svc-tails.py holds the two sides to the same number. + */ +#define SHIM_FRAME_BYTES 256 +#define SHIM_FRAME_OFF_X8 64 + +/* The guest's own X8 while the host borrows the register, handed from the + * rt_sigreturn that borrowed it to a signal delivered later in the same host + * epilogue. + * + * The shim is told to drop its saved exception frame by a marker the host puts + * in X8, and the tail that consumes the marker restores no register, so X8 is + * the one register that cannot hold guest state across that return. Publishing + * the value into the frame slot covers the return itself. It does not cover a + * signal delivered before the vCPU is resumed: that delivery snapshots the live + * registers, and the live X8 is still the marker, so the frame it builds would + * record the marker as the guest's X8 and its rt_sigreturn would hand that + * back. + * + * What makes substituting this safe is that it cannot outlive that window. + * signal_forget_sigreturn_x8 drops the record at every vCPU resume, so the only + * reader that can ever see one is a delivery between the rt_sigreturn that + * wrote it and the guest running again, which is exactly the stretch where the + * live register is not the guest's. Consumption drops it too, so a second + * delivery in the same epilogue reads the register. + * + * The ELR it was parked for is kept and compared, the way syscall_restart_arm + * recognizes its own rewind, but it is an identity check and not the bound: one + * thing moves the guest inside an epilogue and it re-keys the record as it + * does, so a delivery that still finds a different ELR is on a path this did + * not anticipate and reads the register instead. + * + * One path moves the guest inside the epilogue: a ptrace stop consumed on this + * same tail, where the tracer writes a new PC before the epilogue delivers a + * signal. ELR_EL1 is then the tracer's PC, and the comparison would miss the + * one delivery this record is genuinely for, leaving it to read the marker out + * of the register. signal_repark_sigreturn_x8 re-keys the record on the PC the + * stop left behind instead of widening the comparison away, because only the + * address moved: the guest, its epilogue and the X8 owed to it are the same, + * and every delivery that did not come through that stop still has to match. + * What the stop hands the tracer through PTRACE_GETREGSET, and what the shim + * does with an X8 the tracer writes back, are pre-existing and out of this + * record's reach; docs/internals.md measures both. + * + * Per-vCPU: TLS, like cpu_tlbi_req and cpu_restart_req. + */ +static _Thread_local struct { + bool valid; + uint64_t elr; /* ELR_EL1 the value belongs to */ + uint64_t x8; /* what the guest had in X8 at that ELR */ +} sigreturn_x8; + +void signal_forget_sigreturn_x8(void) +{ + sigreturn_x8.valid = false; +} + +void signal_repark_sigreturn_x8(hv_vcpu_t vcpu) +{ + if (!sigreturn_x8.valid) + return; + sigreturn_x8.elr = vcpu_get_sysreg(vcpu, HV_SYS_REG_ELR_EL1); +} + +/* Hand the shim the X8 that EL0 must see when it drops the frame it is holding, + * by writing it into the frame's own X8 slot. + * + * That slot cannot collide with the marker the way a second register would: it + * is EL1-only memory on this vCPU's exception stack, unreachable from EL0 and + * unshared with any other vCPU, it holds nothing live once the frame is being + * dropped, and the shim's pop retires it. A drop that reaches the tail without + * a value published here gets the X8 the exception was taken with, which is + * guest state rather than a wire value. + * + * Only rt_sigreturn publishes, and only because it is the one rebuilder whose + * X8 differs from the one the frame was entered with. It is also the only + * caller that can promise the frame is live: it always arrives through HVC #5, + * where the vector entry has saved the frame and SP_EL1 still points at it. + * + * The bound is the EL1 stack region, not the shim data block that contains it. + * The block holds the shim-globals cache at the bottom -- identity slots, + * urandom ring, attention bitmask -- and only the slots thread_alloc_sp_el1 + * carves from the top are stack. An SP_EL1 that is wrong but still inside the + * block would otherwise pass, and the write would land in that cache: silent + * corruption of live shim state where the whole point of the check is to + * report. thread_sp_el1_region derives the slots from the constants the + * allocator hands them out with. + */ +static void shim_publish_frame_x8(hv_vcpu_t vcpu, + const guest_t *g, + uint64_t x8_for_el0) +{ + uint64_t sp_el1 = vcpu_get_sysreg(vcpu, HV_SYS_REG_SP_EL1); + uint64_t lo, hi; + + thread_sp_el1_region(g, &lo, &hi); + + if (sp_el1 < lo || sp_el1 > hi - SHIM_FRAME_BYTES) { + /* Nothing that took an exception through the shim can be here. Report + * it rather than write; the shim then hands EL0 the X8 the frame was + * entered with. + */ + log_error( + "rt_sigreturn: SP_EL1 0x%llx is outside the EL1 stack region " + "[0x%llx, 0x%llx)", + (unsigned long long) sp_el1, (unsigned long long) lo, + (unsigned long long) hi); + return; + } + + uint8_t *frame = (uint8_t *) g->host_base + sp_el1; + memcpy(frame + SHIM_FRAME_OFF_X8, &x8_for_el0, sizeof(x8_for_el0)); +} + /* Build and install the rt_sigframe for `signum` on the current thread, with * sig_lock held on entry and released on every return path. Shared by * signal_deliver() (signal selected from the process-wide pending set) and @@ -2304,6 +2422,20 @@ static int deliver_signal_locked(hv_vcpu_t vcpu, saved_pstate = vcpu_get_sysreg(vcpu, HV_SYS_REG_SPSR_EL1); } + /* X8 is not the guest's while the drop-frame marker occupies it, and an + * rt_sigreturn earlier in this same epilogue leaves the marker live with + * ELR_EL1 on the instruction the guest is about to resume, which can be an + * SVC it has not executed yet. Snapshotting the register there would record + * the marker as the guest's X8, and the handler's own rt_sigreturn would + * hand it back, so that SVC would run as syscall 2. Take what rt_sigreturn + * parked instead. A record exists only inside the epilogue that wrote it, + * so there is no later delivery for this to reach; see sigreturn_x8. + */ + if (!el0_preempt && sigreturn_x8.valid && sigreturn_x8.elr == saved_pc) { + saved_regs[8] = sigreturn_x8.x8; + sigreturn_x8.valid = false; + } + /* 1b. rseq abort: if the thread is in a restartable sequence critical * section, abort it. Linux does this on every signal delivery. */ @@ -2527,6 +2659,12 @@ static int deliver_signal_locked(hv_vcpu_t vcpu, * consume it. The EL0-preemption path resumes straight into the handler at * EL0 with no shim frame to drop, so the marker is neither needed nor * consulted. + * + * The marker is all this writes. What EL0 sees in X8 on that drop is + * whatever stands in the frame's X8 slot: the value the exception was taken + * with, or the one an rt_sigreturn published earlier in this same epilogue. + * Neither is the marker, which is the whole point; shim_publish_frame_x8 + * says why the slot and not a register. */ if (!el0_preempt) hv_vcpu_set_reg(vcpu, HV_REG_X8, 2); @@ -2828,7 +2966,20 @@ int signal_rt_sigreturn(hv_vcpu_t vcpu, guest_t *g) * restored the complete guest register state here; letting the shim restore * X1-X30 from the rt_sigreturn syscall entry would corrupt the interrupted * context. - */ + * + * X8 is restored above with the other 30 GPRs and then handed over again, + * because the marker is written into the same register. Without that second + * hand-off the marker is what the ERET delivers, and a frame whose saved PC + * sits on an SVC issues it as syscall 2. The same value is parked for a + * signal delivered later in this epilogue, which snapshots the register + * itself and would otherwise write the marker into the frame it builds. The + * park dies at the next vCPU resume whether or not anything took it. + */ + uint64_t restored_x8 = frame.uc.uc_mcontext.regs[8]; + shim_publish_frame_x8(vcpu, g, restored_x8); + sigreturn_x8.valid = true; + sigreturn_x8.elr = restored_pc; + sigreturn_x8.x8 = restored_x8; hv_vcpu_set_reg(vcpu, HV_REG_X8, 2); /* Return SYSCALL_EXEC_HAPPENED to skip the normal X0 writeback, since diff --git a/src/syscall/signal.h b/src/syscall/signal.h index b0068fc2..61b38bad 100644 --- a/src/syscall/signal.h +++ b/src/syscall/signal.h @@ -518,6 +518,41 @@ int signal_deliver_fault(hv_vcpu_t vcpu, */ int signal_rt_sigreturn(hv_vcpu_t vcpu, guest_t *g); +/* Drop the guest X8 that rt_sigreturn parked for a signal delivered later in + * the same host epilogue. + * + * The drop-frame marker occupies X8, so a delivery that follows an rt_sigreturn + * before the guest runs again cannot read the guest's X8 out of the register; + * signal_rt_sigreturn parks it and that delivery takes the parked value. The + * park is only ever correct for that stretch. Once the vCPU is resumed the + * guest owns X8 again, and a record left behind would be handed to some later + * delivery that merely resumes at the same PC -- a fault on an instruction the + * guest returned to, say, whose ELR_EL1 is the faulting PC and whose X8 is + * live. + * + * The vCPU run loop therefore calls this immediately before every resume, and + * scripts/check-svc-tails.py holds every hv_vcpu_run() call site to it. + */ +void signal_forget_sigreturn_x8(void); + +/* Re-key that record on the PC a ptrace stop left the guest at. + * + * Nothing moves the guest between the rt_sigreturn that parks an X8 and the + * resume that ends the record's life, with one exception: a PTRACE_INTERRUPT + * stop is taken inline on that same tail, and the tracer can write a new PC + * before it resumes the tracee with an injected signal. The delivery that + * follows then lands on an ELR_EL1 the record was not parked for, so the ELR + * check that keeps a stale record from being read misses the one delivery the + * record is genuinely for, and the frame it builds records the drop-frame + * marker as the guest's X8. + * + * Called with the vCPU still inside that epilogue, where the guest is the same + * guest with the same X8 owed to it and only the address it resumes from has + * moved. Nothing to do when no value is parked, which is every stop taken + * anywhere else. + */ +void signal_repark_sigreturn_x8(hv_vcpu_t vcpu); + /* Handle rt_sigaction (SYS 134). */ int64_t signal_rt_sigaction(guest_t *g, int signum, diff --git a/tests/manifest.txt b/tests/manifest.txt index 8ad6cce6..eea47d91 100644 --- a/tests/manifest.txt +++ b/tests/manifest.txt @@ -115,6 +115,7 @@ test-mprotect-mt # diff=skip [section] Signal + thread tests test-signal-thread test-signal-in-shim # diff=skip +test-shim-sigreturn-x8 test-sigsuspend test-fault-signal-mt # diff=skip test-exit-group-worker diff --git a/tests/test-matrix.sh b/tests/test-matrix.sh index 8f850b53..a9c495f8 100755 --- a/tests/test-matrix.sh +++ b/tests/test-matrix.sh @@ -646,10 +646,17 @@ test_pipe() # check") binary except the handful that assert elfuse-internal implementation # details with no meaningful counterpart on a real kernel (most of the EL1 shim # fast-path suite -- test-shim-* and test-shim-cred-race, which probe elfuse's -# own shim_data block and identity cache; test-shim-futex-fast is the exception -# and does run here, because every assertion in it is plain Linux futex ABI that -# a real kernel adjudicates (unlike test-mremap-infra, which guards elfuse's -# guest-IPA infra reserve, and test-oom-proc, documented in its own header). +# own shim_data block and identity cache). Two test-shim-* binaries are listed +# below rather than held out, each because every assertion in it is ABI a real +# kernel adjudicates: test-shim-futex-fast, which is plain Linux futex ABI, and +# test-shim-sigreturn-x8, which is plain Linux signal ABI throughout -- what +# rt_sigreturn restores, and what a handler enters with after a W^X permission +# fault or a BRK. Both are named here on purpose: a reader who finds one +# exception recorded takes it for the whole rule and holds the next one out. +# test-shim-sigreturn-x8 was run against this lane by hand when it was +# registered, on the fixture kernel of the day (Alpine 6.18.52-0-virt), and +# passes there. Contrast test-mremap-infra, which guards elfuse's guest-IPA +# infra reserve, and test-oom-proc, documented in its own header. # test-mremap-tail-emfile is listed here as an elfuse-lane regression and marked # QEMU_SKIP because its host-reserve assertion has no Linux analogue. There is # no "core" vs "extended" split here; everything below runs in both @@ -728,6 +735,8 @@ run_unit_tests() "$bindir/test-wait-process-signal" test_check "$runner" "test-wait-sigmask-signal" " - PASS" \ "$bindir/test-wait-sigmask-signal" + test_check "$runner" "test-shim-sigreturn-x8" "0 failed" \ + "$bindir/test-shim-sigreturn-x8" test_check "$runner" "test-ptrace-interrupt" "OK: ptrace-stop reports EL0" \ "$bindir/test-ptrace-interrupt" test_check "$runner" "test-sigsuspend" "PASS|0 failed" "$bindir/test-sigsuspend" diff --git a/tests/test-shim-futex-toctou.c b/tests/test-shim-futex-toctou.c index e86fe294..996cf78d 100644 --- a/tests/test-shim-futex-toctou.c +++ b/tests/test-shim-futex-toctou.c @@ -100,7 +100,9 @@ static void *spin_waiter(void *arg) /* Print the first one. This phase used to count these and say * nothing else, and the counter alone cannot tell a wrong errno * from an SVC that re-executed as a different syscall, which is - * what the open Tier A item about X8=2 turns out to be. + * what the Tier A item about X8=2 turned out to be + * (sysprog21/elfuse#379, now closed; tests/test-shim-sigreturn-x8 + * reaches the same window without a race). */ if (spin_other == 0) fprintf(stderr, "FAIL: unexpected spin rc %ld (round %d)\n", rc, diff --git a/tests/test-shim-sigreturn-x8.c b/tests/test-shim-sigreturn-x8.c new file mode 100644 index 00000000..3929ac2e --- /dev/null +++ b/tests/test-shim-sigreturn-x8.c @@ -0,0 +1,576 @@ +/* + * test-shim-sigreturn-x8.c -- rt_sigreturn hands EL0 the X8 the frame records. + * + * Copyright 2026 elfuse contributors + * SPDX-License-Identifier: Apache-2.0 + * + * A signal handler rewrites its own ucontext so the return lands on a stub that + * reads X8 and then issues an SVC with it. The kernel contract is that + * rt_sigreturn restores all 31 GPRs from the frame, so the stub must see the X8 + * the handler wrote and the SVC must run as that syscall. + * + * elfuse has one more thing to get right there. Its EL1 shim is told to drop + * the exception frame it is holding by a marker the host writes into X8, and + * the tail that consumes the marker restores no register, so the marker itself + * used to reach EL0 in place of the restored X8. A guest whose resumed PC sat + * on an SVC then issued it as syscall 2 and got ENOSYS out of a call it had + * made as something else, which is what tests/test-shim-futex-toctou caught by + * racing. This reaches the same window without a race: the handler puts the + * resume PC on the stub, so every round runs with a live X8 across the return. + * + * Rounds alternate, because the marker reaches the resumed SVC by two routes. + * Plain: the rt_sigreturn itself hands X8 to EL0. Nested: a second signal, + * raised inside the first handler and blocked until rt_sigreturn unblocks it, + * is delivered on the way out, and the frame that delivery builds is the one + * that records X8 for the return that follows. + * + * The nested route needs the host to remember, across the rt_sigreturn, what X8 + * the guest is owed, and a third phase holds that memory to its window. It + * returns to a stub, lets the guest run, and then faults on the same PC with a + * different X8 live. The fault is an ordinary delivery, not a continuation of + * the rt_sigreturn, so what the SIGSEGV handler sees has to be the register the + * guest faulted with and not anything the earlier return left parked. + * + * A fourth phase leaves the syscall route entirely. The same drop tail is + * branched to from the two W^X permission-fault handlers, where the host + * answers a flip request with the marker because the region never had the + * permission and the fault is a real SIGSEGV. Nothing publishes an X8 there, so + * the reload hands back the value the exception was taken with, and the two + * rounds assert that the handler enters with the guest's X8 rather than the + * marker. + * + * A fifth phase does the same for BRK, which carries the marker out of its own + * tail rather than exec_drop_frame's. JIT translators use BRK as a trampoline + * and read X8 in the SIGTRAP handler, so both halves are asserted: the X8 the + * handler enters with, and the X8 the resumed guest carries once its + * rt_sigreturn has run. + * + * Plain Linux signal ABI throughout, so the reference kernel adjudicates it and + * the test is registered in tests/test-matrix.sh rather than exempted from it. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* aarch64 __NR_getpid. Spelled out rather than taken from a header because the + * stub below has to load the same number into X8 from assembly. + */ +#define NR_GETPID 172 + +#define ROUNDS 32 + +/* aarch64 __NR_mprotect, and the two X8 values the stale-park phase tells + * apart. Spelled out because stale_stub below loads all three from assembly. + */ +#define NR_MPROTECT 226 +#define STALE_PARKED 0x55 +#define STALE_LIVE 0x99 + +/* Written by the stub, read by main. Not static: the stub addresses them by + * name through adrp/add. + */ +volatile uint64_t resumed_x8; +volatile int64_t resumed_rc; + +static sigjmp_buf resume; +static int failures; +static volatile sig_atomic_t nest; /* raise the second signal in handler */ +static volatile sig_atomic_t nested; /* the second handler ran */ + +void sigreturn_finish(void); +void sigreturn_stub(void); + +/* Runs at EL0 with the register state rt_sigreturn restored. Records X8, then + * issues the SVC that X8 names without touching it, so a corrupted X8 shows up + * twice: in the recorded value and in the syscall that ran. + */ +__asm__( + ".text\n" + ".globl sigreturn_stub\n" + ".type sigreturn_stub, %function\n" + "sigreturn_stub:\n" + " adrp x9, resumed_x8\n" + " add x9, x9, :lo12:resumed_x8\n" + " str x8, [x9]\n" + " svc #0\n" + " adrp x9, resumed_rc\n" + " add x9, x9, :lo12:resumed_rc\n" + " str x0, [x9]\n" + " b sigreturn_finish\n" + ".size sigreturn_stub, .-sigreturn_stub\n"); + +void sigreturn_finish(void) +{ + siglongjmp(resume, 1); +} + +static void handler(int sig, siginfo_t *info, void *ctx) +{ + (void) sig; + (void) info; + ucontext_t *uc = ctx; + + /* Resume on the stub with getpid live in X8. Everything else the frame + * holds is left alone, including the stack pointer, so the stub returns + * through siglongjmp on the interrupted thread's own stack. + */ + uc->uc_mcontext.pc = (uint64_t) (uintptr_t) sigreturn_stub; + uc->uc_mcontext.regs[8] = NR_GETPID; + + /* SIGUSR2 is in this handler's sa_mask, so it stays pending until + * rt_sigreturn restores the mask and is then delivered on the way out, with + * the resume PC and X8 already set above. + */ + if (nest) + raise(SIGUSR2); +} + +static void nested_handler(int sig) +{ + (void) sig; + nested = 1; +} + +/* Reached twice at the same PC, with X19 holding a writable page. + * + * First pass: rt_sigreturn resumes here with X8 = STALE_PARKED, the load + * succeeds, and the guest runs on. Second pass: the page has been made + * unreadable and X8 is STALE_LIVE, so the load faults with the fault PC equal + * to the PC the earlier return came back to. + */ +void stale_stub(void); + +__asm__( + ".text\n" + ".globl stale_stub\n" + ".type stale_stub, %function\n" + "stale_stub:\n" + " ldr x1, [x19]\n" + " mov x0, x19\n" + " mov x1, #4096\n" + " mov x2, #0\n" + " mov x8, #226\n" /* NR_MPROTECT: PROT_NONE over the page */ + " svc #0\n" + " mov x8, #0x99\n" /* STALE_LIVE */ + " b stale_stub\n" + ".size stale_stub, .-stale_stub\n"); + +static uint8_t *stale_page; +static sigjmp_buf stale_back; +static volatile uint64_t stale_x8; +static volatile uint64_t stale_pc; +static volatile sig_atomic_t stale_faulted; + +static void stale_redirect(int sig, siginfo_t *info, void *ctx) +{ + (void) sig; + (void) info; + ucontext_t *uc = ctx; + uc->uc_mcontext.pc = (uint64_t) (uintptr_t) stale_stub; + uc->uc_mcontext.regs[8] = STALE_PARKED; + uc->uc_mcontext.regs[19] = (uint64_t) (uintptr_t) stale_page; +} + +static void stale_fault(int sig, siginfo_t *info, void *ctx) +{ + (void) sig; + (void) info; + ucontext_t *uc = ctx; + stale_faulted = 1; + stale_x8 = uc->uc_mcontext.regs[8]; + stale_pc = uc->uc_mcontext.pc; + siglongjmp(stale_back, 1); +} + +static int check_stale_park(void) +{ + static char altbuf[SIGSTKSZ * 4]; + struct sigaction sa; + stack_t ss; + int failures = 0; + + stale_page = mmap(NULL, 4096, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (stale_page == MAP_FAILED) { + perror("mmap"); + return 1; + } + + /* The fault is taken on stale_stub, which runs on the interrupted thread's + * own stack; an alternate stack keeps the handler off it. + */ + ss.ss_sp = altbuf; + ss.ss_size = sizeof(altbuf); + ss.ss_flags = 0; + if (sigaltstack(&ss, NULL) < 0) { + perror("sigaltstack"); + return 1; + } + + memset(&sa, 0, sizeof(sa)); + sa.sa_sigaction = stale_redirect; + sa.sa_flags = SA_SIGINFO; + sigemptyset(&sa.sa_mask); + if (sigaction(SIGUSR1, &sa, NULL) < 0) { + perror("sigaction SIGUSR1"); + return 1; + } + + memset(&sa, 0, sizeof(sa)); + sa.sa_sigaction = stale_fault; + sa.sa_flags = SA_SIGINFO | SA_ONSTACK; + sigemptyset(&sa.sa_mask); + if (sigaction(SIGSEGV, &sa, NULL) < 0) { + perror("sigaction SIGSEGV"); + return 1; + } + + if (sigsetjmp(stale_back, 1) == 0) + raise(SIGUSR1); + + if (!stale_faulted) { + fprintf(stderr, "FAIL stale park: the second pass never faulted\n"); + return 1; + } + if (stale_pc != (uint64_t) (uintptr_t) stale_stub) { + fprintf(stderr, + "FAIL stale park: faulted at 0x%llx, want the stub at " + "0x%llx\n", + (unsigned long long) stale_pc, + (unsigned long long) (uintptr_t) stale_stub); + failures++; + } + if (stale_x8 != STALE_LIVE) { + fprintf(stderr, + "FAIL stale park: the fault reports X8 = 0x%llx, want the live " + "0x%x; 0x%x is the value the earlier rt_sigreturn returned " + "with\n", + (unsigned long long) stale_x8, STALE_LIVE, STALE_PARKED); + failures++; + } + return failures; +} + +/* The same drop tail is reached from the W^X permission faults, not only from a + * syscall: the shim asks the host to flip a page and branches to the tail when + * the host answers with the marker, which it does when the region never had the + * permission and the fault is a real SIGSEGV. Nothing publishes an X8 on that + * route, so what the tail reloads is the value the exception was taken with, + * and that reload is the last write to X8 before EL0: signal delivery installs + * the handler's PC, SP, LR and argument registers, and X8 is none of those. X8 + * at handler entry is therefore the guest's own on both routes, which is what + * these two rounds assert; before the reload existed they both entered with the + * marker. + * + * Recorded in assembly at the handler's first instruction. The ucontext cannot + * answer this: the frame the host builds for these two deliveries records the + * shim's HVC arguments in X0 and X1 rather than the guest's, so reading X8 out + * of it would be reading a snapshot taken on the same terms. + * + * The value is spelled twice, here and in the two stubs below, which load it + * from assembly the way stale_stub does. + */ +#define WX_X8 0xa5 + +/* Written by the handler stub, read by the checker. Not static: the stub + * addresses it by name through adrp/add. + */ +volatile uint64_t wx_entry_x8; + +void wx_entry(int sig, siginfo_t *info, void *ctx); +void wx_entry_c(int sig, siginfo_t *info, void *ctx); +void wx_exec(uint64_t page); +void wx_write(uint64_t page); + +__asm__( + ".text\n" + ".globl wx_entry\n" + ".type wx_entry, %function\n" + "wx_entry:\n" + " adrp x9, wx_entry_x8\n" + " add x9, x9, :lo12:wx_entry_x8\n" + " str x8, [x9]\n" + " b wx_entry_c\n" + ".size wx_entry, .-wx_entry\n" + ".globl wx_exec\n" + ".type wx_exec, %function\n" + "wx_exec:\n" + " mov x8, #0xa5\n" + " br x0\n" + ".size wx_exec, .-wx_exec\n" + ".globl wx_write\n" + ".type wx_write, %function\n" + "wx_write:\n" + " mov x8, #0xa5\n" + " str xzr, [x0]\n" + " ret\n" + ".size wx_write, .-wx_write\n"); + +static sigjmp_buf wx_back; + +void wx_entry_c(int sig, siginfo_t *info, void *ctx) +{ + (void) sig; + (void) info; + (void) ctx; + siglongjmp(wx_back, 1); +} + +/* The X8 the handler entered with, through *taken. A stub that returns instead + * of faulting leaves it false, which is a host that granted the permission the + * page was not mapped with rather than an X8 worth reporting. + */ +static uint64_t wx_take_fault(void *page, int write, int *taken) +{ + wx_entry_x8 = 0; + *taken = 1; + if (sigsetjmp(wx_back, 1) == 0) { + if (write) + wx_write((uint64_t) (uintptr_t) page); + else + wx_exec((uint64_t) (uintptr_t) page); + *taken = 0; + return 0; + } + return wx_entry_x8; +} + +static int check_wx_tails(void) +{ + struct sigaction sa; + int failures = 0; + uint32_t ret_insn = 0xd65f03c0; /* ret, so a page that is executable runs */ + + /* A page with no PROT_EXEC, branched to: an instruction permission fault. A + * page with no PROT_WRITE, stored to: a write permission fault. + */ + uint8_t *noexec = mmap(NULL, 4096, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + uint8_t *nowrite = mmap(NULL, 4096, PROT_READ | PROT_EXEC, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (noexec == MAP_FAILED || nowrite == MAP_FAILED) { + perror("mmap"); + return 1; + } + memcpy(noexec, &ret_insn, sizeof(ret_insn)); + + /* Both faults are taken outside any handler and neither faulting stub has + * touched the stack, so the handler runs on the interrupted stack and needs + * no alternate one. check_stale_park, which does need one, runs after this + * and installs it then. + */ + memset(&sa, 0, sizeof(sa)); + sa.sa_sigaction = wx_entry; + sa.sa_flags = SA_SIGINFO; + sigemptyset(&sa.sa_mask); + if (sigaction(SIGSEGV, &sa, NULL) < 0) { + perror("sigaction SIGSEGV"); + return 1; + } + + int taken = 0; + uint64_t got = wx_take_fault(noexec, 0, &taken); + if (!taken) { + fprintf(stderr, + "FAIL W^X exec tail: branching to a page with no PROT_EXEC " + "did not fault\n"); + failures++; + } else if (got != WX_X8) { + fprintf(stderr, + "FAIL W^X exec tail: the handler entered with X8 = 0x%llx, " + "want the guest's 0x%x\n", + (unsigned long long) got, WX_X8); + failures++; + } + + got = wx_take_fault(nowrite, 1, &taken); + if (!taken) { + fprintf(stderr, + "FAIL W^X write tail: storing to a page with no PROT_WRITE " + "did not fault\n"); + failures++; + } else if (got != WX_X8) { + fprintf(stderr, + "FAIL W^X write tail: the handler entered with X8 = 0x%llx, " + "want the guest's 0x%x\n", + (unsigned long long) got, WX_X8); + failures++; + } + return failures; +} + +/* BRK reaches EL0 through a tail of its own, not exec_drop_frame's, and has the + * same problem to solve there. The host delivers SIGTRAP out of HVC #10 and + * leaves the drop-frame marker in X8 as it does on every other delivery, so + * without a reload the handler enters with 2 in place of the guest's X8. A JIT + * translator patching through BRK trampolines is precisely the caller that + * reads that register, which is why the tail loads it back from the frame's own + * X8 slot before dropping the frame. + * + * Both halves are asserted. Handler entry is recorded in assembly, for the + * reason the W^X phase gives: the ucontext is a snapshot the host took on its + * own terms. The resumed value is read after the handler steps the saved PC + * past the BRK and returns, which covers the rt_sigreturn out of a BRK- + * delivered signal, a different tail from the one the entry tests. + * + * The value is spelled twice, here and in brk_fire below, which loads it from + * assembly the way the W^X stubs do. + */ +#define BRK_X8 0xc3 + +/* Written by the handler stub and by brk_fire, read by the checker. Not static: + * both address them by name through adrp/add. + */ +volatile uint64_t brk_entry_x8; +volatile uint64_t brk_resumed_x8; + +void brk_entry(int sig, siginfo_t *info, void *ctx); +void brk_entry_c(int sig, siginfo_t *info, void *ctx); +void brk_fire(void); + +__asm__( + ".text\n" + ".globl brk_entry\n" + ".type brk_entry, %function\n" + "brk_entry:\n" + " adrp x9, brk_entry_x8\n" + " add x9, x9, :lo12:brk_entry_x8\n" + " str x8, [x9]\n" + " b brk_entry_c\n" + ".size brk_entry, .-brk_entry\n" + ".globl brk_fire\n" + ".type brk_fire, %function\n" + "brk_fire:\n" + " mov x8, #0xc3\n" + " brk #0\n" + " adrp x9, brk_resumed_x8\n" + " add x9, x9, :lo12:brk_resumed_x8\n" + " str x8, [x9]\n" + " ret\n" + ".size brk_fire, .-brk_fire\n"); + +void brk_entry_c(int sig, siginfo_t *info, void *ctx) +{ + (void) sig; + (void) info; + ucontext_t *uc = ctx; + + /* Step the saved PC past the BRK so the return lands on the store below it + * rather than re-executing the trap. Nothing else in the frame is touched, + * so the X8 the return delivers is the one the frame recorded. + */ + uc->uc_mcontext.pc += 4; +} + +static int check_brk_tail(void) +{ + struct sigaction sa; + int failures = 0; + + memset(&sa, 0, sizeof(sa)); + sa.sa_sigaction = brk_entry; + sa.sa_flags = SA_SIGINFO; + sigemptyset(&sa.sa_mask); + if (sigaction(SIGTRAP, &sa, NULL) < 0) { + perror("sigaction SIGTRAP"); + return 1; + } + + brk_entry_x8 = 0; + brk_resumed_x8 = 0; + brk_fire(); + + if (brk_entry_x8 != BRK_X8) { + fprintf(stderr, + "FAIL BRK tail: the handler entered with X8 = 0x%llx, want " + "the guest's 0x%x\n", + (unsigned long long) brk_entry_x8, BRK_X8); + failures++; + } + if (brk_resumed_x8 != BRK_X8) { + fprintf(stderr, + "FAIL BRK tail: the resumed guest carries X8 = 0x%llx, want " + "the guest's 0x%x\n", + (unsigned long long) brk_resumed_x8, BRK_X8); + failures++; + } + return failures; +} + +int main(void) +{ + struct sigaction sa; + memset(&sa, 0, sizeof(sa)); + sa.sa_sigaction = handler; + sa.sa_flags = SA_SIGINFO; + sigemptyset(&sa.sa_mask); + sigaddset(&sa.sa_mask, SIGUSR2); + if (sigaction(SIGUSR1, &sa, NULL) < 0) { + perror("sigaction SIGUSR1"); + return 1; + } + + struct sigaction sa2; + memset(&sa2, 0, sizeof(sa2)); + sa2.sa_handler = nested_handler; + sigemptyset(&sa2.sa_mask); + if (sigaction(SIGUSR2, &sa2, NULL) < 0) { + perror("sigaction SIGUSR2"); + return 1; + } + + int64_t want_pid = (int64_t) getpid(); + + for (int round = 0; round < ROUNDS * 2; round++) { + resumed_x8 = 0; + resumed_rc = 0; + nest = round & 1; + nested = 0; + + if (sigsetjmp(resume, 1) == 0) { + raise(SIGUSR1); + fprintf(stderr, "FAIL round %d: handler did not redirect\n", round); + failures++; + continue; + } + + if (resumed_x8 != NR_GETPID) { + fprintf(stderr, + "FAIL round %d: X8 after rt_sigreturn is %llu, " + "want %d\n", + round, (unsigned long long) resumed_x8, NR_GETPID); + failures++; + } + if (resumed_rc != want_pid) { + fprintf(stderr, + "FAIL round %d: the resumed SVC returned %lld, " + "want the pid %lld\n", + round, (long long) resumed_rc, (long long) want_pid); + failures++; + } + if (nest && !nested) { + fprintf(stderr, "FAIL round %d: the nested signal never ran\n", + round); + failures++; + } + } + + failures += check_wx_tails(); + failures += check_brk_tail(); + failures += check_stale_park(); + + if (!failures) + printf( + "PASS: %d rt_sigreturn rounds kept X8, a later fault on the same " + "PC saw its own, and the W^X and BRK tails handed theirs to the " + "handler\n", + ROUNDS * 2); + printf("%d failed\n", failures); + return failures ? 1 : 0; +}