A signal delivered in a syscall epilogue records the TLBI wire values as the guest's `X8` through `X11`

[jotpalch]

Symptom

A guest calls mprotect, a signal is delivered in the host's return epilogue for
that call, and the instruction the guest resumes on is another svc. That svc
runs as a syscall the guest never asked for:

WARN  src/syscall/syscall.c:2841: unimplemented syscall 3 (x0=0x0, x1=0x1000, x2=0x3, x3=0x22, x4=0xffffffffffffffff, x5=0x0)

The guest held 226 (__NR_mprotect) in X8 at both instructions. The handler
sees the same corruption directly, without needing a second svc to expose it:
uc_mcontext.regs[8] reads 0 or 3, and on the deliveries where it reads 3,
regs[9], regs[10] and regs[11] read the operands of a TLB invalidation
rather than what the guest put there.

This is not #379 and is not closed by #383. #383 changes which wrong number
reaches EL0 on this path. It does not change whether one does, and the rate is
unmoved: in the runs below every delivery corrupts the register set on both
trees.

Mechanism

The host uses X8 through X11 as a wire protocol to ask the shim for a TLB
invalidation on the way out of a page-table syscall, and the signal-delivery
path snapshots those registers as though they were the guest's.

1. Dispatch finishes the syscall and calls tlbi_request_emit_to_vcpu
   (src/syscall/syscall.c:2911), which overwrites the live X8 through X11
   with the request (src/core/guest.h:674-708): X8 is the kind, 0 for none,
   1 for broadcast, 3 for a range, 4 for a large range; X9 and X10 carry the
   range start and page count; X11 is the I-cache hint. The comment at
   src/syscall/proc.c:4089-4091 states the invariant this relies on, that the
   helper has already written X8 by the time the epilogue runs.

2. syscall_return_epilogue then delivers any pending signal
   (src/syscall/proc.c:4132), after that write and before the vCPU resumes.

3. deliver_signal_locked snapshots the live GPRs (src/syscall/signal.c:2415)
   and copies them into the rt_sigframe it builds
   (src/syscall/signal.c:2512). The registers it copies are the wire values,
   so the frame records X8 as 0/1/3/4 and X9 through X11 as TLBI operands.

4. The handler's rt_sigreturn restores all 31 GPRs from that frame
   (src/syscall/signal.c:2894) and hands them back to EL0, so the guest
   resumes with the wire values in place of its own registers.

The park that #383 adds does not cover this. It is read at
src/syscall/signal.c:2434 and is keyed on a record only signal_rt_sigreturn
writes (src/syscall/signal.c:2980-2982); on the ordinary syscall-return route
no rt_sigreturn has run in this epilogue, so sigreturn_x8.valid is false and
the snapshot takes the live register.

What #383 does change is the value that arrives. On afdcfce the delivery
writes the drop-frame marker into X8 and the tail restores no register, so
EL0 sees 2. On #383 the tail reloads X8 from the frame slot
(src/syscall/signal.c:2978-2979), and the frame slot holds what step 3
recorded, so EL0 sees the wire value. Both are wrong, and the second is
arguably harder to spot, since 2 is a constant and the wire value varies with
the mapping being changed.

The X9 through X11 half is present identically on both trees. #383 declines
it deliberately, because it is the same epilogue and not the register that
change moves.

Measurements

Apple M1, macOS 15.6.1. The machine was shared for every run below: two
logged-in users, load average 3.24 to 3.73 across the set, so the delivery counts
are a product of that scheduling and are not a rate to quote elsewhere. Rounds
are interleaved, one tree then the other, so both binaries met the same machine.
Population is 60000 mprotect rounds per run under a 200 us ITIMER_REAL, each
round two adjacent svcs with the first the mprotect.

tree	round	deliveries	second svc ran as the wrong call	X8 it ran as	ucontext X8 not 226	X9/X10/X11 not the guest's
#383 at 03bf6e9	1	760	759	0 x462, 3 x297	761 of 761	299
#383 at 03bf6e9	2	704	704	0 x410, 3 x294	704 of 704	294
#383 at 03bf6e9	3	715	715	0 x440, 3 x275	715 of 715	275
afdcfce	1	710	710	2 x710	710 of 710	286
afdcfce	2	705	705	2 x705	705 of 705	275
afdcfce	3	715	715	2 x715	715 of 715	297

Reading the table:

• The ucontext X8 column is 100 percent on both trees. Every signal delivered
  in one of these epilogues reports a syscall number the guest did not set.
• The second svc column is the same 100 percent, one delivery aside in round 1,
  where the last delivery landed between the two counters being printed.
• The only column that moves between the trees is which wrong number arrives:
  2 on afdcfce, 0 or 3 on Restore the guest X8 when the shim drops its frame #383.
• X9 through X11 are corrupted on exactly the deliveries where X8 is 3,
  which is TLBI_RANGE, the only kind that writes them. The observed values are
  X9 = 0x200000000 (the range start, a guest VA), X10 = 0x1 (one page) and
  X11 = 0x0 (no I-cache flush), against sentinels of 0xa9, 0xaa and 0xab.
  The counts match the X8 = 3 counts exactly in five of the six rounds, and are
  off by two in round 1 for the same print-ordering reason.

Reproduction

/* repro-epilogue-x8.c -- does a signal delivered in the epilogue of a
 * page-table syscall record the TLBI wire values as the guest's X8-X11, and
 * does a guest returning from that handler onto an unexecuted SVC issue the
 * shadowed number?
 */
#define _GNU_SOURCE
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/time.h>
#include <ucontext.h>
#include <unistd.h>

#define SENT8 226   /* __NR_mprotect, what the guest holds at the SVC */
#define SENT9 0xa9
#define SENT10 0xaa
#define SENT11 0xab

static volatile uint64_t hits, bad8, bad9, bad10, bad11;
static volatile uint64_t s8[8], s9[8], s10[8], s11[8];
static volatile uint64_t n8, n9, n10, n11;

static void note(volatile uint64_t *set, volatile uint64_t *n, uint64_t v)
{
    for (uint64_t k = 0; k < *n; k++)
        if (set[k] == v)
            return;
    if (*n < 8)
        set[(*n)++] = v;
}

static void h(int s, siginfo_t *i, void *c)
{
    (void) s;
    (void) i;
    ucontext_t *uc = c;
    uint64_t x8 = uc->uc_mcontext.regs[8], x9 = uc->uc_mcontext.regs[9];
    uint64_t x10 = uc->uc_mcontext.regs[10], x11 = uc->uc_mcontext.regs[11];
    hits++;
    if (x8 != SENT8) { bad8++; note(s8, &n8, x8); }
    if (x9 != SENT9) { bad9++; note(s9, &n9, x9); }
    if (x10 != SENT10) { bad10++; note(s10, &n10, x10); }
    if (x11 != SENT11) { bad11++; note(s11, &n11, x11); }
}

/* x0 = page, x1 = iterations. Two adjacent SVCs: the first is mprotect, the
 * second executes with whatever X8 holds when the guest resumes there.
 */
volatile uint64_t twin_bad, twin_hist[16];
void twin_svc(void *page, uint64_t iters);
__asm__(
    ".text\n.globl twin_svc\n.type twin_svc, %function\n"
    "twin_svc:\n"
    "    stp x19, x20, [sp, #-32]!\n"
    "    str x21, [sp, #16]\n"
    "    mov x19, x0\n    mov x20, x1\n    mov x21, #1\n"
    "1:  mov x0, x19\n    mov x1, #4096\n    mov x2, x21\n"
    "    eor x21, x21, #2\n"
    "    mov x9, #0xa9\n  mov x10, #0xaa\n  mov x11, #0xab\n"
    "    mov x8, #226\n"
    "    svc #0\n"
    "    svc #0\n" /* unexecuted SVC at the resume PC */
    "    mov x12, x8\n   cmp x12, #226\n   b.eq 2f\n"
    "    adrp x13, twin_bad\n  add x13, x13, :lo12:twin_bad\n"
    "    ldr x14, [x13]\n add x14, x14, #1\n str x14, [x13]\n"
    "    adrp x13, twin_hist\n add x13, x13, :lo12:twin_hist\n"
    "    and x15, x12, #15\n   add x13, x13, x15, lsl #3\n"
    "    ldr x14, [x13]\n add x14, x14, #1\n str x14, [x13]\n"
    "2:  subs x20, x20, #1\n   b.ne 1b\n"
    "    ldr x21, [sp, #16]\n  ldp x19, x20, [sp], #32\n    ret\n"
    ".size twin_svc, .-twin_svc\n");

static void dump(const char *name, uint64_t bad, volatile uint64_t *set,
                 uint64_t n, uint64_t want)
{
    printf("  %s: %llu of %llu deliveries not 0x%llx; saw", name,
           (unsigned long long) bad, (unsigned long long) hits,
           (unsigned long long) want);
    for (uint64_t k = 0; k < n; k++)
        printf(" 0x%llx", (unsigned long long) set[k]);
    printf("\n");
}

int main(void)
{
    struct sigaction sa;
    struct itimerval it;
    memset(&sa, 0, sizeof sa);
    sa.sa_sigaction = h;
    sa.sa_flags = SA_SIGINFO | SA_RESTART;
    sigaction(SIGALRM, &sa, NULL);
    void *p = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
                   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    it.it_interval.tv_sec = 0;
    it.it_interval.tv_usec = 200;
    it.it_value = it.it_interval;
    setitimer(ITIMER_REAL, &it, NULL);

    twin_svc(p, 60000);
    printf("deliveries=%llu  second-SVC ran as the wrong call %llu of 60000\n",
           (unsigned long long) hits, (unsigned long long) twin_bad);
    printf("  X8 seen at the second SVC:");
    for (int k = 0; k < 16; k++)
        if (twin_hist[k])
            printf(" %d=%llu", k, (unsigned long long) twin_hist[k]);
    printf("\n");
    dump("ucontext X8 ", bad8, s8, n8, SENT8);
    dump("ucontext X9 ", bad9, s9, n9, SENT9);
    dump("ucontext X10", bad10, s10, n10, SENT10);
    dump("ucontext X11", bad11, s11, n11, SENT11);
    it.it_value.tv_sec = 0;
    it.it_value.tv_usec = 0;
    it.it_interval = it.it_value;
    setitimer(ITIMER_REAL, &it, NULL);
    return 0;
}

Save as tests/repro-epilogue-x8.c, then:

make elfuse
aarch64-linux-gnu-gcc -static -O2 -o build/repro tests/repro-epilogue-x8.c
./build/elfuse build/repro

One run takes about 10 seconds. It needs no loop and no load to reproduce: a
single run on #383 at 03bf6e9 printed

deliveries=708  second-SVC ran as the wrong call 708 of 60000
  X8 seen at the second SVC: 0=421 3=287
  ucontext X8 : 709 of 709 deliveries not 0xe2; saw 0x3 0x0
  ucontext X9 : 287 of 709 deliveries not 0xa9; saw 0x200000000
  ucontext X10: 287 of 709 deliveries not 0xaa; saw 0x1
  ucontext X11: 287 of 709 deliveries not 0xab; saw 0x0

The X8 seen at the second SVC line is the whole difference between the trees:
2= on afdcfce, 0= and 3= on #383.

Suggested direction

@jserv's observation on the #383 review is the one to start from, and is quoted
rather than adopted here, since nothing below has been implemented or measured.
The shim's exception frame is live at SP_EL1 for this delivery too, exactly as
it is for the rt_sigreturn delivery that #383 already publishes into. So the
same frame slot that #383 reads could supply X8 for this route as well, and
X9 through X11 beside it, with the parked value winning when it is set.

Two things that would have to be settled and are not settled here. The frame slot
holds the registers the exception was taken with, which is the guest's state for
this route, so a plain read is plausible; whether it is correct for every entry
into the epilogue is not something these runs show. And #383 is careful that
shim_publish_frame_x8 may only be called where the frame is provably live
(src/syscall/signal.c:2298 and the comment above it), so extending the reader
to a second route means extending that proof, not just the call.

What this does not show

• Which kinds other than TLBI_RANGE and TLBI_NONE are affected. Only
  X8 = 0 and X8 = 3 were observed. TLBI_BROADCAST (1) and
  TLBI_RANGE_LARGE (4) are written by the same helper and are presumably
  identical, but they were not produced by this workload and were not measured.
• Any syscall other than mprotect. The reproduction uses one call on one
  page. Every page-table syscall reaching tlbi_request_emit_to_vcpu should
  behave the same way; none was tried.
• The HVC #11 EL0-fault route. tlbi_request_emit_to_vcpu serves that path
  too (src/core/guest.h:667-672 names both call sites). Not tested.
• A consequence in a real program. The second svc in the reproduction is
  there to make the corrupted register visible, and 3 happens to be io_cancel
  on aarch64, which is unimplemented and answers -ENOSYS loudly. A guest
  whose shadowed X8 names an implemented syscall would run it silently, and
  whether that occurs in any real workload was not investigated.
• A rate. The machine was shared throughout. The delivery counts track the
  200 us itimer against a loaded 8-core host and mean nothing on their own; the
  proportions within a run are what the table is for.
• Any interaction with A guest signal does not interrupt futex, epoll_wait or a finite select #378. Not examined.

[jserv]

@Max042004 , can you do measurements on M4 for more evidences?