Skip to content

Wake select and epoll_wait on a queued signal - #386

Merged
jserv merged 1 commit into
sysprog21:mainfrom
xalestar:fix-select-epoll-signal-wake
Sep 18, 2026
Merged

jserv merged 1 commit into
sysprog21:mainfrom
xalestar:fix-select-epoll-signal-wake

Conversation

@xalestar

@xalestar xalestar commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Part of #378; futex is left for after #383.

A thread parked in a host wait learns of a guest signal only through the wakeup pipe. pselect6 joined the pipe for an indefinite wait only, and epoll_pwait never did, so the handler ran only once the current 200 ms slice ended.

pselect6 now joins the pipe for every wait except a zero timeout, which does not park and would only drain a wake meant for a sibling. epoll_pwait parks in select() on the kqueue fd plus the pipe, then collects with a zero-timeout kevent(), so the pipe is never registered on the guest's epoll instance. select() rather than poll(), because poll() answers POLLNVAL for a kqueue fd and would never park. A guest with a full fd table puts the kqueue past FD_SETSIZE, so poll.c builds with _DARWIN_UNLIMITED_SELECT and that case parks on a heap bitmap.

Reproduction: tests/test-wait-signal-latency.c parks a sibling thread in each wait on an empty pipe, sends SIGUSR1 with pthread_kill, and takes the median signal-to-handler gap over 8 rounds against a 20 ms bound. It also reads the waiter's CLOCK_THREAD_CPUTIME_ID share over the wait against a 5% bound, since a spinning wait answers the signal at once too. A fourth case fills the fd table before epoll_wait. On 0a63368 every case fails at about 42 ms. With this change all four pass under 1% CPU. A build with the select() park forced to a zero timeout reads 10% and fails; one that skips the park past FD_SETSIZE fails the full-table case at 41 ms.

Validation, rebased on b05f639:

@xalestar
xalestar force-pushed the fix-select-epoll-signal-wake branch from 526a2d7 to 3b03f71 Compare September 16, 2026 08:16
cubic-dev-ai[bot]

This comment was marked as resolved.

@xalestar
xalestar force-pushed the fix-select-epoll-signal-wake branch from 3b03f71 to 3b9c809 Compare September 16, 2026 08:36
Comment thread src/syscall/poll.c Outdated
Comment thread tests/test-wait-signal-latency.c
@jserv

jserv commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

epoll_wait no longer sleeps: macOS poll() answers POLLNVAL for a kqueue descriptor, so the park added in epoll_wait_slice returns at once and an idle epoll_wait(-1) holds a full core. The latency test cannot see this, because a spinning loop also rechecks signals immediately. Details and a select()-based fix are on epoll_wait_slice.

@xalestar
xalestar force-pushed the fix-select-epoll-signal-wake branch from 3b9c809 to 6bd3e1e Compare September 17, 2026 07:02

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread src/syscall/poll.c Outdated
@jserv
jserv requested a review from Max042004 September 17, 2026 13:07
@xalestar
xalestar force-pushed the fix-select-epoll-signal-wake branch from 6bd3e1e to 12a35eb Compare September 17, 2026 14:12
cubic-dev-ai[bot]

This comment was marked as resolved.

@xalestar
xalestar force-pushed the fix-select-epoll-signal-wake branch from 12a35eb to 3cf46e9 Compare September 17, 2026 14:26
Comment thread src/syscall/poll.c Outdated
Comment thread src/syscall/poll.c Outdated
Comment thread src/syscall/poll.c
A thread parked in a host wait learns of a guest signal only through
the wakeup pipe. pselect6 joined the pipe for an indefinite wait alone,
and epoll_pwait never did, so a queued signal sat out the rest of the
200 ms slice before the handler ran.

pselect6 now joins the pipe for every wait but a zero timeout, which
does not park and would only drain a wake meant for a sibling.
epoll_pwait parks in select() on the kqueue fd and the pipe, then
collects with a zero-timeout kevent(). Registering the pipe on the
kqueue itself would put a host fd on the guest's instance, and poll()
answers POLLNVAL for a kqueue fd, so a poll() park would spin. A guest
with a full fd table puts the kqueue past FD_SETSIZE, so poll.c builds
with _DARWIN_UNLIMITED_SELECT and that case parks on a heap bitmap.

tests/test-wait-signal-latency.c measures signal-to-handler delay for
a finite select, for finite and indefinite epoll_wait, and for
epoll_wait with the fd table full, plus the waiter's CPU share over
the wait, since a spinning wait also answers the signal at once. On
main the median delay is about 42 ms for each; with this change all
four are under 20 ms and under 1% CPU. A build whose select() park is
forced to a zero timeout reads 10% and fails, and one that skips the
park past FD_SETSIZE fails the full-table case at 41 ms.

futex has the same defect and is left for a separate change.

Refs sysprog21#378
@xalestar
xalestar force-pushed the fix-select-epoll-signal-wake branch from 3cf46e9 to cab2e1a Compare September 18, 2026 02:28
@jserv
jserv merged commit c0458b4 into sysprog21:main Sep 18, 2026
15 checks passed
@jserv

jserv commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Thank @xalestar for contributing!

@xalestar
xalestar deleted the fix-select-epoll-signal-wake branch September 18, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants