Skip to content

feat(computers): agent-env's libfaketime for the computer's clock, gated on x86_64 and arm64 (FD-5474) - #619

Open
earakely-scale wants to merge 5 commits into
mainfrom
edgararakelyan/fd-5474-hb14-patched-libfaketime-for
Open

earakely-scale wants to merge 5 commits into
mainfrom
edgararakelyan/fd-5474-hb14-patched-libfaketime-for

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Part of FD-5420, the computer's own clock following the gateway's universe clock. This is FD-5474: the library every computer's session programs will load to read universe time. Every program on a computer reads its own clock, so it can't ask the gateway's get_time the way an MCP server does. Instead, agent-env's build of libfaketime answers those programs' clock reads and clock-time waits from a page a follower process keeps in step with the gateway. Wiring the library into the base computer (/etc/ld.so.preload, the follower, supervisord) is FD-5476.

What it is

src/agent_env/computers/clock/:

File What it is License
libfaketime.patch agent-env's changes to upstream 0.9.12's src/, applied with patch -p1 GPL-2.0-only
agentenv_clock_page.c settings read once at load, who is faked, the clock page's reader; included by the patched libfaketime.c GPL-2.0-only
agentenv_clock_waits.c universe-time reads and waits, the per-process helper thread; included the same way GPL-2.0-only
COPYING GPL-2.0, libfaketime's license
build.sh builds libfaketimeMT.so.1 from upstream's sha256-pinned tarball and the files above Apache-2.0
clockpage.py the clock page's writer, which the follower will use Apache-2.0
clock.conf the library's default settings Apache-2.0

What the library does:

  • Universe time: clock reads (CLOCK_REALTIME, _COARSE, _ALARM, TAI, gettimeofday, time) and waits for a set realtime clock time (clock_nanosleep TIMER_ABSTIME, realtime futex waits, sem_timedwait and sem_clockwait, realtime condvars, absolute realtime timerfds and POSIX timers). Each wait is planned in real time and planned again whenever the page changes.
    • Where the kernel has no futex_waitv (before 5.16, or gVisor), realtime futex and semaphore waits read the page between 100 ms slices instead.
    • Waits stay cancellation points, as glibc's are.
    • A timerfd is followed through its duplicates (dup, dup2, dup3, fcntl, close and close_range are interposed), not by its number.
  • Real time: the monotonic and boot clocks, and every relative sleep and timeout, at any rate.
  • Who: only the session user is faked. A program started with AGENTENV_REAL_TIME=1 stays real, and its children are faked.
  • Settings: read once as the library loads, from /etc/agentenv/clock.conf, or from the environment where there's no such file. While the file exists, the environment changes none of them, so a session program can't point the library at a page of its own. Stock libfaketime's getenv raced programs' own setenv; that crashed Chromium on about one launch in five and hangs it outright in headless mode.

The patch and the engine files are the prototype's library, moved out of the two Python scripts that applied it as string replacements (about 900 lines of C inside a Python string). Their preprocessed source was byte-identical to the prototype's before the fix below.

A bug the gates found, fixed here

The gates found it on arm64: with the follower stalled 5 ms in a write, CLOCK_REALTIME stepped back up to 280 ns at a rate change, 3 to 7 times per 2,000 writes.

  • Cause: a clock read took its real time before copying the page. A reader that waited out a write therefore paired a time from before the write with the page the write made. A write is continuous with the page it replaces only at its own real time.
  • Fix: the read now copies the page, then reads the real time, and keeps the pair only if the page is still the one it copied (ft_snap_at). The real time is ordered before that check: a dependent load and dmb ishld on aarch64, lfence on x86_64.
  • Result: 0 backsteps in three runs of about 90 million reads each.
  • Cost: a second vDSO read per clock read.

Greptile's first review, fixed here

All seven findings were real; each has a gate now.

  • Cancelled condvar waits: they left their entry in the library's table, and the helper could broadcast to a condvar the caller had since freed. A cleanup handler now removes the entry.
  • Semaphore waits ignored cancellation: sem_timedwait used raw futex calls. It now checks for cancellation first, cancels asynchronously for the system call alone, and a cleanup handler takes the thread back off the waiter count. clock_nanosleep's universe sleep does the same.
  • Without futex_waitv: the fallback translated a deadline once, or not at all, and stopped watching the page. It now waits on the caller's word in slices of at most 100 ms and plans again from the page between them. Modal runs sandboxes on gVisor, so this path is live there.
  • Timerfd records followed a number: after dup and close, the helper re-armed a closed number or another timer. Records now hold the timer's descriptors.
  • A dead page writer: it left seq odd, and every later write stayed odd. clockpage.write now goes on from the next even value and skips the continuity rule for that write.
  • Settings docs: clock.conf said the environment could override the file. It can't, on purpose, and the docs now say so.
  • Return codes: the wait gates now check what every wait returns, not only when it returned.

Greptile's second review (4/5) found two more, also fixed:

  • Failed descriptor calls: a dup2, dup3 or close_range that failed still dropped the target timer's record. Calls that replace or close a timerfd now run with their bookkeeping under the table lock, so a failed one keeps the record.
  • Unexpected futex errors: without futex_waitv, a futex error such as EINVAL for an unaligned word was retried as a page wake, spinning a core. Such errors now go back to the caller. futex_waitv is taken as missing only on ENOSYS, or EPERM from a seccomp filter.

The patch's headers are back to a/ and b/ without timestamps; a regeneration on macOS had left pa/ and pb/ in them.

Where it lives

It's at computers/clock/, beside the bundle's other shared recipes (disk/, vm_bundle/), not at artifacts/ubuntu-24.04/clock/ as the issue said. Inside the base computer's folder it would change the base's build context, and with it the image's tag, though nothing loads the library yet. FD-5476 brings it into the base.

Licensing

The patch and the two engine files are derived from libfaketime, so they carry GPL-2.0-only and upstream's COPYING sits beside them. Everything else in the folder and the repo stays Apache-2.0, and the package's license metadata is unchanged.

THIRD_PARTY_NOTICES.md gets a libfaketime section. I also changed its opening sentence, which said no third-party source is incorporated in the repository; that's no longer true for these three files. That sentence is legal wording, so please check the change. Whether the wheel's license metadata should also list GPL-2.0 for these files is a question for legal.

Tests

tst/clock (make clock-test, .github/workflows/clock.yml on ubuntu-24.04 and ubuntu-24.04-arm, not a required check):

  • Setup: it builds this library and stock 0.9.12 in an Ubuntu 24.04 image, the base computer's own base, and runs each gate in a container. Chrome's archive (the browsers bundle's pin, read from chromium.py) and upstream's tarball are fetched once into a cache, checked by sha256, and kept between CI runs.

The gates:

  • Waits: all 64 wait APIs in C, Python, Java and Node at rates 1, 2 and 60, and frozen with a +1 h advance. Elapsed waits stay real (1.000–1.021 s); clock-time waits follow the rate (0.50 s at ×2, 0.017–0.106 s at ×60) and end at the advance. Every C and Python wait returns 0, and every jump waiter returns what it should (ETIMEDOUT, one expiration, 0).
  • Rate change and set_time back: ×1 to ×60 ends clock-time waits at 1.15 s, and a set_time 182 days back leaves them waiting. Three single-call condvar waits return once at the change, the spurious return POSIX allows and the design records.
  • Jumps: 30 forward jumps of 8 h, a day or a week. Waits inside the jump end within 1.5 ms of it (the mutex wait within its 100 ms slice). A wait a jump or a change ends is timed from both ends of the page write: never before the write starts, and within its bound of the write's end. The page changes before the write's last step (the clock file), which once took 55 ms on CI's amd64 runner. Waits past it are never early. Relative waits stay 3.00 s. Monotonic time never steps back.
  • Without futex_waitv: the waits, rate-change, jump, cancellation and semaphore-stress gates run a second time in containers whose seccomp profile answers futex_waitv with ENOSYS, as gVisor and kernels before 5.16 do. A probe checks that it does.
  • Cancellation: threads cancelled in sem_timedwait and clock_nanosleep go at once, and the semaphore keeps no waiter. 600 threads are cancelled in pthread_cond_timedwait, more than the table holds. A condvar wait after them still ends at a jump past its deadline. A realtime futex wait on an unaligned word fails with EINVAL at once.
  • Timerfd duplicates: once each for dup, dup2, dup3, fcntl(F_DUPFD_CLOEXEC) and close_range. A timerfd armed an hour ahead is duplicated and closed, and its number goes to a duplicate of a monotonic timerfd. A 2 h jump fires the duplicate at the jump and leaves the monotonic one alone. A last case makes dup2, dup3 and close_range calls on the timerfd fail, and it still fires at the jump.
  • Stress: realtime condvar handoffs and sem_timedwait posts, with the page changing under them, lose nothing; every handoff is under 2 ms.
  • Dead writer: a page left with seq odd is taken up by the next write, and session programs read it.
  • Stalled writer: 2,000 writes stalled 5 ms each: realtime and monotonic never step back.
  • Torn reads: a black-box test of the library itself. The page is rewritten as fast as possible between two universes while 4 threads read CLOCK_REALTIME: 0 torn reads in about 319 million on arm64.
  • Who is faked, and the marker: only the session user's processes follow the page. A marked program stays real, loses the marker, and its children are faked.
  • Environment race: setenv racing clock reads and timed waits, 3 runs of 10 s each: no crash.
  • Chromium: launched directly 40 times with this library, each launch printing its page and exiting 0. Stock 0.9.12 fails all 10 of its launches (it hangs). The two variants run side by side in containers of their own, and the gate must take at most 30 s.
    • CI's amd64 runner is the tight one: 60 launches in sequence with stock took 32.6 s, and 60 side by side took 31.4 s on a later run.
    • 40 clean launches are still conclusive: at stock's rate of one crash in five, a run would come out clean about once in 7,500.
    • The 40 take about 10 s on 4 CPUs here.

Results:

  • arm64 (this Mac's Docker): all 28 tests pass on 23bc736, in 10 minutes, and the 17 wait tests again on e193801. The jump gate is the longest, at 150 s per variant.
  • x86_64 (kernel 7.0) and arm64 (Graviton) EC2 hosts: all 28 tests pass on 0a54493, in 9 minutes each (all 16 passed on the first commit).
  • CI on 7690e0d: all 36 checks pass, clock.yml included. That job passes all 28 tests on amd64 (10 min) and arm64 (9 min 45 s). Before 7690e0d, amd64 failed once on a jump round's measurement (fixed in e193801) and once on the Chromium time (fixed in 7690e0d).

Docs

The docs site should cover the computer's clock: what follows universe time and what stays real, who is faked, AGENTENV_REAL_TIME, and clock.conf. That page belongs with FD-5476, when computers load the library.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The changes since the previous review appear safe to merge.

Summary

Adds the computer clock library, its shared clock page writer, and Docker-based gates for x86_64 and arm64. Loading the library into computers remains a separate change.

  • The clock library gives session programs a shared universe time.
  • Clock gates check the library on amd64 and arm64.

Reviews (8) · Last reviewed commit: "test(computers): 40 Chrome launches with..." · Reviewed by Greptile

…ted on x86_64 and arm64 (FD-5474)

src/agent_env/computers/clock is the library every computer's session programs will load to read universe
time: upstream libfaketime 0.9.12 from its sha256-pinned tarball with agent-env's changes, as a small
libfaketime.patch and two engine files it includes (agentenv_clock_page.c, agentenv_clock_waits.c;
GPL-2.0-only, as libfaketime is), built by build.sh, with the page writer the follower uses (clockpage.py)
and the library's default settings (clock.conf). Clock reads and waits for a set realtime clock time follow
the universe; monotonic and boot clocks and every relative wait stay real; only the session user is faked;
settings are read once at load.

A clock read now copies the page before reading the real time and keeps the pair only if the page is still
the one copied: a reader that waited out a write paired a time from before it with the page it wrote, which
stepped CLOCK_REALTIME back by up to ~280 ns at a rate change.

tst/clock (make clock-test, .github/workflows/clock.yml on x86_64 and arm64) builds it and stock 0.9.12 in an
Ubuntu 24.04 image and gates it: every wait API in C, Python, Java and Node at several rates, a rate change,
a set_time back and 30 jumps, condvar and semaphore stress, a stalled writer, torn reads, who is faked, the
marker, the environment race and Chromium launched directly 60 times (stock 0.9.12 as the positive control),
in under 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 11, 2026 17:20
Comment thread tst/clock/waits_test.py Resolved
…s by file, a dead page writer (FD-5474)

Greptile's review of the clock library:
- a condvar wait cancelled mid-wait gives up its table entry (cleanup handler);
- sem_timedwait and the universe clock_nanosleep are cancellation points again, and a cancelled semaphore wait leaves
  the waiter count as it found it;
- without futex_waitv (before Linux 5.16, gVisor, a seccomp filter), realtime futex and semaphore waits wait on the
  word in slices and plan again from the page between them, instead of a deadline translated once or not at all;
- a timerfd's record holds its descriptors, kept through dup, dup2, dup3, fcntl, close and close_range;
- clockpage.write takes up a page a dead writer left with seq odd;
- clock.conf and the README say settings are file-only while the file exists.

The gates check every wait's return code, and new gates cover cancellation, timerfd duplicates, a dead writer, and the
waits again in containers that refuse futex_waitv. The patch's headers are a/ and b/ again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 3 commits November 11, 2026 10:29
…eep a timerfd's record, Chromium variants side by side (FD-5474)

- Without futex_waitv, a slice's end is the only reason to read the page again; any other futex error (EINVAL for an
  unaligned word) is the caller's, and futex_waitv is taken as missing only on ENOSYS or a seccomp EPERM.
- dup2 and dup3 that replace a timerfd, and close_range that closes one, make the call and its bookkeeping under the
  table lock, so a failed call keeps the record.
- The Chromium gate runs its 60 launches with this library and stock's 10 in containers side by side: CI's amd64 runner
  took 32.6 s in sequence, over the 30 s budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s write (FD-5474)

The page changes, and waits wake, before the write's last step (the clock file); on CI's amd64 runner that step once
took 55 ms, so waits a jump had ended correctly read as 55 ms early against the write's end. A wait a jump or a change
ends must now end no earlier than its write started and within its bound of the write's end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s 30 s on CI's amd64 runner (FD-5474)

60 launches alone took 31.4 s there on one run. 40 clean launches are still conclusive: at stock's one crash in five, a
run would come out clean about once in 7,500.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant