[earakely-scale]

Part of FD-5420: a computer's own clock follows the gateway's universe clock. This is FD-5471, the gateway side. MCP servers read the gateway's get_time on every call, but a computer can't, because every program on it reads its own clock. So the computer follows the gateway through a long poll, and a clock change doesn't return until every computer shows it.

What changes

• One version. The clock has one opaque version, a per-boot id and a count. Every set_time, advance and clear changes it, so a version never repeats across gateway restarts. /clock/state and every write's answer carry it.
• GET /clock/watch?after=<version>&follower=<name>. It answers at once when the clock's version isn't after, else at the next change or after 25 s. The answer is the clock's state with the gateway's wall-clock received and sent stamps, which a follower uses for an NTP-style offset. A watch is its computer's acknowledgement of the version it names. It counts only when it carries that computer's read token, and never for a version the clock hasn't had.
• Declared computers. The gateway provider lists each child whose card serves clock/v1 sync_time in CLOCK_FOLLOWER_TOKENS (name=read token), on the VM and container paths alike. For now the read token is the computer's control token, a slot a separate token can fill later with no protocol change. TOKEN in the name keeps the tokens out of logged compose files.
• Writes that wait. set_time, advance and clear take wait_followers_s (0 to 30). Given one, the write waits for every declared computer to show its version. It answers 504 naming the computers still behind, and the change itself stands. A write without one, or to a gateway with no declared computers, answers as before.
• set_rate (PUT /clock/set-rate). It changes an armed clock's rate from its time as of the write. Like advance, it keeps t0 and the clock's generation, so time triggers keep their relative marks and their waits, and it wakes the trigger engine, so a wait runs at the new rate. set_time still requires a virtual_time, as before.
  • In the prototype, a rate change made through set_time with a time computed from a state read a moment earlier stepped the universe back by the gap. A computer, which never steps back, then ran 65–85 ms ahead.
• One path for every clock change.
  • Every change goes through Gateway._clock_changed: it re-evaluates the triggers when time moved, answers every held watch, and waits for the followers. Future writers, such as restoring the clock after a restart (FD-5433) or a snapshot reload, can't skip a step.
  • The four write routes share _clock_write's body handling.
• The watch contract in the card. The card's watch method carries its request (after, follower) and hold_s (25), which a follower's request timeout must exceed. Every wait_followers_s is bounded 0–30 in the schema.
• Restart. A restarted gateway is unarmed on a new boot id, so every follower's next watch answers at once.

Nothing on main declares a follower yet: computers start serving sync_time in FD-5476. Until then CLOCK_FOLLOWER_TOKENS is never rendered, and every clock route answers as before.

Not in this PR:

• FD-5472: 403s for unknown or mismatched tokens, and clock writes behind the gateway's control token.
• FD-5473: core passing wait_followers_s.

Docs

The clock extension's page should cover version, the watch method, wait_followers_s with its 504, and set_time without virtual_time as a rate change.

Testing

• Unit: tst/unit/env/gateway/clock_watch_test.py and clock_version_test.py cover:

  • version changes;
  • watch timing;
  • acknowledgements only with the computer's token and never for a version the clock hasn't had;
  • the 504 naming stale computers, with the change standing;
  • unarmed after a restart;
  • set_rate continuous with the clock before it, keeping t0, the generation and a relative trigger's mark;
  • set_time without a time and set_rate without a rate refused;
  • every write refusing a body that isn't a JSON object;
  • 100 advances acknowledged in order.

  Provider and multi-env tests cover rendering and redacting CLOCK_FOLLOWER_TOKENS. The full unit suite passes locally (14,387 tests), apart from one Windows disk-recipe test that fails only on macOS.

• E2E (071322a2, an EC2 host in our AWS account):

  • Setup: the branch's gateway image in Docker, declaring two computers in CLOCK_FOLLOWER_TOKENS. Each computer is a real follower process long-polling /clock/watch with its token.
  • Hundred advances: 100 of 100 advances with wait_followers_s: 2 answered 200, each acknowledged by both computers, in order. The write took 1.46 ms at p50 and 1.87 ms at p95; the prototype measured an acknowledgement wait of about 1.3 ms at p50.
  • Rate changes: the step a computer would see at the write was under 0.13 ms, across ×1↔×60 and ×1↔×2. Taking the time from a state read 70 ms before the write instead steps the clock back 74.3 ms, which is the prototype's 65–85 ms drift.
  • A paused computer (SIGSTOP): the write answered 504 stale: ["laptop"] after 2,002 ms, and the change stood. After SIGCONT, the next write was acknowledged in 2.3 ms.
  • A killed computer: 504 naming it after 1,002 ms. Once its follower restarted, the next write was acknowledged in 3.0 ms.
  • Forged acknowledgements made during a write's wait all left the laptop stale (504): another computer's token, a wrong token, no token, and the laptop's own token naming a version the clock hadn't had. The laptop's own token naming the current version, the control case, ended the wait.
  • Gateway restart (docker restart): both followers' next watches answered unarmed on a new boot id within 46 ms of the gateway coming up, and re-arming with a wait answered 200.
  • clear: with a wait, it answered 200 once both computers showed it unarmed.
  • No declared computers: a write with a wait answered at once, and the old bodies still work (set_time without wait_followers_s, clear with no body).

• Remote providers (261462b6, the CI's Modal keys):

  • Setup: the branch's gateway image, built by the provider from its context, ran as the gateway provider runs it, on both modal (a container on the private network) and modal_vm (a container in the VM). It declared two computers, each followed through the gateway's public HTTPS tunnel: desktop from a Modal sandbox of its own, laptop from outside Modal.
  • Advances: 30 of 30 with a wait answered 200 and were acknowledged in order by both computers. A write took 147 ms at p50 on modal and 149 ms on modal_vm, about two round trips through the tunnel.
  • Long polls: across an 80 s idle stretch, every watch was held its full 25 s and answered 200, with no errors over the whole run. The tunnels don't cut them.
  • set_rate kept t0, and both computers picked up ×60.
  • A paused computer: 504 naming the laptop after 3.07 s, the change standing, then acknowledged in about 147 ms once resumed.
  • clear with a wait answered 200.
  • Other remote providers weren't run: this account has no credentials for them. FD-5487 covers them.

• Integration on the same host (local sandboxes):

  • gateway_test.py's clock tests pass: test_gateway_clock_extension, test_gateway_clock_rate_accuracy, test_gateway_time_trigger_extension, test_gateway_autonomous_time_trigger and test_sync_env_clock_step.
  • The hook tests pass: test_advance_to_next_event_hook_local, test_next_env_event_hook_local, test_register_env_triggers_local and test_prompt_agent_notifications_local.
  • The Windows disk-recipe test that fails on macOS passes on Linux.

• What the e2e caught: the entrypoint parsed CLOCK_FOLLOWER_TOKENS with the URL-map parser, so a gateway with declared computers didn't start (fixed in 91428dc1, with a unit test). It also caught gateway_test.py's exact unarmed-state assertion (updated in 071322a2).

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no blocking issue remains.

Summary

Adds versioned clock watches, token-checked computer acknowledgements, writes that wait for followers, and continuous rate changes through set_rate.

• Clock rate changes keep time continuous and give writes fresh versions.
• Clock writes wait for declared computers to show the new time.

Diagram

sequenceDiagram
    participant Caller
    participant Gateway
    participant Computer
    Computer->>Gateway: Watch with shown version and read token
    Caller->>Gateway: Clock write with wait_followers_s
    Gateway->>Gateway: Change version and wake held watches
    Gateway-->>Computer: New clock state
    Computer->>Computer: Apply clock state
    Computer->>Gateway: Watch naming new version
    Gateway-->>Caller: 200 after acknowledgements, or 504 naming stale computers

Loading

_{Reviews (12) · Last reviewed commit: "fix(gateway): a wait or rate past a floa..." · Reviewed by Greptile}