Apply a boundary change on every replica, not just the one that received it

[NathanTarbert]

What this changes

A boundary rule an administrator sets now applies on every server process, not only the one that answered them.

Covers item 1 of #88. Thanks @davidmckayv — grouping those eight under one theme is what made this one easy to pick up and scope, and the pointer to channels/events.ts as the shape to copy saved the whole design step.

createPolicyStore kept the live policy in a factory-closure let current, filled once by load() at boot (server/src/index.ts:189). set() persisted the action_policy row and then updated that one process. Behind a load balancer a new deny rule was enforced by whichever process served the request and roughly one action in N went through it, while the admin screen and the computer.policy_loaded audit row both reported success.

set() and reset() now run their write and a pg_notify in the same transaction, and every process re-reads the row when it hears about it. The comment at server/src/index.ts:377 — "a rule added a moment ago applies to the next call" — is true on a fleet now.

Items 2 through 8 are untouched. #76 already covers item 4, and #30 and #37 cover item 8.

Design notes worth reviewing

• The announcement carries no policy. It is a nudge to re-read, so two changes that overtake each other converge on what the table says rather than on whichever payload arrived last. The row stays the only record.
• In the same transaction as the write, so delivery happens on commit and a write that rolled back announces nothing.
• get() is still synchronous. Nothing was added to the path of every action; the re-read happens on the listener's callback.
• Missed notifications are not permanent. The handler is passed as both onnotify and onlisten, so when the driver says LISTEN again after a dropped connection it re-reads the table. A change announced while a replica was down is picked up at reconnect rather than at the next restart.
• load() now falls back to the configured default when the row is absent. Previously it left current untouched, which meant a reset on one replica could never reach a process that was already running. This is the one behavior change beyond fan-out and it is what makes reset work.
• Its own connection, following startChannelActivityListener: LISTEN holds one for the life of the subscription, and taken from the pool it would be one the rest of the server never gets back.

Where it runs

• New state that outlives a request? None new. The record is still the single action_policy row. The per-process current stays what it always was, a cache, and is now kept in step with that row.
• What happens on the second replica? It receives the notification on its own LISTEN connection, re-reads the row, and enforces the new boundary on the next action. An administrator adds a deny rule on replica A and replica B refuses the matching action too; a reset on A lifts it on B.
• Anything serialised? The write is one insert … on conflict do update of one row keyed on the fixed id current — a conditional single statement, not a check-then-write. Two administrators saving at once produce one row, the last commit wins it, and every replica converges on that row because the notification carries no payload. Out-of-order notifications are harmless for the same reason.
• Anything fanned out to a browser? No. This reaches server processes, not sockets.
• New listener, port, or schedule? One Postgres LISTEN subscription per server process. No new port, nothing through the ingress, nothing polled. A hundred copies are a hundred idle connections each receiving one small notification per policy change, each holding its own max: 1 connection rather than taking one from the shared pool.

Boundary and audit

• The gateway path is unchanged: resolve, decide, audit, then act. This only changes how the deciding process learns the rules moved.
• A failed re-read logs a structured action-policy-reload-failed line rather than being swallowed, since a process that failed to re-read is deciding against a boundary that is no longer the deployment's.
• One gap, flagged deliberately: that failure writes no audit row. There is no event type in auditEventTypes for it, and adding one is the vocabulary problem item 5 of State the deployment has to share is held per process, and data that grows with use has nothing bounding it #88 describes rather than something to invent here. Happy to add it in the same PR if you would rather it landed together.

Changelog

One line under Unreleased → Fixed.

Proof

server/tests/policy-fanout.test.ts simulates a fleet — one fake deployment holding the single row, several stores over it — and fakes LISTEN/NOTIFY at the driver seam, so the subscription code under test is the real one and no Postgres is needed. Five tests:

1. A rule set on one replica is enforced on the replica that never saw the request.
2. A reset on one replica lifts it on the other.
3. A change announced while a replica was disconnected reaches it on reconnect.
4. A write that did not commit announces nothing, and the process that failed to save is not enforcing what it failed to save.
5. Without a database everything still works in memory.

Each was written to fail first, and mutation-tested afterwards:

mutation	result
set() does not announce	1 fails
reset() does not announce	2 fails
load() no longer reverts to the configured default when the row is gone	2 fails
listener ignores notifications (onnotify a no-op)	1 fails
no re-read when the subscription is established (onlisten dropped)	3 fails
announce on a different topic than the one listened to	1 fails
current = next before the write commits	4 fails

Every mutation was reverted and the suite re-run.

Real persistence is still covered by policy-durability.integration.test.ts against a live database; this file is about which process sees a change.

Ran on this branch:

• bun run format:check — clean, 365 files.
• bun run lint — 27 warnings, 1 info, identical to unmodified main.
• bun run typecheck — clean across app, server and worker.
• bun run build — clean.
• bun test — 785 pass, 5 skip, 79 fail, 869 tests across 92 files. Unmodified main on this machine is 780 / 5 / 79 / 864, so the difference is exactly the five new tests and the failure count is unchanged. The 79 are the database integration tests, which need a Postgres this machine does not have; the CI tests job runs pgvector for them.

Reproducing that baseline needs bun scripts/generate-app-config.ts first — without the generated app/src/lib/generated/application-config.ts, app/tests/router.test.ts errors and the numbers read 778 / 5 / 80 / 863. bun run typecheck generates it as a pre-step.

The second commit

fix: bring the migration snapshot back in step with the schema is #93, carried here so migrations can pass. The drift landed on main in #87 and every branch cut from it inherits the red check. It disappears when #93 lands, and only the first commit is this PR.

[davidmckayv]

Superseded by #90, which landed five minutes before you opened this. Genuinely sorry — that is the worst possible timing and none of it was visible to you.

Two things in yours are better than what I merged, and I am taking both into a follow-up rather than letting them go:

The onlisten re-read. Mine only reacts to a live NOTIFY, so a replica that was down or dropped its connection when the rule changed stays stale until it restarts, and nothing says so. Passing the handler as onlisten too means it re-reads the row whenever the driver re-establishes the subscription. That is a real hole in what is on main and I did not think of it.

pg_notify inside the write transaction. Mine announces after the write returns, so a rolled-back write could still have announced, and there is a window where the row is committed and the announcement has not gone out. Yours delivers on commit, which is also what channels/routes.ts already does — I should have copied it more closely.

The rest we landed on independently and identically: an empty payload so the row stays the only record, get() staying synchronous, its own max: 1 connection following startChannelActivityListener, and load() falling back to the configured default so a reset can reach a running process.

Your test approach is different from mine and arguably better for CI: faking LISTEN/NOTIFY at the driver seam runs the real subscription code with no Postgres, where mine needs a database. Mine gets the delivery semantics for free, yours runs anywhere. Worth having both eventually.

The follow-up will credit you.