feat(eventing): approved-agent identity — sign and verify requests and responses

eventing/agentdocs/DESIGN_PHASE2.md

@@ -249,12 +249,13 @@ identity, which is §4's territory.
 
 ---
 
-## 4. Agent identity: groundwork, not yet wired
+## 4. Agent identity: wired
 
 The second question — *which agent produced this answer?* — matters more than it
-first appears. Today anything with write access to the `responses` topic gets its
-output stored, rendered in the HTML transcript, and pushed to the operator's
-phone **as a legitimate agent answer**.
+first appears. Before this, anything with write access to the `responses` topic
+got its output stored, rendered in the HTML transcript, and pushed to the
+operator's phone **as a legitimate agent answer**. With verification enabled it
+lands as a rejection instead.
 
 ### 4.1 What exists
 
@@ -273,14 +274,21 @@ phone **as a legitimate agent answer**.
 an unnamed token is ambiguous, and guessing would mean accepting a signature
 from *any* approved agent for an event that named none of them.
 
-### 4.2 The blocker, stated plainly
+### 4.2 The blocker, now cleared
 
-**`sign_event()` has no production caller.** `ER_REQUIRE_SIGNATURE=true` today
-rejects 100% of traffic — it is a kill switch, not a feature.
-`IMPLEMENTATION_REPORT1.md` §811-814 already says "EventBridge does not sign."
-Nothing about signed attributes means anything until that is fixed, which is why
-`submitter` is **not** in `SIGNED_ATTRS` yet: adding it before the signing side
-exists would only invalidate canonicalisation twice.
+This section used to read "**`sign_event()` has no production caller**" —
+`ER_REQUIRE_SIGNATURE=true` rejected 100% of traffic, a kill switch rather than a
+feature. That is fixed. EventBridge signs the requests and group events it
+publishes, EventRunner signs terminal responses, and each verifies the other's
+output against the approved-key set.
+
+`submitter`, `submitteriss` and `groupid` joined `SIGNED_ATTRS` **after** the
+signing side existed, in that order deliberately: covering them earlier would
+have invalidated canonicalisation twice for no benefit. Nothing had ever
+published a signed event, so the change needed no compatibility flag — but if
+signing is already enabled somewhere, both services must be upgraded together,
+because every grouped request now carries a signed `groupid` that an older
+verifier omits when it recomputes.
 
 ### 4.3 Why not Keycloak, and why not HMAC
 
@@ -298,27 +306,61 @@ upgrades cleanly: SPIRE later distributes the same keys rooted in workload
 attestation, and the verification code does not change — only where keys come
 from.
 
-### 4.4 The remaining work
-
-1. Sign requests in `kafka_out.publish_request`, after `ce.new_event` (which
-   fills `id`/`time`, both signed) and before `to_kafka_binary`.
-2. Sign responses in `emit.py` between event construction and serialisation.
-   Hold the seed **on the Emitter** — its constructor runs once, so none of the
-   seven `emit()` call sites change. Sign **terminal events only**: `emit()` is
-   on the hot path for every `stdout` frame and Ed25519 costs ~150 ms here.
-3. Verify responses in `kafka_in.py` while the `CloudEvent` is still in hand.
-   Two hazards: the decode and store write are **not** inside a `try`, so a raise
-   kills the consumer thread — the verification path must degrade, never raise.
-   And group events are published by EventBridge itself, carry `groupid` but no
-   `correlationid`, so they need either their own `kid` or a skip.
-4. Then add `submitter`, `submitteriss` and the missing `groupid` to
-   `SIGNED_ATTRS`. `DESIGN_PHASE1.md` §21.9.9 requires `groupid`; its absence
-   means signatures currently say nothing about batch membership.
-
-On failure, set `phase="error"`. That reuses machinery already wired: ntfy
-**priority 5** with an error tag, a visually distinct bubble in the live SSE
-transcript, and `raw_json` persistence for audit. The demo artifact costs
-nothing extra.
+### 4.4 How it is wired
+
+1. **Requests** are signed in `kafka_out.publish_request`, after `ce.new_event`
+   fills `id`/`time` (both signed) and before `to_kafka_binary`, so the signature
+   covers exactly what goes on the wire.
+2. **Terminal responses only** are signed in `emit.py`. The seed lives on the
+   Emitter, whose constructor runs once, so none of the seven `emit()` call sites
+   changed. `emit()` runs per `stdout` frame and Ed25519 costs ~150-200 ms here,
+   so signing every frame would add minutes to a chatty run.
+
+   **The verifier has to match that policy, and initially did not.** Verifying
+   all-or-nothing rewrote every streamed frame of every genuine run to
+   `phase="error"` — found by running it against a live broker, not by any unit
+   test, because every test until then used terminal events. So an event carrying
+   **no** signature and **not** terminal is passed through; an unsigned *terminal*
+   event is still refused, since that is the one the transcript presents as the
+   answer. A frame that presents a bad signature is still checked — the exemption
+   is for absence, not for failure.
+
+   The honest limit that remains: this proves *who finished a run*, not *what it
+   said along the way*. Forged `final=false` frames still render. Closing that needs
+   either cheap signatures or a signed digest chain across frames.
+3. **Requests are verified** in `consume.py`, keyed on the token's `kid` when a
+   keyset is configured and falling back to the single-key path otherwise. A
+   rejection commits the offset without running, because a bad signature is still
+   bad on redelivery, and increments `rejected_unsigned` so it is distinguishable
+   from the stale-request drop.
+4. **Responses are verified** in `kafka_in.py` while the `CloudEvent` is still in
+   hand — the check needs `.attrs`/`.data`, which `envelope_dict` has flattened.
+   The decision itself is a pure function in `shared/signing.py`, which is what
+   makes this path testable at all: the consumer had no behavioural test before.
+5. **Group events are signed** under EventBridge's own `kid`, and the verifier
+   accepts *only* that kid for them. Skipping them instead would have left the one
+   hole worth closing — a forged `group.completed` ends a batch early and fires a
+   "finished" notification for work that never ran. A flat keyset is not enough
+   here: any approved runner would otherwise do.
+6. `submitter`, `submitteriss` and `groupid` joined `SIGNED_ATTRS` last (§4.2).
+
+**Both hazards in step 4 were real.** `from_kafka_binary` and `insert_response`
+are not inside a `try`, so anything raising in the verification path would end the
+consume loop for the life of the pod — silently, with the process still healthy.
+The guard around the decision call fails closed only where enforcement is on: if
+the verifier itself is broken, an unverifiable event is not evidence of anything.
+
+On failure the event is stored with `phase="error"` rather than dropped — a drop is
+indistinguishable from an agent that never answered. That reuses machinery already
+wired: ntfy **priority 5** with an error tag (the error body comes from
+`data["text"]`, so the key name is load-bearing), a red card in the live SSE
+transcript, and `raw_json` persistence for audit.
+
+**Rollout is two flags.** A keyset alone verifies and logs while storing events
+unchanged; `EB_REQUIRE_RESPONSE_SIGNATURE=true` is what rewrites them. Enforcement
+mutates persisted rows and pages a phone, so there is a step where the reject rate
+is observable first. EventBridge refuses to start with a keyset but no
+`EB_SIGNING_KID`, since there would be nothing to attribute a group event to.
 
 ---
 
@@ -332,9 +374,28 @@ nothing extra.
 | `EB_AUTH_TOKENS` | empty | `name:token,...` fallback. Empty plus no GitHub config means auth is off. |
 | `EVENTBRIDGE_TOKEN` | unset | CLI: overrides the stored token, so a shell can act as another identity. |
 
-The client id lives in `config.toml` because it is not a capability. `test_manifests.py`
-pins the opposite rule for `NTFY_TOPIC`/`NTFY_TOKEN`, and that distinction is the
-point: one is public by construction, the others grant access.
+Agent identity (§4). Every one of these is off by default, so the e2e path is
+unaffected until an operator opts in:
+
+| Variable | Default | Effect |
+|---|---|---|
+| `EB_SIGNING_KEY_PATH` | empty | Ed25519 seed EventBridge signs requests and group events with. Empty = no signing. |
+| `EB_SIGNING_KID` | empty | Names EventBridge's key. Also the **only** kid accepted on group lifecycle events. |
+| `EB_VERIFY_KEYSET_PATH` | empty | Approved-key set for responses. Empty = no verification. |
+| `EB_REQUIRE_RESPONSE_SIGNATURE` | `false` | `false` = verify and log (audit); `true` = rewrite failures to `phase=error`. |
+| `ER_SIGNING_KEY_PATH` | empty | Seed EventRunner signs terminal responses with. |
+| `ER_SIGNING_KID` | empty | Names this runner's key. Needed once more than one runner is approved. |
+| `ER_REQUIRE_SIGNATURE` | `false` | Refuse unsigned or badly-signed requests. |
+| `ER_VERIFY_KEYSET_PATH` | empty | Approved-key set for requests. Empty falls back to `ER_VERIFY_KEY_PATH`'s single key. |
+
+Seed paths name **Secret** mounts and are env-only, never `config.toml`. Keyset
+paths name **ConfigMaps** — only public keys belong in them, and `keyset.load()`
+cannot tell a seed from a public key by length, so the file's location is what keeps
+the distinction reviewable.
+
+The GitHub client id lives in `config.toml` because it is not a capability.
+`test_manifests.py` pins the opposite rule for `NTFY_TOPIC`/`NTFY_TOKEN`, and that
+distinction is the point: one is public by construction, the others grant access.
 
 ---
 

eventing/agentdocs/IMPLEMENTATION_REPORT1.md

@@ -812,6 +812,9 @@ That is RQ-1 behaving exactly as designed, observed by accident.
   tested against the RFC vectors and the flag is wired through `consume.py`, but no
   publisher signs requests yet, so the verify-and-reject path has not been
   exercised on a cluster. EventBridge does not sign.
+  *(Resolved in Phase 2: EventBridge signs requests and group events, EventRunner
+  signs terminal responses, and both verify against the approved-key set. The
+  reject paths now have tests. See `DESIGN_PHASE2.md` §4.)*
 - **Resource requests and limits are still placeholders** (§18). A real run has
   not produced numbers.
 - **Single-broker, ephemeral storage.** A broker restart loses all topic data, and
@@ -824,7 +827,7 @@ That is RQ-1 behaving exactly as designed, observed by accident.
 ## 9. Signing: the cost of the pure-Python rule
 
 §1.1 bans C extensions, which rules out `cryptography`, so Ed25519 is implemented
-from RFC 8032 in `eventrunner/signing.py` (~120 lines) and checked against the
+from RFC 8032 in `shared/signing.py` (~120 lines) and checked against the
 RFC's own test vectors — all three pass for key derivation, signing and
 verification, plus tamper, wrong-key, malformed-input and `alg` confusion cases.
 

eventing/agentdocs/README.md

@@ -18,7 +18,7 @@ and what is *not* verified.
 | [`DESIGN_PHASE1.md`](DESIGN_PHASE1.md) | The Phase 1 design — a delta over Phase 0, not a replacement: KEDA scaling on consumer lag, scale-to-zero, the three cluster findings that shaped it, the §16 gaps (A: rebalance floor, B: ephemeral session state, C: idle replay), §3.2 on supporting a local Kind cluster, and §21 designing agent **groups** — batch fan-out with a tracked fan-in, its own page and exactly two notifications. |
 | [`IMPLEMENTATION_REPORT1.md`](IMPLEMENTATION_REPORT1.md) | What Phase 1 built, the measured results on both clusters, 28 findings including two real bugs only a scale-to-zero deployment could expose, and an honest account of what is still blocked and why. |
 | [`README_PHASE1.md`](README_PHASE1.md) | The Phase 1 runbook in full detail, covering all four ways to run it: locally without containers, under Docker, on a Kind cluster, and on OpenShift. |
-| [`DESIGN_PHASE2.md`](DESIGN_PHASE2.md) | The Phase 2 design — identity on the event path. Why GitHub's opaque user token rules out local verification and forces a `GET /user` lookup plus a load-bearing cache; why `401` and `403` are kept distinct; what `ce_submitter` is and is not worth while it remains unsigned; the `kid` and approved-key-set groundwork for proving *which agent* answered, and the blocker that nothing signs yet. |
+| [`DESIGN_PHASE2.md`](DESIGN_PHASE2.md) | The Phase 2 design — identity on the event path. Why GitHub's opaque user token rules out local verification and forces a `GET /user` lookup plus a load-bearing cache; why `401` and `403` are kept distinct; what `ce_submitter` is and is not worth, and what a signature over it does and does not prove; how the `kid` and the approved-key set prove *which agent* answered, why group events are pinned to EventBridge's own key, and why a rejected response is stored as an error rather than dropped. |
 
 ## Reading order
 

eventing/eventbridge/__main__.py

@@ -21,6 +21,7 @@
 from eventbridge.openapi import spec
 from eventbridge.router import Dispatcher
 from eventbridge.store import Store
+from shared import keyset, signing
 from shared.pidfile import PidFile
 
 
@@ -46,10 +47,40 @@ def main() -> int:
     for corr in store.all_correlations(limit=10000):
         minter.remember(corr)
 
+    # §11 — key material is loaded ONCE, here, and deliberately not caught. A bridge
+    # that believes it is signing but is not fails silently; one that will not start
+    # says so in `kubectl logs` before it accepts a request.
+    seed = None
+    if cfg.signing_key_path:
+        seed = signing.load_seed(cfg.signing_key_path)
+        print(f"[eventbridge] request signing ON as kid={cfg.signing_kid or '(unnamed)'} "
+              f"(seed {cfg.signing_key_path})")
+    else:
+        print("[eventbridge] request signing OFF (set EB_SIGNING_KEY_PATH to enable)")
+
+    # Loaded once, not per record: live reload is deliberately absent (keyset.py), so a
+    # mid-run edit must not silently widen the set of agents this bridge trusts.
+    ks = keyset.load_if_set(cfg.verify_keyset_path)
+    if ks is not None:
+        mode = "ENFORCING" if cfg.require_response_signature else "audit only"
+        print(f"[eventbridge] response verification ON ({mode}) — {len(ks)} kid(s): "
+              f"{', '.join(ks.kids)}")
+        if not cfg.signing_kid:
+            # Without a bridge kid there is nothing to compare a group event against,
+            # so any approved runner could forge one. Refuse rather than accept-any.
+            raise SystemExit(
+                "[eventbridge] EB_VERIFY_KEYSET_PATH is set but EB_SIGNING_KID is not: "
+                "group lifecycle events could not be attributed to this bridge. "
+                "Set EB_SIGNING_KID to the kid naming EventBridge's key.")
+    else:
+        print("[eventbridge] response verification OFF "
+              "(set EB_VERIFY_KEYSET_PATH to enable)")
+
     # The producer needs the RESPONSES topic too: group lifecycle events go there,
     # not on requests, because EventRunner would try to execute anything on requests.
     producer = Producer(cfg.kafka_bootstrap, cfg.request_topic, cfg.source_uri,
-                        response_topic=cfg.response_topic)
+                        response_topic=cfg.response_topic,
+                        seed=seed, kid=cfg.signing_kid or None)
     groups = GroupService(cfg, store, producer, minter)
 
     ntfy = NtfyPublisher(cfg.ntfy, cfg.public_base_url, store=store)
@@ -59,7 +90,10 @@ def main() -> int:
     consumer = Consumer(cfg.kafka_bootstrap, cfg.response_topic, store,
                         on_event=ntfy.submit,
                         on_group_event=groups.on_group_event,
-                        on_member_event=groups.on_member_event)
+                        on_member_event=groups.on_member_event,
+                        keyset=ks,
+                        require_signature=cfg.require_response_signature,
+                        bridge_kid=cfg.signing_kid or None)
     consumer.start()
 
     # Back-fill prompts from the requests topic — also gives us prompt visibility

eventing/eventbridge/auth.py

@@ -18,12 +18,21 @@
   onto the request event as `ce_submitter`, which is the whole point — a `401`
   tells you nothing after the fact, a recorded submitter does.
 
-What this is NOT: the submitter attribute is **unsigned**. Anyone who can write
-to the Kafka `requests` topic can forge it, and the broker is plaintext. The
-honest claim is "EventBridge refuses unauthenticated submissions and records who
-it believes submitted this" — not "this event proves who submitted it." Proving
-it needs `submitter` inside `signing.SIGNED_ATTRS` and a producer that actually
-signs (today nothing does; see agentdocs/IMPLEMENTATION_REPORT1.md §811-814).
+What this is NOT, and the two limits that still apply. `submitter` is now inside
+`signing.SIGNED_ATTRS` and EventBridge signs the requests it publishes, so when
+signing is configured the attribute cannot be altered in flight without invalidating
+the signature. But:
+
+* **Signing is opt-in.** With no `EB_SIGNING_KEY_PATH` the events are unsigned, the
+  broker is plaintext, and anyone who can write to the `requests` topic can forge a
+  submitter. The claim only holds where verification is actually enabled.
+* **A signature proves the assertion, not the identity.** It shows EventBridge said
+  this, not that the name is real. A static `EB_AUTH_TOKENS` entry is a name an
+  operator typed into an env var; `ce_submitteriss` is what distinguishes it from a
+  login GitHub verified.
+
+So the honest claim is "EventBridge refuses unauthenticated submissions, records who
+it believes submitted, and — when signing is on — makes that record tamper-evident."
 """
 from __future__ import annotations
 

eventing/eventbridge/config.py

@@ -58,6 +58,25 @@ class Cfg:
     # GitHub API call per request would spend a 5000/hour budget and add GitHub's
     # latency to the request path.
     github_cache_ttl_s: float = 300.0
+    # §11 — signing the requests and group events EventBridge publishes. Empty
+    # disables it, which is the default: a key is a capability, and the demo has to
+    # work with none. Env only, never config.toml — the file it points at is a
+    # Secret mount (same rule as auth_tokens and NTFY_TOKEN above).
+    signing_key_path: str = ""
+    # The kid written into the protected header, naming EventBridge's own key in the
+    # approved set. Also the kid group lifecycle events must be signed by: the set is
+    # otherwise flat, so without this any approved runner could forge a
+    # group.completed and end a batch early.
+    signing_kid: str = ""
+    # §11 — the approved-key set used to verify responses coming back off the
+    # responses topic. Empty disables verification entirely. A ConfigMap path, not a
+    # Secret: only public keys belong in it.
+    verify_keyset_path: str = ""
+    # Whether a failed verification is ENFORCED. With a keyset but this false,
+    # EventBridge verifies and logs but stores the event unchanged — audit mode.
+    # Enforcement rewrites persisted rows and raises a priority-5 notification, so
+    # there has to be a way to watch the reject rate before turning it on.
+    require_response_signature: bool = False
     ntfy: NtfyCfg = field(default_factory=NtfyCfg)
 
 
@@ -120,6 +139,15 @@ def load() -> Cfg:
         cfg.allowed_users = ghauth.parse_allowed_users(allowed_env)
     cfg.github_cache_ttl_s = float(e("EB_GITHUB_CACHE_TTL_S",
                                      str(cfg.github_cache_ttl_s)))
+    # §11. Env only: the seed path names a Secret mount, and keeping the whole
+    # signing block in one layer means it cannot be half-configured from a
+    # committed file.
+    cfg.signing_key_path    = e("EB_SIGNING_KEY_PATH",    cfg.signing_key_path)
+    cfg.signing_kid         = e("EB_SIGNING_KID",         cfg.signing_kid)
+    cfg.verify_keyset_path  = e("EB_VERIFY_KEYSET_PATH",  cfg.verify_keyset_path)
+    cfg.require_response_signature = (
+        e("EB_REQUIRE_RESPONSE_SIGNATURE",
+          "true" if cfg.require_response_signature else "false").lower() == "true")
 
     cfg.ntfy.enabled  = (e("NTFY_ENABLED", "true" if cfg.ntfy.enabled else "false").lower() == "true")
     cfg.ntfy.base_url = e("NTFY_BASE_URL", cfg.ntfy.base_url)