fix(cloud-connection): an install-local uninstall withdraws the package's flows from the automation engine and unbinds their triggers (#27395)

.changeset/27395-install-local-uninstall-flow-resync.md

@@ -0,0 +1,14 @@
+---
+'@objectstack/cloud-connection': patch
+---
+
+fix(cloud-connection): an install-local uninstall withdraws the package's flows from the automation engine at once, and unbinds their triggers
+
+Clause-②: no
+Metadata-protocol: none
+
+`DELETE /api/v1/marketplace/install-local/:manifestId` withdrew the package from the registry and ran its uninstall cleanups, but never told the running kernel. The automation engine withdraws a flow only when it re-syncs on `metadata:reloaded`, so the uninstalled package's flows stayed registered: `GET /api/v1/meta/flow/NAME` answered 404 while `GET /api/v1/automation/NAME` still answered `active`, `POST /api/v1/automation/NAME/trigger` still ran the flow and wrote a run row, and its record and schedule triggers stayed armed. That lasted until some unrelated reload, such as a view publish or another package's install. The door's note said the package was withdrawn from the running kernel.
+
+- **Now.** After the registry withdrawal and the cleanups, the door announces `metadata:reloaded` with `{ changed: ['app/MANIFEST_ID'] }`, the same signal the install door's hot load announces. The engine's re-sync withdraws every flow it had armed that the package no longer provides and unbinds each one's trigger. Right after the `DELETE` returns, `GET /api/v1/automation/NAME` and its trigger door answer 404, and a write to a watched object runs nothing.
+- **When the re-sync does not run.** If a subscriber throws, or the kernel context cannot announce, the uninstall still answers 200, and `cleanups` carries one failed outcome named `metadata:reloaded`. The note then says the engine can still hold the package's flows until the next restart. The cause goes to the operator log at `warn`, with the restart as the remedy.
+- **Unchanged.** Nothing is announced when the uninstall did not happen (a refused caller, an id this door never installed, a ledger write that failed), or when the registry refused the withdrawal, because the package is still registered.

packages/cloud-connection/src/marketplace-install-local-plugin.ts

@@ -111,7 +111,10 @@
  *            serving them at once too (#24070), then runs the uninstall
  *            cleanups domain plugins registered with the protocol (#21490) —
  *            the package's permission sets and their grants go with it — and
- *            reports each outcome as `cleanups`, a refused withdrawal included.
+ *            last announces the withdrawal as `metadata:reloaded`, so the
+ *            automation engine withdraws the package's flows and unbinds their
+ *            triggers at once (#27395); it reports each outcome as `cleanups`,
+ *            a refused withdrawal or a re-sync that did not run included.
  *
  * Persistence layout:
  *   <cwd>/.objectstack/installed-packages/<safe-manifest-id>.json
@@ -303,6 +306,13 @@ const REGISTRY_WITHDRAWAL = 'registry.uninstallPackage';
  */
 const OBJECT_COPY_WITHDRAWAL = 'metadata.unregister';
 
+/**
+ * [#27395] The outcome name this door reports when the running kernel was not
+ * told of the withdrawal — named for the event it announces, the re-sync the
+ * automation engine withdraws an uninstalled package's flows on.
+ */
+const KERNEL_RESYNC = 'metadata:reloaded';
+
 /**
  * [#8976] The capability every MUTATING install-local route demands.
  *
@@ -2291,13 +2301,28 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
             ...(copies ? [copies] : []),
             ...await this.runUninstallCleanups(ctx, manifestId, admission.userId, declared),
         ];
-        const withdrawals = (withdrawal ? 1 : 0) + (copies ? 1 : 0);
+        // [#27395] LAST — after the registry withdrawal and the cleanups, never
+        // before, for the window described above — announce the withdrawal as
+        // `metadata:reloaded`, the re-sync the install's hot load announces, so
+        // the consumers that hold what they read from a package re-read it now
+        // that it is gone. ⛔ Nothing here withdraws a flow or unbinds a trigger
+        // itself: the uninstall drives the one teardown a reload drives. Not
+        // when the registry refused: the package is still registered, so a
+        // re-sync would only read it back.
+        const resync = withdrawal ? undefined : await this.announceUninstall(ctx, manifestId);
+        if (resync) cleanups.push(resync);
+        const withdrawals = (withdrawal ? 1 : 0) + (copies ? 1 : 0) + (resync ? 1 : 0);
         ctx.logger?.info?.(
             `[MarketplaceInstallLocal] uninstalled ${manifestId} (cached manifest removed; `
             + (withdrawal
                 ? 'the running kernel did NOT withdraw it, so it stays loaded until the next restart; '
-                : copies
-                    ? 'withdrawn from the running kernel\'s registry, but the metadata service kept copies of its objects until the next restart; '
+                : copies || resync
+                    ? 'withdrawn from the running kernel\'s registry, but '
+                        + [
+                            ...(copies ? ['the metadata service kept copies of its objects'] : []),
+                            ...(resync ? ['the automation engine can still hold its flows'] : []),
+                        ].join(' and ')
+                        + ' until the next restart; '
                     : 'withdrawn from the running kernel; ')
             + `${cleanups.length - withdrawals} uninstall cleanup(s) ran)`,
         );
@@ -2308,13 +2333,81 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
                 cleanups,
                 note: withdrawal
                     ? 'Cached manifest removed, and the uninstall cleanups this runtime\'s plugins registered ran — each one\'s outcome is in `cleanups`. The running kernel did not withdraw the package, so it stays loaded until the next restart; that is the `registry.uninstallPackage` entry in `cleanups`.'
-                    : copies
-                        ? 'Cached manifest removed, the package withdrawn from the running kernel\'s registry, and the uninstall cleanups this runtime\'s plugins registered ran — each one\'s outcome is in `cleanups`. The metadata service kept copies of the package\'s objects, so the object listing and reads keep serving them until the next restart; that is the `metadata.unregister` entry in `cleanups`.'
+                    : copies || resync
+                        ? 'Cached manifest removed, the package withdrawn from the running kernel\'s registry, and the uninstall cleanups this runtime\'s plugins registered ran — each one\'s outcome is in `cleanups`.'
+                            + (copies
+                                ? ' The metadata service kept copies of the package\'s objects, so the object listing and reads keep serving them until the next restart; that is the `metadata.unregister` entry in `cleanups`.'
+                                : '')
+                            + (resync
+                                ? ' The running kernel\'s re-sync did not run, so the automation engine can still hold the package\'s flows — triggerable by name, their triggers armed — until the next restart; that is the `metadata:reloaded` entry in `cleanups`.'
+                                : '')
                         : 'Cached manifest removed, the package withdrawn from the running kernel, and the uninstall cleanups this runtime\'s plugins registered ran — each one\'s outcome is in `cleanups`.',
             },
         }, 200);
     };
 
+    /**
+     * [#27395] Announce an uninstall this door just completed to the running
+     * kernel as `metadata:reloaded` — the re-sync signal
+     * {@link announceHotInstall} announces for an install, with the same
+     * payload — so every consumer that holds what it read from a package
+     * re-reads a kernel that no longer has it:
+     *
+     *   - `service-automation` re-syncs its flows from the protocol
+     *     (`resyncFlowsFromProtocol`) and withdraws each flow it armed that the
+     *     protocol's view no longer lists — the uninstalled package's, which
+     *     the registry withdrawal took out of that view — and `withdrawFlow`
+     *     unbinds each one's record or schedule trigger in the same step.
+     *
+     * Before this, the door withdrew the package from the registry and the
+     * engine never heard of it: `GET /meta/flow/NAME` answered 404 while
+     * `GET /automation/NAME` still answered `active`, and the trigger door, a
+     * record write or a schedule still ran the uninstalled package's flow
+     * until an unrelated reload — a view publish, another package's install —
+     * cleared it. ⛔ No engine-side list of the package's flows is kept or
+     * withdrawn here: a second teardown is how the two would drift.
+     *
+     * Never throws: the uninstall has already happened. Two answers:
+     *   - `undefined` — announced, and every subscriber returned;
+     *   - one failed outcome named {@link KERNEL_RESYNC}, when the context
+     *     cannot announce or a subscriber threw. It rides on `cleanups`, the
+     *     way a refused registry withdrawal does, and the operator log says
+     *     what it costs and the remedy; the cause is logged, never put on the
+     *     wire. A FUNCTIONAL degradation, so `warn` — the level the install's
+     *     announce failure uses.
+     */
+    private announceUninstall = async (
+        ctx: PluginContext,
+        manifestId: string,
+    ): Promise<UninstallCleanupOutcome | undefined> => {
+        const failed: UninstallCleanupOutcome = {
+            name: KERNEL_RESYNC,
+            success: false,
+            removed: 0,
+            error: 'the running kernel did not re-sync — the automation engine can still hold the package\'s flows until the next restart',
+        };
+        const cost = 'its flows can stay registered in the automation engine — triggerable by name, their record and '
+            + 'schedule triggers still armed — until the runtime restarts. The outcome is on the response (`cleanups`). '
+            + 'Remedy: restart the runtime — the ledger entry is already gone, so the restart does not bring the package back.';
+        if (typeof ctx.trigger !== 'function') {
+            ctx.logger?.warn?.(
+                `[MarketplaceInstallLocal] uninstalled ${manifestId}, but this kernel context cannot announce `
+                + `metadata:reloaded — ${cost}`,
+            );
+            return failed;
+        }
+        try {
+            await ctx.trigger('metadata:reloaded', { changed: [`app/${manifestId}`] });
+            return undefined;
+        } catch (err: any) {
+            ctx.logger?.warn?.(
+                `[MarketplaceInstallLocal] uninstalled ${manifestId}, but the metadata:reloaded re-sync FAILED `
+                + `(${err?.message ?? err}) — ${cost}`,
+            );
+            return failed;
+        }
+    };
+
     /**
      * [#21576] Withdraw a package this door just removed from its ledger from
      * the running kernel — through `SchemaRegistry.uninstallPackage`, the ONE

packages/cloud-connection/src/marketplace-install-local-uninstall-cleanups.test.ts

@@ -136,6 +136,9 @@ async function bootPlugin(protocolMode: ProtocolMode = { kind: 'runner' }, opts:
             return svc;
         },
         logger,
+        // Every kernel context can announce; the uninstall's `metadata:reloaded`
+        // (#27395) is pinned in the resync file, so here it reaches no subscriber.
+        trigger: async () => undefined,
     };
     const plugin = new MarketplaceInstallLocalPlugin({ controlPlaneUrl: 'off', storageDir: dir });
     await plugin.start(ctx);

packages/cloud-connection/src/marketplace-install-local-uninstall-meta-reads.test.ts

@@ -187,6 +187,9 @@ async function bootPlugin(opts: { refuseUnregister?: string; before?: (metadata:
             return services[name];
         },
         logger,
+        // Every kernel context can announce; the uninstall's `metadata:reloaded`
+        // (#27395) is pinned in the resync file, so here it reaches no subscriber.
+        trigger: async () => undefined,
     };
     const plugin = new MarketplaceInstallLocalPlugin({ controlPlaneUrl: 'off', storageDir: dir });
     await plugin.start(ctx as never);