From 91c43cae52bde50fe1638044250239a74d4b7f3d Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Thu, 10 Sep 2026 12:09:52 -0400 Subject: [PATCH 1/2] Start one icons server per environment, and register it only once it answers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In an env-mode CI stack start-icons.sh ran twice for the same environment: once as the listed services:icons task and once spawned by services:realm-server, which starts it so that dev-all — whose run-p list has no services:icons — still gets an icons server. Each run picked a free port, backgrounded http-server, and wrote the environment's one icons route file straight away, so the second run's port replaced the first's whether or not its server ever came up. In one host shard the second http-server never did: the route pointed at a port nothing listened on, Traefik answered 502 for every icon without CORS headers, the standby pages could not import an icon module, and every base render timed out idle for the rest of the run while the first server sat unused after wait-on's single probe. The script now reads the port from any existing route file and steps aside when that server answers a dist file, so whichever task starts second leaves one server registered. It registers only after its own http-server answers, and exits non-zero without writing a route if the server dies or never answers, so a route can no longer name a dead upstream. The probe goes to the port directly, as Traefik does, rather than through the hostname, so it depends on nothing but the server being up. When the server later exits, the script removes the route it wrote, so Traefik answers 404 for a missing route rather than 502 for a dead one and the next start registers afresh instead of stepping aside. The route directory is resolved once in the shell and handed to the registration script instead of being re-derived there. Co-Authored-By: Claude Fable 5.1 --- mise-tasks/services/realm-server | 4 + packages/realm-server/scripts/start-icons.sh | 86 ++++++++++++++++---- 2 files changed, 73 insertions(+), 17 deletions(-) diff --git a/mise-tasks/services/realm-server b/mise-tasks/services/realm-server index 54b15eb13a0..d7fc3d19f21 100755 --- a/mise-tasks/services/realm-server +++ b/mise-tasks/services/realm-server @@ -15,6 +15,10 @@ SCRIPTS_DIR="./scripts" # loki.source.file. Alloy's Docker discovery doesn't see native processes. LOG_TEE="../observability/scripts/dev-log-tee.sh" +# Started here so that dev-all, which does not list services:icons, still has +# an icons server. The stacks that do list it start one as well; start-icons.sh +# steps aside when icons already answer for this environment, so either order +# leaves one server registered. sh "$SCRIPTS_DIR/start-icons.sh" & ICONS_PID=$! cleanup_icons_server() { diff --git a/packages/realm-server/scripts/start-icons.sh b/packages/realm-server/scripts/start-icons.sh index 98bb16f1a51..9fc7f187355 100644 --- a/packages/realm-server/scripts/start-icons.sh +++ b/packages/realm-server/scripts/start-icons.sh @@ -4,37 +4,78 @@ SCRIPTS_DIR="$(cd "$(dirname "$0")" && pwd)" . "$SCRIPTS_DIR/../../../scripts/env-slug.sh" if [ -n "$BOXEL_ENVIRONMENT" ]; then - # In environment mode, use port 0 (dynamic) and register with Traefik. - # http-server doesn't support port 0, so we pick a free port ourselves. + ENV_SLUG=$(resolve_env_slug) + # A file from the dist rather than `/`: a 200 for it means an icons server + # is answering, not merely something on the port. + PROBE_PATH="/@cardstack/boxel-icons/v1/icons/folder-pen.js" + + # The route file lives in the directory the running Traefik container + # watches, which in a worktree can differ from this checkout's own + # traefik/dynamic (see scripts/start-traefik.sh). Resolved once, here, and + # handed to the registration below so the check and the write agree. + DYNAMIC_DIR=$(docker inspect boxel-traefik --format '{{range .Mounts}}{{if eq .Destination "/etc/traefik/dynamic"}}{{.Source}}{{end}}{{end}}' 2>/dev/null) + [ -n "$DYNAMIC_DIR" ] || DYNAMIC_DIR="$SCRIPTS_DIR/../../../traefik/dynamic" + CONFIG_PATH="$DYNAMIC_DIR/${ENV_SLUG}-icons.yml" + + # More than one task starts this script for the same environment: + # services:icons on its own, and services:realm-server, which spawns it so + # that dev-all — which does not list services:icons — still gets an icons + # server. Each start registers its own port under this one route file, last + # writer winning, so a second instance that failed to come up would take the + # route away from a first that was serving. An instance that finds the + # registered server answering steps aside. The check goes straight to the + # port the route names, the way Traefik reaches it, rather than through the + # hostname: that depends on nothing but the server being up — not on DNS, + # the mkcert trust, or Traefik having finished a reload. + if [ -f "$CONFIG_PATH" ]; then + REGISTERED_PORT=$(sed -n 's/.*host\.docker\.internal:\([0-9]*\).*/\1/p' "$CONFIG_PATH" | head -1) + if [ -n "$REGISTERED_PORT" ] && curl --fail --silent --max-time 2 "http://127.0.0.1:${REGISTERED_PORT}${PROBE_PATH}" >/dev/null 2>&1; then + echo "icons already served for icons.${ENV_SLUG}.localhost on port ${REGISTERED_PORT}, skipping startup" + exit 0 + fi + fi + + # http-server doesn't support port 0, so pick a free port here. The port is + # found by binding it and letting it go, so http-server's own bind can still + # fail; the wait below is what makes that safe. ICONS_PORT=$(node -e 'const s=require("net").createServer();s.listen(0,()=>{console.log(s.address().port);s.close();})') echo "Starting icons server on dynamic port ${ICONS_PORT}" cd "$(dirname "$0")/../../boxel-icons" && npx http-server --cors=Origin,X-Requested-With,Content-Type,Accept,Range,Authorization,X-Boxel-Assume-User --port "${ICONS_PORT}" dist & ICONS_PID=$! + # Register only a server that answers. http-server binds after this script + # has moved on, and a route written before it does would point Traefik at + # the port whether or not anything ever listens there; every icon request + # would then fail with a 502 that no service log records, and the host app + # cannot render a card without its icons. + tries=0 + until curl --fail --silent --max-time 2 "http://127.0.0.1:${ICONS_PORT}${PROBE_PATH}" >/dev/null 2>&1; do + if ! kill -0 "$ICONS_PID" 2>/dev/null; then + echo "icons server exited before it answered on port ${ICONS_PORT}; not registering it" >&2 + exit 1 + fi + tries=$((tries + 1)) + if [ "$tries" -ge 60 ]; then + echo "icons server did not answer on port ${ICONS_PORT} within 30s; not registering it" >&2 + kill "$ICONS_PID" 2>/dev/null + exit 1 + fi + sleep 0.5 + done + # Register icons service with Traefik via a small node script. # Mirrors dev-service-registry.ts: a `websecure` router terminates TLS # at Traefik (mkcert leaf) and a sibling `-http` router on :80 # 308-redirects to https. The host bundle is loaded over https, so an # `http://icons..localhost/...` upstream would be mixed-content # blocked AND fail the CORS preflight on the redirect. - ENV_SLUG=$(resolve_env_slug) - node -e " + CONFIG_PATH="$CONFIG_PATH" node -e " const fs = require('fs'); - const path = require('path'); - const { execSync, spawn } = require('child_process'); - let dir; - try { - const mounted = execSync( - \"docker inspect boxel-traefik --format '{{range .Mounts}}{{if eq .Destination \\\"/etc/traefik/dynamic\\\"}}{{.Source}}{{end}}{{end}}'\", - { encoding: 'utf-8' }, - ).trim(); - if (mounted) dir = mounted; - } catch {} - if (!dir) dir = path.resolve(__dirname, '..', '..', 'traefik', 'dynamic'); + const { spawn } = require('child_process'); const slug = '${ENV_SLUG}'; const routerKey = 'icons-' + slug; const redirectMiddleware = routerKey + '-https-redirect'; - const configPath = path.join(dir, slug + '-icons.yml'); + const configPath = process.env.CONFIG_PATH; const entry = [ 'http:', ' routers:', @@ -66,7 +107,6 @@ if [ -n "$BOXEL_ENVIRONMENT" ]; then const tmp = configPath + '.tmp'; fs.writeFileSync(tmp, entry, 'utf-8'); fs.renameSync(tmp, configPath); - console.log('Registered icons at icons.${ENV_SLUG}.localhost -> localhost:${ICONS_PORT}'); // Bounce Traefik on macOS — Docker Desktop's bind mounts don't // propagate inotify, and Traefik v3 file provider has no polling // option. See dev-service-registry.ts for the full rationale. @@ -78,8 +118,20 @@ if [ -n "$BOXEL_ENVIRONMENT" ]; then child.unref(); } " + echo "Registered icons at icons.${ENV_SLUG}.localhost -> localhost:${ICONS_PORT}" + # The route is right only while this server is up. When it exits, take the + # route with it — if it still names this port — so Traefik answers 404 for a + # missing route rather than 502 for a dead upstream, and the next start of + # any icons task registers afresh instead of stepping aside for a route that + # only looks live. wait $ICONS_PID + ICONS_STATUS=$? + if grep -qF "host.docker.internal:${ICONS_PORT}\"" "$CONFIG_PATH" 2>/dev/null; then + rm -f "$CONFIG_PATH" + echo "icons server on port ${ICONS_PORT} exited (status ${ICONS_STATUS}); removed its route" >&2 + fi + exit $ICONS_STATUS else if curl --fail --silent --show-error http://localhost:4206 >/dev/null 2>&1; then echo "icons server already running on http://localhost:4206, skipping startup" From 13ec61aca690b835597c0584ebb0d961e8538173 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Thu, 10 Sep 2026 12:22:17 -0400 Subject: [PATCH 2/2] Stand by for the registered icons server instead of stepping aside A start that found the registered icons server answering exited, on the strength of one probe. When a stack starts while the previous one is still shutting down, that probe finds the old server alive; both of the new stack's starts exit, the old server then goes away and takes its route with it, and the new environment has no icons server for wait-on to ever see. The instance now stands by instead: it keeps probing the registered port and, the moment that server stops answering, starts and registers a server of its own. The environment therefore never ends with a route and nothing behind it, whichever stack the answering server belonged to. Co-Authored-By: Claude Fable 5.1 --- packages/realm-server/scripts/start-icons.sh | 28 ++++++++++++++------ 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/packages/realm-server/scripts/start-icons.sh b/packages/realm-server/scripts/start-icons.sh index 9fc7f187355..8b8db24a8be 100644 --- a/packages/realm-server/scripts/start-icons.sh +++ b/packages/realm-server/scripts/start-icons.sh @@ -8,6 +8,9 @@ if [ -n "$BOXEL_ENVIRONMENT" ]; then # A file from the dist rather than `/`: a 200 for it means an icons server # is answering, not merely something on the port. PROBE_PATH="/@cardstack/boxel-icons/v1/icons/folder-pen.js" + answers() { + curl --fail --silent --max-time 2 "http://127.0.0.1:$1${PROBE_PATH}" >/dev/null 2>&1 + } # The route file lives in the directory the running Traefik container # watches, which in a worktree can differ from this checkout's own @@ -23,15 +26,24 @@ if [ -n "$BOXEL_ENVIRONMENT" ]; then # server. Each start registers its own port under this one route file, last # writer winning, so a second instance that failed to come up would take the # route away from a first that was serving. An instance that finds the - # registered server answering steps aside. The check goes straight to the - # port the route names, the way Traefik reaches it, rather than through the - # hostname: that depends on nothing but the server being up — not on DNS, - # the mkcert trust, or Traefik having finished a reload. + # registered server answering stands by instead. The check goes straight to + # the port the route names, the way Traefik reaches it, rather than through + # the hostname: that depends on nothing but the server being up — not on + # DNS, the mkcert trust, or Traefik having finished a reload. + # + # Standing by rather than exiting: the server just probed may belong to a + # stack that is on its way down — a restart overlaps the previous stack's + # shutdown — and a single probe cannot tell. This instance keeps probing and + # starts a server of its own the moment the registered one stops answering, + # so the environment never ends up with a route and nothing behind it. if [ -f "$CONFIG_PATH" ]; then REGISTERED_PORT=$(sed -n 's/.*host\.docker\.internal:\([0-9]*\).*/\1/p' "$CONFIG_PATH" | head -1) - if [ -n "$REGISTERED_PORT" ] && curl --fail --silent --max-time 2 "http://127.0.0.1:${REGISTERED_PORT}${PROBE_PATH}" >/dev/null 2>&1; then - echo "icons already served for icons.${ENV_SLUG}.localhost on port ${REGISTERED_PORT}, skipping startup" - exit 0 + if [ -n "$REGISTERED_PORT" ] && answers "$REGISTERED_PORT"; then + echo "icons already served for icons.${ENV_SLUG}.localhost on port ${REGISTERED_PORT}; standing by to replace it" + while answers "$REGISTERED_PORT"; do + sleep 5 + done + echo "icons server on port ${REGISTERED_PORT} stopped answering; starting a replacement" fi fi @@ -49,7 +61,7 @@ if [ -n "$BOXEL_ENVIRONMENT" ]; then # would then fail with a 502 that no service log records, and the host app # cannot render a card without its icons. tries=0 - until curl --fail --silent --max-time 2 "http://127.0.0.1:${ICONS_PORT}${PROBE_PATH}" >/dev/null 2>&1; do + until answers "$ICONS_PORT"; do if ! kill -0 "$ICONS_PID" 2>/dev/null; then echo "icons server exited before it answered on port ${ICONS_PORT}; not registering it" >&2 exit 1