Reliability: auto-restart FileMill on crash (supervised Windows service)

[brocla]

Backlog: Reliability & durability

Problem

FileMill is started by a scheduled task at logon (scripts/Install-FileMillScheduledTask.ps1 → Start-FileMill.ps1). That covers reboots and logins, but a scheduled task is a launcher, not a supervisor: if the filemill process crashes, it stays down until the next logon. Today's resilience is "survives reboot, does not survive a crash." For an always-on service this is the biggest gap.

Proposed solution

Run FileMill under a supervisor configured to restart it, easiest first:

1. Wrap as a Windows Service with NSSM or WinSW (recommended) — turns the console app into a service with a "restart on exit/failure" recovery policy, no code change. This is exactly how cloudflared (the tunnel) already runs on the same box, so there's precedent.
2. Add restart-on-failure to the scheduled task — Task Scheduler Settings → "If the task fails, restart every N minutes." Weaker, zero new tooling.
3. Native Windows service (handle service control messages in Go) — most work, least benefit over NSSM/WinSW.

Notes

• Auto-restart is one leg of a three-legged stool: restart (this) + catch-up on missed mail (Durability: catch up on missed mail at startup (pull from Mailgun storage) #2) + heartbeat (separate issue). The crash+restart gap loses mail unless Durability: catch up on missed mail at startup (pull from Mailgun storage) #2 exists.
• Recommended order across the three: this first (biggest gap, least effort), then Durability: catch up on missed mail at startup (pull from Mailgun storage) #2, then heartbeat.

[brocla]

Implementation plan (chosen: Option A — supervisor loop)

Decisions made: Option A, crash-loop log-only for now (upgrade to alert with #7), backoff immediate first retry then escalate. Mailgun env vars confirmed Machine (System) scope, so the supervisor inherits them.

Constraints

• Runs as the user account (transformers need the user's venv/Python + locally installed deps).
• Working directory = repo root (FileMill uses os.Getwd() for config/ and data/).
• cloudflared already runs as an auto-restarting service — precedent for the future Option B.

Design

scripts\Supervise-FileMill.ps1 runs filemill run in the foreground and restarts it on crash:

• Immediate first retry, then escalating backoff (5s/15s/30s/60s/120s, capped).
• A run ≥ 60s is "healthy" and resets the backoff.
• Clean exit (code 0 = intentional Ctrl+C / shutdown) stops the supervisor.
• Crash-loop logged only (alerting → Ops: email operator error alerts to support@keywind.cc #7). Supervisor log: data\logs\supervisor.log.

The logon scheduled task launches the supervisor (hidden window) with a task-level restart backstop; uninstall stops the supervisor then the worker.

Interactions

• Pairs with Durability: catch up on missed mail at startup (pull from Mailgun storage) #2 (catch-up) — the crash→restart gap loses mail without it.
• Pairs with Reliability: external heartbeat / dead-man's-switch for total-down detection #5 (heartbeat) / Ops: email operator error alerts to support@keywind.cc #7 (alerting) — a persistent crash-loop should eventually alert.

Not done here

• Option B (real Windows service via WinSW/NSSM) — deferred; would add survive-logoff and stronger supervision.

Implemented in the linked PR.