ci: probe production health off-host on a schedule

[aterga]

Summary

Item 8 of the rollout plan — "Set up alerts based on the status page".

The status dashboard already runs every check this needs (MCP, the full OAuth suite, TLS, and the Internet Identity linkage). But it runs on the host and it is pull-only: something has to look at it. A host that is down cannot tell you it is down, and that is both the failure mode that matters most and the one the current setup structurally cannot report.

This runs the same suite from GitHub's infrastructure every fifteen minutes. No new service, no new vendor, no dependencies — monitoring/mcp-status has none, so the probe is a checkout and a node invocation.

Related issues

Complements the hosted dashboard rather than replacing it. The longer-term answer is the organisation's Prometheus/OpsGenie stack (dfinity-ops/k8s), which is a separate conversation with #eng-observability.

Changes

• .github/workflows/health.yml — a scheduled probe job.

Design decisions worth review:

• Escalation is labelled honestly, not overstated. A failure fails the workflow, which notifies watchers, and posts to Slack once a webhook exists. Neither is a page and nobody is on call. The header comment says so rather than implying coverage that isn't there.
• The Slack step is conditional on the secret being present. The workflow is useful the day it lands, and gains routing to #eng-identity-imcp2 the moment someone adds SLACK_WEBHOOK_URL — no code change. That deliberately keeps a configuration decision from blocking the signal.
• Deliberately not --strict. That flag escalates warnings to failures, and a third of the suite probes Internet Identity and the IC — infrastructure this team does not operate. Paging on someone else's degradation is how a channel learns to ignore its alerts. Without it the CLI exits non-zero exactly when overall is fail, the same threshold the dashboard uses to serve 503.
• The summary names the failing checks, so the reader is not sent on a log dive at the moment they are least inclined to do one.

Testing

Three defects were found by running it rather than reasoning about it, and all three would have shipped a workflow that looked correct:

1. The probe refuses any origin outside its SSRF allow-list. Production is not id.ai, so it must be named explicitly — otherwise the run reports a usage error instead of a health verdict.
2. Report sections carry title, not label. The summary table would have rendered every section as ?.
3. The Slack step was gated on failure(), which never fires here. The probe swallows its own exit code on purpose, so nothing has failed at that point in the job. The notification would have been silently dead while the run still went red and looked correct from outside.

Verified against three inputs:

• Healthy — live production. Exit 0, overall: pass, 18 checks, summary reads "All checks passed."
• Unhealthy — an origin that answers but is not the service. Exit 1, overall: fail, 8 checks fail, summary table renders real section titles and check labels.
• Unreachable — no report produced at all. Graceful message rather than a traceback, and the Slack line says the host may be unreachable.
• YAML parsed and every step's if: asserted by parsing the file, not by eye.
• $GITHUB_OUTPUT value collapsed to one line and length-capped — that file is line-oriented, so a newline in a value could declare further outputs. Static strings today; free to harden now.
• Not run: cargo / npm — no application code changed.

Checklist

• I have read the Contributing guidelines.
• Docs updated — the rationale and the optional secret are documented in the workflow header.
• No secrets, credentials, or internal-only information are included.

Limits

Fifteen minutes is the cadence, but GitHub's scheduler is best-effort and drifts under load, so treat this as noticed within the hour, not an SLA. Scheduled workflows also disable themselves after 60 days without repository activity.

There is no flap suppression: a sustained outage produces one failure per tick. If that proves noisy, transition-detection is the follow-up — I would rather see the real noise level than guess at it now.

To enable Slack: add a repository secret SLACK_WEBHOOK_URL pointing at an incoming webhook for #eng-identity-imcp2. Nothing else changes.

---

Generated by Claude Code