Skip to content

Durability: catch up on missed mail at startup (pull from Mailgun storage) #2

Description

@brocla

Problem

FileMill's inbound path is push-only: it processes an email only when Mailgun's route action POSTs it to the webhook. There is no code that pulls from Mailgun. So if FileMill (or the Cloudflare tunnel) is down when an email arrives, the message is not fetched on the next startup — FileMill has no "catch up" mechanism.

The only current safety net is Mailgun's retry of the failed forward(url) POST, which covers brief outages but has a finite window. FileMill runs on a personal laptop that sleeps, reboots, and only starts at logon, so "down when mail arrives" is a recurring situation, not an edge case. An overnight/weekend outage past Mailgun's retry window means the message is lost to FileMill (it only survives in the sender's Sent folder).

If store() is enabled on the route, Mailgun retains the raw message ~3 days, but FileMill has no code to enumerate or fetch stored messages, so that copy just expires unused.

Proposed solution

Turn Mailgun storage into a durable inbox that FileMill drains:

  1. Switch (or add) the route action to store(notify="<webhook URL>") for ~3-day retention (keep forward(url) too if we still want the low-latency push path).
  2. Add a FileMill pull: on startup and on a timer, query Mailgun's Messages API for stored messages, fetch any not yet processed, and submit them.
  3. Dedupe by Message-Id — the idempotency layer already handles this for free (App.BeginEmail keyed on Message-Id + App.EmailHasJob), so a message that arrived by both push and pull won't be processed twice.

Why it matters

This is the highest-value durability feature for an intermittently-available service: it makes "a message was waiting on Mailgun when FileMill started" actually get picked up — behavior the current push-only design cannot provide. Pairs with an external heartbeat (separate item) for full "is the service alive / did it miss anything" coverage.

Notes / open questions

  • Requires the private API key (already configured as MAILGUN_API_KEY) to call the Messages API and fetch stored message content + attachments.
  • Attachments in stored messages are referenced by URL, so this also covers the "fetch attachments from storage" gap (relevant if we ever move off forward(url), which inlines them).
  • Decide the pull cadence (startup + every N minutes) and how far back to scan (Mailgun retains ~3 days).
  • Confirm Mailgun's actual forward(url) retry window empirically to know how large the gap really is.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions