Skip to content

feat(fygaro): signature-failure alerts + bankowner float monitoring - #474

Merged
islandbitcoin merged 7 commits into
mainfrom
feat/fygaro-ops-hardening
Aug 11, 2026
Merged

feat(fygaro): signature-failure alerts + bankowner float monitoring#474
islandbitcoin merged 7 commits into
mainfrom
feat/fygaro-ops-hardening

Conversation

@islandbitcoin

Copy link
Copy Markdown
Contributor

What & why

Two ops-hardening changes that close silent-failure gaps in the live Fygaro card-top-up path.

1. Alert on Fygaro webhook signature failures (gap closed)

verify-signature previously only baseLogger.warned on an HMAC mismatch or a missing/empty secret set. A rotated or wrong webhook secret therefore 401s every real payment while the service looks healthy — this exact gap caused hours of silent card-top-up failures during setup.

Now the two "our side is misconfigured" cases — HMAC mismatch ("we hold a secret but it didn't match") and no secrets configured at all — fire alertBridge (severity warning, source fygaro-webhook, title "Fygaro webhook signature verification failing — check the webhook secret"). A plain missing / malformed / expired signature (random internet noise) stays silent, as before. The alert carries only the public key id, never the secret.

2. Bankowner treasury float monitoring

2a — Proactive scheduled check. A new cron task (checkFygaroTreasuryFloat, registered in src/servers/cron.ts) reads the bankowner treasury USDT balance from IBEX each run (~15 min, matching the reconcile cadence) and fires a warning ("Fygaro treasury float low — top up bankowner") when it drops below a configurable floor. It self-guards on FygaroConfig.enabled and never throws — an IBEX read blip or resolver error logs and returns, so it can't crash the cron (exit 99 → CrashLoopBackOff).

2b — Distinct "float exhausted" alert on credit failure. credit-topup now maps an insufficient-treasury-balance send failure to a distinct step insufficient-treasury-float, and payment.ts raises a distinct critical ("Fygaro treasury float EXHAUSTED — top up bankowner immediately") so ops knows to top up rather than debug a bug. Every other credit failure keeps the generic "manual credit needed" critical. The row still stays Fiat Received; idempotency / record-only / never-double-spend are unchanged.

Rate-limiting via dedup keys

Three new static dedup keys — fygaroSignatureFailure, fygaroFloatLow, fygaroFloatExhausted (no per-request suffix) — so the built-in TTL suppression collapses a flood of failing requests/polls into one alert per window instead of one per event.

Config

New fygaro.float block — { floorUsd: number }, default 2000 (~4× the $500 auto-credit limit) — wired through schema.ts, schema.types.d.ts, and base-config.yaml alongside the existing fygaro config.

How the float balance is read

checkFygaroTreasuryFloat resolves the treasury wallet via getBankOwnerWalletId() (walletId is the IBEX accountId) and reads the balance with Ibex.getAccountDetails(walletId, WalletCurrency.Usdt), converting the returned USDTAmount via .asNumber(). An absent balance (drained/never-funded account) is treated as $0, correctly tripping the floor.

How the insufficient-balance case is detected

By error class, not message: credit-topup checks the send result for InsufficientIbexBalance (the flash IBEX-custodial path — ibex/errors.ts maps the IBEX "insufficient balance" ApiError) or the domain-level InsufficientBalanceError. Matching by class avoids misclassifying a generic send error with coincidental wording. No pre-flight balance read is done — mapping the failure is sufficient and adds no latency (per the task's "cleaner impl" note).

Tests

  • verify-signature: mismatch fires the deduped alert; no-secrets fires it; missing / malformed header and out-of-skew timestamp do not alert; the secret never appears in the payload.
  • float cron: below-floor alerts; at / above floor does not; drained-account (absent balance) → 0 → alerts; IBEX read failure and resolver throw don't crash and don't alert; disabled feature skips; default-floor fallback.
  • credit-topup / payment: an IBEX/domain insufficient-balance failure yields the insufficient-treasury-float step and the distinct float-exhausted critical; other failures keep the generic alert.

yarn tsc-check, yarn eslint-check, and the full yarn test:unit all pass (176 suites, 1513 passed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NcEjF6SS3Ci5D4CzZqdPjb

bobodread876 and others added 7 commits August 10, 2026 21:54
Close two silent-failure gaps in the Fygaro card-top-up path.

1. Signature-failure alerting. verify-signature previously only logged on
   an HMAC mismatch or a missing/empty secret set, so a rotated or wrong
   webhook secret 401s every real payment while the service looks healthy
   (this exact gap caused hours of silent failures during setup). Now those
   two cases fire alertBridge (warning, source fygaro-webhook) under a
   static dedup key so the built-in TTL suppression rate-limits the flood to
   one page per window. Plain missing/malformed/expired signatures (internet
   noise) stay silent. The alert carries only the (public) key id, never the
   secret.

2. Bankowner treasury float monitoring.
   a. A cron task (checkFygaroTreasuryFloat) reads the bankowner treasury
      USDT balance from IBEX each run (~15 min) and warns when it drops below
      a configurable floor (fygaro.float.floorUsd, default $2000 — ~4x the
      $500 auto-credit limit). Self-guards on FygaroConfig.enabled and never
      throws, so an IBEX read blip logs but cannot crash the cron.
   b. credit-topup now maps an insufficient-treasury-balance send failure
      (InsufficientIbexBalance / InsufficientBalanceError, matched by class)
      to a distinct "insufficient-treasury-float" step, and payment.ts raises
      a distinct critical ("float EXHAUSTED — top up bankowner immediately")
      under its own static dedup key so ops tops up rather than debugging a
      bug. The row still stays Fiat Received; idempotency and no-double-spend
      are unchanged.

Tests cover: mismatch/no-secrets alert vs. noise stays silent; float below /
at / above floor, drained-account zero, IBEX read failure and resolver throw
don't crash, disabled feature skips; insufficient-balance -> float step and
the distinct exhausted alert while other failures keep the generic one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NcEjF6SS3Ci5D4CzZqdPjb
…t wallet

Code review found the proactive float monitor read the wrong account. It
resolved the balance via getBankOwnerWalletId() — the bankowner account's
default (USD) wallet — while auto-credit actually spends from the bankowner
USDT wallet (credit-topup.ts selects find(Usdt) ?? find(Usd)). In flash's
IBEX-custodial model each walletId is its own IBEX account, so the monitor was
reading a different account's balance and mis-parsing it as USDT. A drained
USDT float would hide behind a funded USD wallet (no page ever fires) and a low
USD wallet would false-alarm as "USDT float low".

Resolve the funding wallet exactly the way credit-topup does — findByRole
("bankowner") -> listByAccountId -> find(Usdt) ?? find(Usd) — and read THAT
wallet's balance in its own currency, so the monitored account is provably the
funding source. Parse both USDT and USD balances so the USD fallback is not
scored as an empty USDT float.

Tests now drive the real resolution path: assert getAccountDetails is called
with the USDT wallet id (USD wallet listed first, proving currency- not
order-based selection), add a USD-fallback case, and cover the
account/wallet-resolution failure paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QwZjNcKfkMhrBE33HcUVN
…olver

Two code-review fixes to the Fygaro float monitor:

- Gate the cron on the auto-credit master flag, not just the feature flag.
  During the record-only phase (fygaro.enabled=true, credit.enabled=false)
  nothing spends from the bankowner treasury, so paging "top up bankowner"
  every window was premature noise contradicting the alert's own instruction.
  Now returns early on `!FygaroConfig.enabled || !FygaroConfig.credit?.enabled`.

- Extract `resolveFygaroTreasuryFundingWallet` in credit-topup and call it from
  both the credit path and the float monitor. The funding-wallet selection
  (bankowner -> USDT ?? USD) was duplicated verbatim; the "monitor the exact
  wallet auto-credit spends from" invariant is now enforced by shared code
  instead of two byte-identical copies that could silently drift apart.

Tests: assert the monitor performs no repository/IBEX read and fires no alert
when auto-credit is disabled (and when the credit block is absent), mirroring
the feature-disabled test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QwZjNcKfkMhrBE33HcUVN
…rker

Residual review finding: the float monitor runs as a one-shot cron Job, so
alertBridge's process-local in-memory dedup resets every ~15-min run and
cannot suppress the float-low warning across runs — a treasury below the
floor would page every run (~4/hr) instead of the ~1/hr the alert layer
implies. Gate the alert on a Redis NX marker (1h TTL) that survives the
process restarts. A Redis error falls through to alerting: over-notifying
is the safe failure mode for a draining float, never silence. Tests: the
marker suppresses across runs, and a Redis outage still alerts (fail-open).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NcEjF6SS3Ci5D4CzZqdPjb
The typos linter flags "mis-parse"/"mis-parsed" (wants miss/mist). No
behavior change; comment wording only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NcEjF6SS3Ci5D4CzZqdPjb
Two silent-failure gaps the ops-hardening work was meant to close:

- float-monitor read IBEX via Ibex.getAccountDetails directly, so a
  drained/never-funded treasury account (IBEX answers 404) came back as an
  IbexError, hit the `instanceof Error` bail, and returned WITHOUT paging —
  the exact empty-float condition the monitor exists to catch. Read through
  the shared getBalanceForWallet helper instead, which maps 404 -> ZERO (and
  absent balance -> ZERO), so a dry treasury trips the low-float alert.

- verify-signature 401'd every real webhook on a systematic clock skew (server
  drift / NTP down) but only warn-logged it. Now it pages via a distinct static
  dedup key (fygaroClockSkew) so a stuck clock surfaces on its own and replayed
  old webhooks still collapse to one warning per window — without masking, or
  being masked by, the wrong-secret alert.

Tests: 404-drain trips low-float with balance_usd 0; the read-failure case now
models a real (non-404) IbexError; skew rejection asserts a page under its own
dedup key.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QwZjNcKfkMhrBE33HcUVN
The float-exhausted and generic credit-failure tests asserted only
severity+title, never the dedupKey. A regression of the branch that
swaps the per-transaction key (fygaroCreditFailed) for the static
key (fygaroFloatExhausted) would pass both title assertions while a
real treasury outage floods on-call with one page per transaction —
the exact flood this hardening exists to prevent.

- Replace the constant `generateDedupKey` stub (Proxy returning
  "dedup" for every key) with the real, pure dedup-key generator so
  the branch actually produces distinct keys under test.
- Exhausted test: assert dedupKey === "fygaro:float-exhausted".
- Generic-failure test: assert the per-transaction
  "fygaro:credit-failed:<tx>" key is still used.

Both sides of the ternary are now pinned; verified by mutating the
source to always use the per-transaction key (float-exhausted test
fails) with no false positive on the generic test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QwZjNcKfkMhrBE33HcUVN
@islandbitcoin
islandbitcoin merged commit a626d6a into main Aug 11, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants