[Bug]: CRITICAL→CLEAR notification never dispatched for long-lived alert instances (transition stored in Cloud, no provider handoff) #1156

Bug description

Summary

Two alert instances on a single monitored node transitioned CRITICAL → CLEAR at
2026-10-27 09:25:36 UTC. Both transitions were received and stored by Netdata Cloud
and are individually viewable in the Cloud alert history. Neither produced a
recovery (CLEAR) notification.

Our email provider has no acceptance event for either message. In provider
semantics, an acceptance event is emitted the moment the message is handed over.
Its absence means the notifications were never handed off — the messages stopped
inside Netdata Cloud's notification processing.

This is therefore not a bounce, not spam classification, not provider-side
suppression, and not a mailbox filter.

Timeline (all UTC, 2026-10-27 unless stated)

The two missing notifications are bracketed on both sides by successful CLEAR
deliveries from the same node, to the same recipient, on the same channel, within
16 minutes. The recovery path was demonstrably working end-to-end before and after
the failure.

What makes this selective rather than a configuration problem

The two alarms that failed are long-lived instances whose last notification
predates the 2026-10-25 agent upgrade. Every alarm instance created after the
upgrade — including the user's deliberate synthetic control — delivered correctly.

Two additional distinguishing facts

1. Both missing transitions occurred in the same second, on the same node.
2. The recipient's mailbox had zero bounces, drops or deferrals across every
   message delivered to it over a 30-day window. No flood-protection,
   channel-disabled or alerts-misconfigured notice was emitted to the user either.

Excluded, each by evidence rather than assumption

Not a duplicate of #984 — please read before closing

#984 was closed with the correct rule stated: a CLEAR is filtered unless an active
(WARNING/CRITICAL) notification was previously sent. That rule's precondition was
satisfied here: an active CRITICAL notification for each of these two alarms was
delivered to this recipient on 2026-10-24, and is confirmed delivered in provider
records.

The same applies to the per-recipient "last notified status" check described in
#785. Either:

• (a) that per-recipient state was lost or invalidated — most plausibly when the
  node re-registered after the 2026-10-25 agent upgrade — in which case the state
  loss is itself the defect, because the CRITICAL demonstrably reached the
  recipient; or
• (b) the state was intact and Cloud-side notification processing dropped the
  message.

In both cases the user-visible contract was violated: you received the Critical,
therefore you receive the Clear. Neither #984's nor #820's documented withholding
conditions (flood protection, silencing, alert pruning) apply — each is excluded by
the table above.

Impact

Silent loss of recovery notifications is the worst class of alerting defect, because
the victim cannot observe it. The user continued to believe two alarms were in a
CRITICAL state that had in fact resolved ~2 hours earlier. There is no user-visible
record of dispatch outcome that would have revealed this (see #317, #713, #655), and
no supported way to replay a notification for an already-recorded transition.

Expected behavior

Every alert transition to CLEAR that follows a delivered active (WARNING/CRITICAL)
notification to a recipient produces a recovery notification to that recipient, on
every configured channel.

Specifically:

1. Both CRITICAL → CLEAR transitions at 09:25:36 UTC should have produced two
   independent recovery notifications.
2. Notification dispatch decisions should be observable by the affected user or by
   support without internal access — a transition that Cloud stores and serves in
   the UI, but which never reaches the provider, is currently indistinguishable from
   "no notification was configured".
3. If a notification is withheld by any filter, the reason should be recorded and
   surfaced (this is the request already open in Alerts Smartboard: Enhance the Configured Alerts display to show notification details #317 / [Feat]: add email status to feed #713 / [Feat]: Expose alert delay in Netdata Cloud #655).

Steps to reproduce

The exact trigger is not yet confirmed, and I am stating that plainly rather than
presenting a repro I have not validated. The conditions below are the observed
preconditions; the re-registration path is the leading suspect.

1. A Space with a working email notification method, confirmed delivering recovery
   notifications for other alarms.
2. Two custom alarm instances reach CRITICAL, and active CRITICAL notifications are
   delivered to the recipient.
3. The alarm instances persist across an agent auto-upgrade / node re-registration
   boundary (here: v2.12.2 on 2026-10-25), without being recreated.
4. Both instances transition CRITICAL → CLEAR within the same second on the same
   node.
5. Observe: both transitions are stored and individually viewable in Cloud alert
   history; no recovery notification arrives; the email provider holds no acceptance
   event for either message.

Control that did work, and is worth running for comparison: create a new alarm
instance after the upgrade, drive it CRITICAL then CLEAR. The CLEAR delivers.

The distinguishing variable appears to be alarm-instance age across the
re-registration boundary, not the transition itself.

Screenshots

No response

Error Logs

No response

Desktop

No response

Additional context

Environment

• Netdata Agent v2.12.2, auto-upgraded 2026-10-25 (stable channel), single node
• Netdata Cloud, email notification method, one recipient
• Two custom (user-authored) alarm definitions, long-lived instances

Related issues — same subsystem

• [Bug]: No emails received about alert. #984 — states the CLEAR-suppression rule; precondition satisfied here, see above
• [Bug]: Unnecessary/invalid email notifications #785 — per-recipient "last notified status" dedup state (fixed in 2023; suspect here)
• [Bug]: Notification integrations that open an incident have indicent close issues #820 — documented cases where Cloud withholds CLEAR (flood protection, silencing, pruning)
• [Bug]: Alert notifications UNKNOWN → CLEAR #514 — precedent: the Cloud-side "should we notify" gate was itself defective and fixed
• [Bug]: Alert events raised right before agent shutdown will be missed on the cloud #587 — nearest mechanism, but there the transition was never stored; here it is
• alert_transitions API returns incomplete transitions (raises missing) #1155 — open, same subsystem: alert_transitions API returns incomplete transitions
• [Bug]: Alert trigger time does not correspond with feed events #780 — node vs node-instance aggregation and the REMOVED/UNKNOWN churn that daily
  auto-upgrades generate
• Alerts Smartboard: Enhance the Configured Alerts display to show notification details #317, [Feat]: add email status to feed #713, [Feat]: Expose alert delay in Netdata Cloud #655 — open, no user-visible dispatch/silence/delay outcome

Hypothesis — labelled as hypothesis, not a finding

Both alarms' last notification predates the 2026-10-25 agent upgrade, and both
missing transitions are same-second. Two failure modes fit equally well and I have
not distinguished them:

1. Per-recipient "last notified" state was invalidated when the node re-registered
   as a new node instance, so the CLEAR hit the [Bug]: Unnecessary/invalid email notifications #785-style skip.
2. Cloud-side notification processing discarded both messages on stale instance state.

I am deliberately not committing to either — that is an internal forensics question.
What is established independently of the mechanism is the contract violation.

Verification already performed

• Provider delivery records were searched for both transition identifiers: no
  acceptance event, no delivery, no bounce, no drop, no spam report.
• The notifications the user reported receiving are all confirmed delivered with
  provider message IDs and SMTP acceptance transcripts.
• Cloud alert-history views of both transitions rendered successfully at multiple
  times on 2026-10-27, confirming the transitions are stored and servable.

One fact worth knowing for triage, and it is derivable from the open-source
Agent: the Agent treats local transport acceptance as delivery —
aclk_send_alarm_log_entry() returns success once the MQTT layer accepts the
message, after which the Agent advances the queue version and deletes the row
(src/aclk/aclk_alarm_api.c, src/database/sqlite/sqlite_aclk_alert.c). The
Agent's inbound Cloud message handler table contains no message type acknowledging an
individual alert transition. Neither the Agent nor the user can therefore prove
ingestion — only Cloud-side state can. That is why this had to be investigated from
our side.