diagnostic_bridge: keep the numbers behind a fault, and stop confirming every planned STALE

docs/api/rest.rst

@@ -1514,7 +1514,15 @@ Query and manage faults.
 
    **Snapshot Types:**
 
-   - ``freeze_frame``: Data captured at fault confirmation. Entity frames for
+   - ``freeze_frame``: Data captured at fault confirmation: one entry per
+     captured topic, plus one entry named ``freeze_frame`` whose ``data`` maps
+     each topic to its value. That entry can also hold **reporter evidence**
+     under the key ``x-medkit-reported``: the key-value pairs a reporter sent
+     with its FAILED reports (for example the ``DiagnosticStatus`` values from
+     the diagnostic bridge), as string values, last value per key. Topic keys
+     start with ``/``, so they cannot clash with it. Evidence is written on
+     every FAILED report, so a fault with no capture config can still have a
+     ``freeze_frame`` entry with only evidence in it. Entity frames for
      faults that were already confirmed when the gateway started are captured
      at gateway start instead and carry ``"capture_origin": "startup"`` in
      their ``x-medkit`` block. An entity frame also carries ``source`` in

docs/config/diagnostic-bridge.rst

@@ -37,7 +37,31 @@ The diagnostic bridge:
      - SEVERITY_ERROR (2)
    * - STALE
      - FAILED
-     - SEVERITY_ERROR (2)
+     - ``stale_severity``, SEVERITY_CRITICAL (3) by default
+
+.. warning::
+
+   This table previously documented STALE as SEVERITY_ERROR. The bridge has always sent
+   SEVERITY_CRITICAL, and CRITICAL **bypasses debounce** in the fault manager, so a STALE
+   status confirms a CRITICAL fault on its first sample.
+
+   That matters because STALE is the one level a node can reach by design: a GPS goes STALE
+   in every tunnel, an IMU reports covariance -1 while it settles. Left unconfigured, every
+   such outage is a confirmed CRITICAL fault. Use ``stale_severity`` to set the level, or
+   ``stale_severity_overrides`` to name just the diagnostic names that go STALE on purpose;
+   those statuses then debounce like any other.
+
+.. important::
+
+   A non-CRITICAL STALE is only debounced if the fault manager is set up for it. With the
+   default ``confirmation_threshold: -1``, the first FAILED report confirms the fault. The
+   bridge's local filter only waits for 3 reports within 10 s. So with both defaults, a
+   planned STALE still becomes a confirmed fault after about three cycles. Set
+   ``confirmation_threshold`` to ``-3`` or lower to debounce it.
+
+   This threshold is picked by ``source_id``. All statuses from the bridge share one
+   ``source_id`` (``/diagnostic_bridge``, unless ``use_hardware_id_as_source_id`` is on), so
+   a threshold for the bridge applies to **every** diagnostic it forwards, not only STALE ones.
 
 Parameters
 ----------
@@ -49,6 +73,9 @@ Parameters
        diagnostics_topic: "/diagnostics"   # Topic to subscribe to
        auto_generate_codes: true           # Auto-generate fault codes from names
        keyvalue_codes: ["fault_code"]      # Take the code from these key-value keys
+       stale_severity: "WARN"              # What a STALE status reports at
+       # Per diagnostic name, longest matching prefix wins:
+       "stale_severity_overrides.gps": "WARN"
 
 .. list-table::
    :header-rows: 1
@@ -71,6 +98,61 @@ Parameters
        as its value, so a publisher can name its own code instead of relying on
        a mapping. Checked after ``name_to_code`` and before auto-generation.
        Empty strings in the list are ignored.
+   * - ``stale_severity``
+     - ``CRITICAL``
+     - Severity a STALE status reports at, one of ``INFO``, ``WARN``, ``ERROR``,
+       ``CRITICAL`` (case-insensitive). Applies to STALE only: the other levels are
+       facts about the status, not deployment decisions. A name that does not parse
+       is reported and ``CRITICAL`` is used.
+   * - ``stale_severity_overrides.<name>``
+     - ``-``
+     - Severity for STALE statuses whose name starts with ``<name>``. Diagnostic names
+       are conventionally ``<component>: <check>``, so a component prefix covers every
+       check it publishes. The **longest matching prefix** wins. An override that does
+       not parse is reported and **ignored**, leaving ``stale_severity`` in force -
+       applying ``CRITICAL`` to a typo would restore the immediate-confirm behaviour the
+       operator was configuring their way out of.
+
+Evidence
+--------
+
+A FAILED report carries the ``DiagnosticStatus`` key-values to the fault manager, which keeps
+them in the fault's freeze frame and serves them from
+``GET /api/v1/apps/{app}/faults/{code}``. The fault is reported under the bridge's
+``source_id``, so with the bridge defaults the app is ``diagnostic_bridge``:
+
+.. code-block:: console
+
+   $ curl -s localhost:8080/api/v1/apps/diagnostic_bridge/faults/FUSION_DIVERGED | jq '.environment_data.snapshots[] | select(.type == "freeze_frame" and .name == "freeze_frame") | .data["x-medkit-reported"]'
+   {
+     "rejected_fixes": "37",
+     "nis": "0.03"
+   }
+
+If topics were also captured for the code, the list has one entry per topic before the
+freeze frame. Those entries also have type ``freeze_frame``, so the filter checks the name
+too.
+
+A node that publishes outlier counts and gate statistics has already computed why it is
+unhappy; before this the bridge read those values only to pick a fault code and dropped the
+rest, so the fault record said a node complained but not what it saw.
+
+Evidence is written on **every** FAILED report, not only the one that confirms the fault, so a
+code that never confirms still has evidence. Only the **last** value of each key is kept, and
+near-miss entries do not store evidence. So a code with several near misses shows only the
+numbers from its latest report.
+
+The frame's other keys are topic names sampled by the fault manager, always fully qualified
+and so always starting with ``/``. ``x-medkit-reported`` cannot collide with one, which is what lets
+a reader tell a value the reporter asserted from one the fault manager sampled.
+
+.. note::
+
+   Evidence limits per fault code: at most 32 keys, 128 bytes per key and 512 bytes per value
+   (bytes, not characters). There is no limit on total size. Bytes that are not valid UTF-8
+   are stored as U+FFFD, and the limits are checked after that. An entry over a limit is
+   dropped, not cut short, and a warning is logged. The fault is recorded either way. A key already stored can always be updated, so a steady reporter
+   at the bound can still refresh its own numbers.
 
 Custom Fault Code Mappings
 --------------------------

docs/requirements/specs/faults.rst

@@ -48,6 +48,19 @@ Faults
    shall take precedence over global defaults. Unspecified fields shall inherit from global
    configuration. When no entity prefix matches, global defaults shall apply.
 
+.. req:: Reporter-Supplied Fault Evidence
+   :id: REQ_INTEROP_110
+   :status: verified
+   :tags: Faults
+
+   A FAILED fault report shall be able to carry key-value measurements, which the fault
+   manager shall retain in the fault's freeze frame under a reserved key distinct from
+   captured topic values, and serve with the fault's environment data. Evidence shall be
+   retained across reports that carry none, preserved when a confirmation capture rebuilds
+   the frame, and bounded per fault code in entry count and in key and value length
+   (measured in bytes), with entries exceeding a bound dropped whole and reported rather
+   than truncated.
+
 .. req:: Fault Snapshot and Rosbag Capture
    :id: REQ_INTEROP_088
    :status: verified

src/ros2_medkit_diagnostic_bridge/CMakeLists.txt

@@ -117,6 +117,8 @@ if(BUILD_TESTING)
   # repo convention) so it runs only in the integration phase.
   medkit_add_launch_test(test_integration test/test_integration.test.py TIMEOUT 60
     LABELS "integration")
+  medkit_add_launch_test(test_stale_and_evidence test/test_stale_and_evidence.test.py TIMEOUT 90
+    LABELS "integration")
   ros2_medkit_relax_vendor_warnings()
 endif()
 

src/ros2_medkit_diagnostic_bridge/README.md

@@ -15,7 +15,64 @@ to the FaultManager as faults.
 | OK (0) | - | Sends PASSED event (healing) |
 | WARN (1) | WARN (1) | Reports fault |
 | ERROR (2) | ERROR (2) | Reports fault |
-| STALE (3) | CRITICAL (3) | Reports fault |
+| STALE (3) | `stale_severity` (CRITICAL by default) | Reports fault |
+
+STALE is the one level whose severity is a deployment decision rather than a fact. A GPS in
+every tunnel and a dead sensor both publish STALE, and CRITICAL
+[bypasses debounce](../ros2_medkit_fault_manager/README.md), so an expected outage confirms a
+CRITICAL fault on its first sample. Set `stale_severity`, or name the noisy diagnostic names
+with `stale_severity_overrides`, and those statuses go through debounce like any other.
+
+This only helps if the fault manager debounces. With the default `confirmation_threshold: -1`
+the first FAILED report confirms, and the bridge's local filter only waits for 3 reports in
+10 s, so a planned STALE still confirms after about three cycles. Set `confirmation_threshold`
+to `-3` or lower. The threshold is picked by `source_id`, and all statuses from the bridge
+share one (`/diagnostic_bridge`, unless `use_hardware_id_as_source_id` is on), so it applies
+to every diagnostic the bridge forwards.
+
+## Evidence
+
+A FAILED report carries the status's key-values to the fault manager, which keeps them in the
+fault's freeze frame under the reserved key `x-medkit-reported` and serves them back from
+`GET /api/v1/apps/{app}/faults/{code}` (with the bridge defaults the app is
+`diagnostic_bridge`):
+
+```jsonc
+{
+  "environment_data": {
+    "snapshots": [
+      // ...one entry per captured topic, if the code has a capture configured...
+      {
+        "type": "freeze_frame",
+        "name": "freeze_frame",
+        "data": {
+          "x-medkit-reported": { "rejected_fixes": "37", "nis": "0.03" }
+        },
+        "x-medkit": {
+          "topic": "",
+          "message_type": "",
+          "full_data": {
+            "x-medkit-reported": { "rejected_fixes": "37", "nis": "0.03" }
+          },
+          "captured_at": "2026-10-08T09:15:02.123456789Z"
+        }
+      }
+    ]
+  }
+}
+```
+
+The numbers a publisher already computed to decide something was wrong are usually the whole
+explanation of the fault, and before this they were dropped. The frame's other keys are topic
+names sampled by the fault manager, which are always fully qualified and start with `/`, so a
+reader can always tell a value the reporter asserted from one the fault manager sampled.
+
+`keyvalue_codes` still reads the same values to pick a fault code; the two uses are
+independent and either can be used without the other. Only the last value of each key is
+kept, and near-miss entries do not store evidence. Limits per fault code: 32 keys, 128 bytes
+per key, 512 bytes per value, no limit on total size. An entry over a limit is dropped with a
+warning, not cut short, and the fault is recorded either way. Bytes that are not valid UTF-8
+are stored as U+FFFD.
 
 ## Quick Start
 
@@ -39,6 +96,8 @@ ros2 run ros2_medkit_diagnostic_bridge diagnostic_bridge_node
 | `max_tracked_sources` | integer | `512` | Maximum number of per-source FaultReporter instances retained by the bridge |
 | `name_to_code.<name>` | string | - | Custom mapping from diagnostic name to fault code |
 | `keyvalue_codes` | string[] | - | List of keys used to search the diagnostic values for the fault code |
+| `stale_severity` | string | `CRITICAL` | Severity a STALE status reports at. One of `INFO`, `WARN`, `ERROR`, `CRITICAL` |
+| `stale_severity_overrides.<name>` | string | - | Severity for STALE statuses whose name starts with `<name>`. Longest matching prefix wins |
 
 ### Example Configuration
 

src/ros2_medkit_diagnostic_bridge/config/diagnostic_bridge.yaml

@@ -14,6 +14,19 @@ diagnostic_bridge:
     # Maximum number of per-source FaultReporter instances retained by the bridge.
     max_tracked_sources: 512
 
+    # Severity a STALE status reports at. STALE is the one level a node can reach by
+    # design - a GPS in every tunnel, an IMU reporting covariance -1 - and CRITICAL
+    # bypasses debounce, so leaving it CRITICAL confirms a fault on the first sample.
+    # One of INFO, WARN, ERROR, CRITICAL.
+    stale_severity: "CRITICAL"
+
+    # Per-diagnostic STALE severity. Matches on the start of the diagnostic name, so a
+    # component prefix covers every check it publishes. Longest matching prefix wins.
+    # Format: "stale_severity_overrides.<diagnostic name or prefix>": "<SEVERITY>"
+    # Example:
+    # "stale_severity_overrides.gps": "WARN"
+    # "stale_severity_overrides.gps: antenna": "ERROR"
+
     # Custom diagnostic name to fault code mappings
     # Format: "name_to_code.<diagnostic_name>": "<FAULT_CODE>"
     # Example:

src/ros2_medkit_diagnostic_bridge/design/index.rst

@@ -48,12 +48,14 @@ The following diagram shows the data flow and component relationships.
            +name: string
            +message: string
            +hardware_id: string
+           +values: KeyValue[]
        }
    }
 
    package "ros2_medkit_fault_reporter" {
        class FaultReporter {
            +report(fault_code, severity, description)
+           +report(fault_code, severity, description, evidence)
            +report_passed(fault_code)
        }
    }
@@ -62,8 +64,10 @@ The following diagram shows the data flow and component relationships.
 
        class DiagnosticBridgeNode {
            +map_to_fault_code(status): string
-           +map_to_severity(level): optional<uint8>
-           +is_ok_level(level): bool
+           +map_to_severity(level, name): optional<uint8>
+           +stale_severity_for(name): uint8
+           +{static} parse_severity_name(name): optional<uint8>
+           +{static} is_ok_level(level): bool
            --
            -diagnostics_callback(msg)
            -process_diagnostic(status)
@@ -102,9 +106,10 @@ Data Flow
        Reporter -> FM : ReportFault(PASSED)
        note right: Triggers healing
    else level == WARN/ERROR/STALE
-       Bridge -> Reporter : report(fault_code, severity, msg)
-       Reporter -> FM : ReportFault(FAILED)
-       note right: Creates/updates fault
+       Bridge -> Reporter : report(fault_code, severity, msg, values)
+       Reporter -> FM : ReportFault(FAILED, evidence)
+       note right: Creates/updates fault,
+keeps evidence in freeze frame
    end
 
    @enduml
@@ -131,8 +136,9 @@ The bridge maps DiagnosticStatus levels to FaultManager severities:
      - ERROR (2)
      - Error-level fault
    * - STALE (3)
-     - CRITICAL (3)
-     - Stale data treated as critical
+     - ``stale_severity`` (CRITICAL by default), per diagnostic name via
+       ``stale_severity_overrides``
+     - See `Configurable STALE severity`_
 
 Main Components
 ---------------
@@ -148,7 +154,8 @@ Main Components
 
 3. **FaultReporter Integration** - Bridges to FaultManager
    - OK status → ``report_passed()`` for healing
-   - WARN/ERROR/STALE → ``report()`` with mapped severity
+   - WARN/ERROR/STALE → ``report()`` with mapped severity and the status ``values`` as
+     evidence
 
 Configuration
 -------------
@@ -172,6 +179,13 @@ Parameters
    * - ``name_to_code.<name>``
      - (none)
      - Custom mapping from diagnostic name to fault code
+   * - ``stale_severity``
+     - ``CRITICAL``
+     - Severity a STALE status reports at (``INFO``, ``WARN``, ``ERROR``, ``CRITICAL``)
+   * - ``stale_severity_overrides.<name>``
+     - (none)
+     - Severity for STALE statuses whose diagnostic name starts with ``<name>``; longest
+       prefix wins
 
 Example Launch Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -208,8 +222,23 @@ Examples:
 - ``"/robot/arm"`` → ``"ROBOT_ARM"``
 - ``"sensor: status"`` → ``"SENSOR_STATUS"``
 
-STALE as CRITICAL
-~~~~~~~~~~~~~~~~~
+Configurable STALE severity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+STALE maps to ``stale_severity``, CRITICAL by default, because stale data often means a lost
+connection or a crashed node. But some nodes go STALE on purpose (a GPS in a tunnel, an IMU
+while it settles), and CRITICAL skips debounce in the fault manager. So the severity is
+configurable. ``stale_severity_overrides.<name>`` sets it per diagnostic name, and the longest
+matching prefix wins. That is why ``map_to_severity`` takes the diagnostic name as well as the
+level. An override that cannot be parsed is logged and ignored, and ``stale_severity`` is used.
+
+A non-CRITICAL STALE is only held back if the fault manager's ``confirmation_threshold`` needs
+more than one report. That threshold is picked by ``source_id``, which all statuses from the
+bridge share, so it applies to every diagnostic the bridge forwards.
+
+Evidence
+~~~~~~~~
 
-STALE diagnostics are mapped to CRITICAL severity because stale data typically indicates
-a communication failure or node crash, which is a serious system health issue.
+The status ``values`` are sent as ``ReportFault.evidence`` with every FAILED report. The fault
+manager keeps them in the fault's freeze frame under ``x-medkit-reported``. See
+:doc:`/config/diagnostic-bridge`.