fault_manager: debounce a fault code by the code, not only by who reported it

docs/config/fault-manager.rst

@@ -304,12 +304,105 @@ threshold overrides:
 
    When multiple entities report the same ``fault_code``, each event applies the
    thresholds resolved from that event's ``source_id``. This means the debounce
-   behavior follows the reporting entity, not the fault.
+   behavior follows the reporting entity, not the fault. The debounce counter,
+   however, belongs to the fault code, so the two entities share one counter under
+   two policies - see `Per-Fault-Code Thresholds`_ for what that does and how to
+   settle it. The node warns when it happens.
 
    ``auto_confirm_after_sec`` is global-only and cannot be overridden per-entity.
    Critical faults skip debounce and confirm on their first occurrence; that is
    built in, not a parameter, so it can be neither disabled nor set per entity.
 
+Per-Fault-Code Thresholds
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Per-entity thresholds are the right tool when you know which subsystems are noisy but not
+which codes they will emit. They have one limit: the debounce counter is kept per
+``fault_code``, while the override is chosen per ``source_id``. Two entities reporting one
+code therefore share a counter and debounce it under two policies, and the report that
+happens to arrive decides the transition:
+
+.. code-block:: text
+
+   motor  (confirmation_threshold=-5) reports OVERHEAT -> counter=-1, PREFAILED
+   lidar  (confirmation_threshold=-1) reports OVERHEAT -> counter=-1, CONFIRMED
+
+The motor's policy was bypassed. Per-fault_code thresholds are the layer that removes
+that: they are matched on the code itself, so they resolve the same whoever reports.
+
+.. code-block:: yaml
+
+   fault_manager:
+     ros__parameters:
+       # Path to YAML file with per-fault_code overrides
+       fault_thresholds:
+         config_file: "/etc/ros2_medkit/fault_thresholds.yaml"
+
+The file is a map of fault codes to threshold overrides, the same three fields an entity
+override carries:
+
+.. code-block:: yaml
+
+   # fault_thresholds.yaml
+   MOTOR_OVERHEAT:
+     confirmation_threshold: -5    # five events, whoever reports them
+     healing_threshold: 10
+
+   LIDAR_FAIL:
+     confirmation_threshold: -1    # instant
+     healing_threshold: 1
+
+.. list-table::
+   :header-rows: 1
+   :widths: 35 15 50
+
+   * - Parameter
+     - Default
+     - Description
+   * - ``fault_thresholds.config_file``
+     - ``""``
+     - Path to YAML file with per-fault_code threshold overrides. Empty = disabled.
+
+**How the layers combine:**
+
+- Three layers, each applied on top of the last: the **global** defaults, then the
+  **entity** override whose prefix matches the reporting ``source_id``, then the
+  **fault code's** own override. A field a layer does not set is left as the layer below
+  had it, so ``fault_code`` > ``source_id`` > global for every field independently.
+- Matching on the code is **exact**. A fault code is an identifier, not a path: an entry
+  for ``MOTOR`` does not capture ``MOTOR_OVERHEAT`` the way ``/sensors`` captures
+  ``/sensors/lidar``.
+- A code that is not listed resolves exactly as it did before: entity override if one
+  matches, global otherwise. The layer is opt-in and changes nothing on its own.
+- Like the entity file, this one is loaded once at node startup. Changes require a restart.
+
+.. note::
+
+   An override that names only some fields settles only those. If ``MOTOR_OVERHEAT`` pins
+   ``confirmation_threshold`` but not ``healing_threshold``, the healing direction still
+   follows whichever entity reported, and the node still reports the conflict below. Pin
+   all three fields to settle a code completely.
+
+**When one code is debounced two ways**
+
+The node resolves the policy for every report, and warns the first time two sources resolve
+different policies for one code:
+
+.. code-block:: text
+
+   [WARN] Fault code 'OVERHEAT' is debounced two ways: '/powertrain/motor/left' resolves
+   confirmation=-5 healing_enabled=false healing=10, '/sensors/lidar/front' resolves
+   confirmation=-1 healing_enabled=true healing=1. The debounce counter belongs to the
+   fault code, so whichever source reports decides the transition and the other policy is
+   bypassed. Pin confirmation_threshold, healing_enabled and healing_threshold for this code
+   in fault_thresholds.config_file.
+
+The warning names both sources and both resolved policies, and is emitted **once per fault
+code** for the life of the node, so a busy reporter does not turn it into a log storm. It
+is a diagnostic, not an error: the configuration is legal and the fault is still debounced,
+just not under a policy an operator chose. Giving the code an entry in
+``fault_thresholds.config_file`` that pins all three fields ends it.
+
 Snapshot Configuration
 ----------------------
 
@@ -723,6 +816,10 @@ Complete Example
        entity_thresholds:
          config_file: "/etc/ros2_medkit/entity_thresholds.yaml"
 
+       # Per-fault_code debounce overrides (applied on top of the entity ones)
+       fault_thresholds:
+         config_file: "/etc/ros2_medkit/fault_thresholds.yaml"
+
        # Snapshots
        snapshots:
          enabled: true

docs/requirements/specs/faults.rst

@@ -48,6 +48,18 @@ Faults
    shall take precedence over global defaults. Unspecified fields shall inherit from global
    configuration. When no entity prefix matches, global defaults shall apply.
 
+.. req:: Per-Fault-Code Debounce Thresholds
+   :id: REQ_INTEROP_107
+   :status: verified
+   :tags: Faults
+
+   The fault manager shall support per-fault_code debounce threshold configuration using
+   exact matching on the reported fault code. A fault-code override for
+   ``confirmation_threshold``, ``healing_enabled``, or ``healing_threshold`` shall take
+   precedence over both the per-entity override resolved for the reporting source and the
+   global defaults. Unspecified fields shall inherit from the layer below. When a fault code
+   has no override, the resolved per-entity or global configuration shall apply unchanged.
+
 .. req:: Fault Snapshot and Rosbag Capture
    :id: REQ_INTEROP_088
    :status: verified

src/ros2_medkit_fault_manager/CMakeLists.txt

@@ -128,7 +128,7 @@ set(FAULT_MANAGER_FILES
     src/correlation/config_parser.cpp
     src/correlation/pattern_matcher.cpp
     src/correlation/correlation_engine.cpp
-    src/entity_threshold_resolver.cpp
+    src/threshold_resolver.cpp
 )
 if(POSTGRES_SUPPORT)
   list(APPEND FAULT_MANAGER_FILES
@@ -270,6 +270,11 @@ endif()
   target_link_libraries(test_entity_thresholds fault_manager_lib)
   medkit_target_dependencies(test_entity_thresholds rclcpp ros2_medkit_msgs)
 
+  # Fault-code threshold resolver tests
+  medkit_add_gtest(test_fault_code_thresholds test/test_fault_code_thresholds.cpp)
+  target_link_libraries(test_fault_code_thresholds fault_manager_lib)
+  medkit_target_dependencies(test_fault_code_thresholds rclcpp ros2_medkit_msgs)
+
   # Integration tests
   install(DIRECTORY test
     DESTINATION share/${PROJECT_NAME}
@@ -289,6 +294,12 @@ endif()
   medkit_add_launch_test(test_entity_thresholds_integration test/integration/test_entity_thresholds_integration.test.py
     TIMEOUT 60 LABELS "integration")
 
+  medkit_add_launch_test(test_fault_code_thresholds_integration
+    test/integration/test_fault_code_thresholds_integration.test.py TIMEOUT 60 LABELS "integration")
+
+  medkit_add_launch_test(test_fault_code_healing_restart
+    test/integration/test_fault_code_healing_restart.test.py TIMEOUT 120 LABELS "integration")
+
   # Drives healing with the event counts a one-event-per-transition reporter
   # actually sends: one FAILED per raise, one PASSED per clear. Parametrized over
   # healing_threshold, so the node launches twice, and one case holds a settled

src/ros2_medkit_fault_manager/README.md

@@ -51,7 +51,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
   when a cleared fault is raised again - and tracks all reporting sources
 - **Severity escalation**: Fault severity is updated if a higher severity is reported
 - **Persistent storage**: SQLite (default) or PostgreSQL backend ensures faults survive node restarts
-- **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides
+- **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity and per-fault_code threshold overrides
 - **Snapshot capture**: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless `snapshots.retain_on_clear` is set)
 - **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
 - **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below)
@@ -70,6 +70,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
 | `healing_threshold`             | int    | `3`                                | Counter value at which faults are healed                                        |
 | `auto_confirm_after_sec`        | double | `0.0`                              | Auto-confirm PREFAILED faults after timeout (0 = disabled)                      |
 | `entity_thresholds.config_file` | string | `""`                               | Path to YAML file with per-entity debounce threshold overrides                  |
+| `fault_thresholds.config_file`  | string | `""`                               | Path to YAML file with per-fault_code debounce threshold overrides, applied on top of the entity ones |
 | `near_miss.max_per_fault`       | int    | `200`                              | Near-miss entries retained per fault code, oldest evicted first (0 = unlimited) |
 
 ### Snapshot Parameters
@@ -174,7 +175,11 @@ rows written before the field existed.
 
 With per-entity thresholds the recorded `confirmation_threshold` is the one belonging to the
 **reporting source**, while the debounce counter is shared by every source of that fault code. It
-is therefore not by itself the distance to confirmation for the fault as a whole.
+is therefore not by itself the distance to confirmation for the fault as a whole. An entry in
+`fault_thresholds.config_file` that sets `confirmation_threshold` makes the two agree again: it
+resolves the same for every source, so the recorded threshold is the fault's own. An entry with
+only healing fields keeps the per-source threshold. The node warns
+once per code when two sources resolve different policies for it.
 
 Entries are kept and evicted in **arrival order**, not by their timestamps. Reporters carry their
 own clocks, so a report can arrive carrying a timestamp behind one already stored; ordering the

src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp

@@ -24,11 +24,11 @@
 #include "rclcpp/rclcpp.hpp"
 #include "ros2_medkit_fault_manager/capture_thread_pool.hpp"
 #include "ros2_medkit_fault_manager/correlation/correlation_engine.hpp"
-#include "ros2_medkit_fault_manager/entity_threshold_resolver.hpp"
 #include "ros2_medkit_fault_manager/fault_audit_log.hpp"
 #include "ros2_medkit_fault_manager/fault_storage.hpp"
 #include "ros2_medkit_fault_manager/rosbag_capture.hpp"
 #include "ros2_medkit_fault_manager/snapshot_capture.hpp"
+#include "ros2_medkit_fault_manager/threshold_resolver.hpp"
 #include "ros2_medkit_msgs/msg/fault_event.hpp"
 #include "ros2_medkit_msgs/srv/clear_fault.hpp"
 #include "ros2_medkit_msgs/srv/get_fault.hpp"
@@ -187,9 +187,21 @@ class FaultManagerNode : public rclcpp::Node {
   /// Extract topic name from full topic path (last segment)
   static std::string extract_topic_name(const std::string & topic_path);
 
-  /// Resolve debounce config for a given source_id using entity threshold resolver.
-  /// Falls back to global config if no entity-specific overrides match.
-  DebounceConfig resolve_config(const std::string & source_id) const;
+  /// Resolve the debounce config a report is debounced under.
+  /// Three layers, each applied on top of the last: the global config, the
+  /// longest-prefix entity override matching @p source_id, and the exact-match
+  /// override for @p fault_code. A layer only sets the fields it configures.
+  DebounceConfig resolve_config(const std::string & source_id, const std::string & fault_code) const;
+
+  /// Warn once per fault code when two sources debounce it under different
+  /// policies. The counter belongs to the fault code and the entity override to
+  /// the source, so the report that arrives decides the transition and the other
+  /// source's policy is silently bypassed.
+  /// @param fault_code The code just reported.
+  /// @param source_id The source that reported it.
+  /// @param resolved The config that report resolved to.
+  void warn_on_conflicting_debounce_policy(const std::string & fault_code, const std::string & source_id,
+                                           const DebounceConfig & resolved);
 
   /// Create the tamper-evident audit log from parameters (nullptr if disabled).
   std::unique_ptr<FaultAuditLog> create_audit_log();
@@ -216,7 +228,22 @@ class FaultManagerNode : public rclcpp::Node {
   QueueFullPolicy capture_queue_full_policy_{QueueFullPolicy::kRejectNewest};
   DebounceConfig global_config_;  ///< Global debounce config (built from ROS params)
   std::unique_ptr<FaultStorage> storage_;
-  std::unique_ptr<EntityThresholdResolver> threshold_resolver_;  ///< Per-entity threshold overrides
+  std::unique_ptr<EntityThresholdResolver> threshold_resolver_;      ///< Per-entity threshold overrides
+  std::unique_ptr<FaultCodeThresholdResolver> fault_code_resolver_;  ///< Per-fault_code threshold overrides
+
+  /// The first debounce policy seen for a fault code, and who reported it.
+  /// Kept only to notice a second source resolving a different policy for the
+  /// same code, and warned about once. One entry per fault code in the store
+  /// (it is only written once the store has the fault), so it is bounded by the
+  /// same thing the fault store is. Written from
+  /// the ReportFault callback, which the node's single-threaded executor
+  /// serialises with every other callback that touches node state.
+  struct DebouncePolicyWitness {
+    DebounceConfig config;  ///< The policy the first report resolved to
+    std::string source_id;  ///< The source that reported it
+    bool warned{false};     ///< Whether the conflict has already been reported
+  };
+  std::unordered_map<std::string, DebouncePolicyWitness> debounce_policy_witness_;
 
   /// Tamper-evident audit log of fault transitions (nullptr when disabled).
   std::unique_ptr<FaultAuditLog> audit_log_;

src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp

@@ -16,6 +16,7 @@
 
 #include <cstdint>
 #include <exception>
+#include <functional>
 #include <map>
 #include <mutex>
 #include <optional>
@@ -463,11 +464,17 @@ class FaultStorage {
   /// @return Vector of all faults in storage
   virtual std::vector<ros2_medkit_msgs::msg::Fault> get_all_faults() const = 0;
 
-  /// One-time startup cleanup: reclassify HEALED faults as CLEARED. Called when healing is disabled,
-  /// so a HEALED row left by a previous (healing-enabled) run does not behave inconsistently under
-  /// the latch. Default is a no-op (in-memory storage starts empty).
+  /// Predicate for reclassify_healed_as_cleared: true keeps that fault code HEALED.
+  using KeepHealedFn = std::function<bool(const std::string & fault_code)>;
+
+  /// One-time startup cleanup: reclassify HEALED faults as CLEARED, so a HEALED row left by a
+  /// previous (healing-enabled) run does not behave inconsistently under the latch.
+  /// Default is a no-op (in-memory storage starts empty).
+  /// @param keep_healed Codes for which this returns true stay HEALED (their own healing is still
+  ///        enabled). Empty means reclassify every HEALED fault.
   /// @return fault codes of the reclassified faults, so the caller can audit each transition
-  virtual std::vector<std::string> reclassify_healed_as_cleared() {
+  virtual std::vector<std::string> reclassify_healed_as_cleared(const KeepHealedFn & keep_healed = {}) {
+    (void)keep_healed;
     return {};
   }
 
@@ -535,7 +542,7 @@ class InMemoryFaultStorage : public FaultStorage {
   std::vector<RosbagFileInfo> get_all_rosbag_files() const override;
   std::vector<RosbagFileInfo> list_rosbags_for_entity(const std::string & entity_fqn) const override;
   std::vector<ros2_medkit_msgs::msg::Fault> get_all_faults() const override;
-  std::vector<std::string> reclassify_healed_as_cleared() override;
+  std::vector<std::string> reclassify_healed_as_cleared(const KeepHealedFn & keep_healed = {}) override;
 
  private:
   /// Update fault status based on debounce counter and given config

src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/postgres_fault_storage.hpp

@@ -105,7 +105,7 @@ class PgFaultStorage : public FaultStorage {
   std::vector<RosbagFileInfo> get_all_rosbag_files() const override;
   std::vector<RosbagFileInfo> list_rosbags_for_entity(const std::string & entity_fqn) const override;
   std::vector<ros2_medkit_msgs::msg::Fault> get_all_faults() const override;
-  std::vector<std::string> reclassify_healed_as_cleared() override;
+  std::vector<std::string> reclassify_healed_as_cleared(const KeepHealedFn & keep_healed = {}) override;
 
   /// Get the connection info string used to initialize the database
   const std::string & conn_info() const {

src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp

@@ -93,7 +93,7 @@ class SqliteFaultStorage : public FaultStorage {
   std::vector<RosbagFileInfo> get_all_rosbag_files() const override;
   std::vector<RosbagFileInfo> list_rosbags_for_entity(const std::string & entity_fqn) const override;
   std::vector<ros2_medkit_msgs::msg::Fault> get_all_faults() const override;
-  std::vector<std::string> reclassify_healed_as_cleared() override;
+  std::vector<std::string> reclassify_healed_as_cleared(const KeepHealedFn & keep_healed = {}) override;
 
   /// Get the database path
   const std::string & db_path() const {