Skip to content

refactor(mqtt,instrument): pair DSX publish metrics and logs#3752

Merged
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3732
Jul 21, 2026
Merged

refactor(mqtt,instrument): pair DSX publish metrics and logs#3752
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3732

Conversation

@chet

@chet chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

The API state-change hook, managed-host republisher, and rack BMS worker counted DSX publish outcomes separately from their matching diagnostics.

Typed sibling Events now pair those signals while preserving the frozen {component,status} series and each publisher's historical log behavior. Operational detail remains log-only. The narrow #[label(name = "component")] alias keeps the existing metric key without colliding with the Event log's reserved component identity, and field-attribute diagnostics reject unsupported metadata instead of silently ignoring it.

Table-driven tests cover the outcome matrix and alias guardrails.

Related issues

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

The bounded component/status mappings and publish-outcome metric/log matrix use value_scenarios!; the proc-macro diagnostics use named check_values rows. Stateful queue, republisher, and rack BMS behavior remains covered by their existing focused tests.

Verified with:

  • cargo test -p carbide-instrument-macros -p carbide-instrument -p carbide-mqtt-common
  • cargo test -p carbide-api-core mqtt_state_change_hook --lib
  • cargo test -p carbide-rack bms_client --lib
  • cargo make format-nightly
  • cargo make clippy
  • cargo make carbide-lints
  • cargo make check-event-names
  • cargo xtask check-metric-docs
  • cargo make check-workspace-deps
  • cargo make check-licenses
  • cargo make check-bans

Additional Notes

The #[label(name = "component")] guardrail tests and code-local rustdoc stay with the implementation. The broader instrumentation-guide update is split into #3767 and depends on this PR.

This PR and #3746 both extend Event field-attribute parsing. Merge reconciliation must retain #3746's #[context(value)] mode and this PR's bare-only #[observation] validation; metric-key aliases remain exclusive to #[label(name = "...")].

@copy-pr-bot

copy-pr-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 21b41492-5fc0-4881-aa7f-5a159e399b7d

📥 Commits

Reviewing files that changed from the base of the PR and between 4bf329d and 02c2406.

📒 Files selected for processing (8)
  • crates/api-core/src/mqtt_state_change_hook/hook.rs
  • crates/api-core/src/mqtt_state_change_hook/republisher.rs
  • crates/instrument-macros/src/lib.rs
  • crates/instrument/src/lib.rs
  • crates/instrument/tests/matrix.rs
  • crates/mqtt-common/src/hook.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/rack/src/bms_client.rs

Summary by CodeRabbit

  • Monitoring & Observability
    • Upgraded MQTT event-bus metrics with separate managed-host vs BMS outcome events, richer context (topic, machine id, error), and updated carbide_dsx_event_bus_* naming.
    • Improved state-change/republish and publish paths to record the correct per-outcome serialization and overflow metrics.
  • Bug Fixes
    • Replaced generic logging/metrics with outcome-specific metric recording for serialization, overflow, and publish/timeout errors.
  • Documentation
    • Clarified #[label(name = ...)] metric-only aliasing and tightened #[observation] usage rules.
  • Tests
    • Added label-alias end-to-end coverage and expanded macro diagnostics to validate alias/duplicate handling and observation syntax.

Walkthrough

The PR introduces outcome-specific MQTT metrics with structured context, propagates machine identifiers through managed-host publishing, adds BMS-specific outcome reporting, and supports metric-only label aliases while preserving log field names.

Changes

MQTT observability metrics

Layer / File(s) Summary
Outcome-specific MQTT metric contracts
crates/mqtt-common/src/metrics.rs
Defines managed-host and BMS events, recording methods, stable (component, status) labels, structured context, and coverage for emitted metrics and logs.
Managed publish correlation flow
crates/mqtt-common/src/hook.rs, crates/api-core/src/mqtt_state_change_hook/*
Carries machine_id through queued publishing and records managed-host queue, serialization, publish, timeout, and republish outcomes with topic and correlation context.
BMS outcome integration
crates/rack/src/bms_client.rs
Routes metadata parsing, payload serialization, publish success, publish errors, and timeouts through dedicated BMS metric methods.

Metric label aliasing

Layer / File(s) Summary
Metric label alias macro support
crates/instrument-macros/src/lib.rs
Parses and validates #[label(name = "...")], separates metric keys from log keys, detects resolved-name collisions, and rejects invalid observation or label arguments.
Alias behavior documentation and coverage
crates/instrument/src/lib.rs, crates/instrument/tests/matrix.rs
Documents metric-only aliases and verifies that metric keys change while generated log fields retain Rust identifiers.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant StateChangeHook
  participant QueuedMessage
  participant PublishWorker
  participant MQTTBroker
  participant MqttHookMetrics
  StateChangeHook->>QueuedMessage: enqueue payload, topic, machine_id, deadline
  PublishWorker->>QueuedMessage: consume queued message
  PublishWorker->>MQTTBroker: publish payload before deadline
  MQTTBroker-->>PublishWorker: success, error, or timeout
  PublishWorker->>MqttHookMetrics: record managed outcome with topic and machine_id
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main refactor: pairing DSX publish metrics and logs across mqtt and instrument crates.
Description check ✅ Passed The description is directly aligned with the change set and accurately describes the metric/log refactor and test coverage.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@chet
chet marked this pull request as ready for review July 21, 2026 00:32
@chet
chet requested review from a team and polarweasel as code owners July 21, 2026 00:32
@chet

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Understood — I’ll perform a complete review of PR #3752, including the instrumentation behavior, metric compatibility, macro diagnostics, tests, and documentation changes.

🐇🔍

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 17 minutes.

@github-actions

Copy link
Copy Markdown

@chet

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Hey @coderabbitai can you do a full_review when you get a chance? CICD isn't letting me continue because the CodeRabbit check "failed". Thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Understood. I’ve requested a complete re-review of #3752 to rerun the CodeRabbit check.

🐇🔍

✅ Action performed

Full review finished.

@chet chet changed the title refactor(mqtt,instrument): let DSX publish records move the existing counter refactor(mqtt,instrument): pair DSX publish metrics and logs Jul 21, 2026

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Understood — I’ve requested a complete re-review of #3752, including metric-contract compatibility, typed event behavior, macro diagnostics, tests, and documentation.

🐇🔍

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 20 minutes.

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

1 similar comment

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Requested a complete re-review of PR #3752, including metric-contract compatibility, typed-event behavior, macro diagnostics, tests, and documentation.

🐇🔍

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/instrument-macros/src/lib.rs`:
- Around line 569-585: Update classify_field and its diagnostic handling to
reject metadata arguments on #[context] and #[observation] attributes, allowing
only path-form attributes for those field kinds. Add explicit diagnostic cases
for the unsupported non-path forms, while preserving label metadata parsing and
existing field classification behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4a63be55-6612-4701-b732-cbcfe181392b

📥 Commits

Reviewing files that changed from the base of the PR and between 4ad44c6 and 231bb38.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (9)
  • crates/api-core/src/mqtt_state_change_hook/hook.rs
  • crates/api-core/src/mqtt_state_change_hook/republisher.rs
  • crates/instrument-macros/Cargo.toml
  • crates/instrument-macros/src/lib.rs
  • crates/instrument/src/lib.rs
  • crates/instrument/tests/matrix.rs
  • crates/mqtt-common/src/hook.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/rack/src/bms_client.rs

Comment thread crates/instrument-macros/src/lib.rs

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Requested a complete re-review of #3752, including metric-contract compatibility, typed-event behavior, macro diagnostics, tests, and documentation.

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/mqtt-common/src/metrics.rs (1)

366-366: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider record_managed_overflow for naming symmetry. Queue overflow is a managed-host-only outcome, yet this method drops the managed_ qualifier that its siblings (record_managed_success, record_managed_timeout, record_managed_publish_error) carry. Aligning the name now, while this public surface is fresh, keeps the API self-documenting and avoids a later rename touching every call site.

♻️ Proposed rename
-    pub fn record_overflow(&self, topic: String, machine_id: String, error: String) {
+    pub fn record_managed_overflow(&self, topic: String, machine_id: String, error: String) {

Update the call site in crates/api-core/src/mqtt_state_change_hook/hook.rs (record_overflowrecord_managed_overflow) and the test in this file accordingly.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mqtt-common/src/metrics.rs` at line 366, Rename the public method
record_overflow to record_managed_overflow in the metrics implementation for
naming consistency with its managed outcome siblings. Update every call site,
including the MQTT state-change hook, and adjust the related test to use the new
method name.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/mqtt-common/src/metrics.rs`:
- Line 366: Rename the public method record_overflow to record_managed_overflow
in the metrics implementation for naming consistency with its managed outcome
siblings. Update every call site, including the MQTT state-change hook, and
adjust the related test to use the new method name.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 157ef725-6dfd-49c9-81b3-2b43e75ba284

📥 Commits

Reviewing files that changed from the base of the PR and between 4ad44c6 and 6f066be.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (9)
  • crates/api-core/src/mqtt_state_change_hook/hook.rs
  • crates/api-core/src/mqtt_state_change_hook/republisher.rs
  • crates/instrument-macros/Cargo.toml
  • crates/instrument-macros/src/lib.rs
  • crates/instrument/src/lib.rs
  • crates/instrument/tests/matrix.rs
  • crates/mqtt-common/src/hook.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/rack/src/bms_client.rs

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Requested a complete re-review of #3752, including metric-contract compatibility, typed-event behavior, macro diagnostics, tests, and documentation.

🐇🔍

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/instrument-macros/src/lib.rs`:
- Around line 347-349: Update the Meta::Path branch in the attribute-name
parsing logic to validate ident.to_string() with the same metric-key rules used
for name aliases, rejecting non-ASCII names consistently. Add a regression test
covering a bare label with a non-ASCII identifier.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 82559268-d421-404b-a70e-4c98fe4f43d6

📥 Commits

Reviewing files that changed from the base of the PR and between 6f066be and cb70aca.

📒 Files selected for processing (8)
  • crates/api-core/src/mqtt_state_change_hook/hook.rs
  • crates/api-core/src/mqtt_state_change_hook/republisher.rs
  • crates/instrument-macros/src/lib.rs
  • crates/instrument/src/lib.rs
  • crates/instrument/tests/matrix.rs
  • crates/mqtt-common/src/hook.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/rack/src/bms_client.rs
🚧 Files skipped from review as they are similar to previous changes (7)
  • crates/api-core/src/mqtt_state_change_hook/hook.rs
  • crates/instrument/src/lib.rs
  • crates/instrument/tests/matrix.rs
  • crates/rack/src/bms_client.rs
  • crates/api-core/src/mqtt_state_change_hook/republisher.rs
  • crates/mqtt-common/src/hook.rs
  • crates/mqtt-common/src/metrics.rs

Comment thread crates/instrument-macros/src/lib.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
crates/instrument-macros/src/lib.rs (1)

339-385: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Bare #[label] names still bypass the metric-key ASCII guard.

Meta::Path(_) => Ok(ident.to_string()) returns the Rust identifier verbatim with no validation, while the name = "..." alias branch enforces ASCII-letter/underscore-start plus ASCII-alphanumeric/underscore rest. Rust identifiers may legally contain non-ASCII Unicode characters (stable non_ascii_idents), so a bare-labeled field can reach the metric label path with a key that violates the same contract the alias enforces.

♻️ Proposed fix to share the validation
     match &attr.meta {
-        Meta::Path(_) => Ok(ident.to_string()),
+        Meta::Path(_) => {
+            let name = ident.to_string();
+            validate_metric_label_name(&name, ident)?;
+            Ok(name)
+        }
         Meta::List(_) => {

Factor the ASCII check (currently inline in the Meta::List branch) into a shared validate_metric_label_name(&str, &impl ToTokens) -> syn::Result<()> helper and call it from both branches.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/instrument-macros/src/lib.rs` around lines 339 - 385, Factor the ASCII
metric-label name validation from label_metric_name into a shared
validate_metric_label_name helper, accepting the name and an appropriate
ToTokens span source for diagnostics. Invoke it for both bare #[label]
identifiers and name = "..." aliases, preserving the existing error message and
returning the original name only after validation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@crates/instrument-macros/src/lib.rs`:
- Around line 339-385: Factor the ASCII metric-label name validation from
label_metric_name into a shared validate_metric_label_name helper, accepting the
name and an appropriate ToTokens span source for diagnostics. Invoke it for both
bare #[label] identifiers and name = "..." aliases, preserving the existing
error message and returning the original name only after validation.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d5f44b56-9628-414c-b667-5233ea2afa4f

📥 Commits

Reviewing files that changed from the base of the PR and between 4f01182 and cb70aca.

📒 Files selected for processing (8)
  • crates/api-core/src/mqtt_state_change_hook/hook.rs
  • crates/api-core/src/mqtt_state_change_hook/republisher.rs
  • crates/instrument-macros/src/lib.rs
  • crates/instrument/src/lib.rs
  • crates/instrument/tests/matrix.rs
  • crates/mqtt-common/src/hook.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/rack/src/bms_client.rs

The API state-change hook, managed-host republisher, and rack BMS worker counted DSX publish outcomes separately from their matching diagnostics.

Typed sibling `Event`s now pair those signals while preserving the frozen `{component,status}` series and each publisher's historical log behavior. Operational detail remains log-only. The narrow `#[label(name = "component")]` alias keeps the existing metric key without colliding with the Event log's reserved `component` identity, and field-attribute diagnostics reject unsupported metadata instead of silently ignoring it.

Table-driven tests cover the outcome matrix and alias guardrails.

This supports NVIDIA#3732

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Requested a complete re-review of #3752, covering metric-contract compatibility, typed-event behavior, macro diagnostics, tests, and documentation.

🐇🔍

✅ Action performed

Full review finished.

@chet
chet enabled auto-merge (squash) July 21, 2026 17:09
@chet
chet merged commit 8ffed51 into NVIDIA:main Jul 21, 2026
61 checks passed
@chet
chet deleted the gh-issue-3732 branch July 21, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants