refactor(dpa-manager): pair monitor latency with failure logs

[chet]

Exit code: 0
Wall time: 0.5 seconds
Output:
The DPA monitor recorded iteration latency separately from the WARN that explained a failed pass.

This closes each pass with one histogram-backed Event inside its iteration span. Successful passes remain metric-only; failed passes retain the historical warning and error context with the pass's span_id. Work-lock diagnostics, cadence changes, gauges, and batched counters keep their existing behavior.

carbide_dpa_monitor_iteration_latency_milliseconds, its HELP text, fractional-millisecond precision, and label-free series do not change. Table-driven tests cover successful and failed emissions plus the Prometheus contract.

Related issues

• Supports Pair DPA monitor latency with failed-pass logs #3791
• Part of Unify logging and metrics behind a single instrumentation standard #3169

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

• This PR contains breaking changes

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed
• No testing required (docs, internal refactor, etc.)

Verified with:

• cargo test -p carbide-dpa-manager metrics::tests
• cargo make format-nightly
• cargo make clippy
• cargo make carbide-lints
• cargo xtask check-event-names
• cargo xtask check-metric-docs
• cargo xtask check-workspace-deps
• cargo make --no-workspace check-licenses
• cargo make --no-workspace check-bans

Additional Notes

docs/observability/core_metrics.md is generated from the Event declaration and intentionally stays with the code change.

Closes #3791

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5b36ec6d-1518-49ee-8bdb-d2cece815b0c

📥 Commits

Reviewing files that changed from the base of the PR and between aab093d and 820731d.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

• crates/dpa-manager/Cargo.toml
• crates/dpa-manager/src/lib.rs
• crates/dpa-manager/src/metrics.rs
• docs/observability/core_metrics.md

🚧 Files skipped from review as they are similar to previous changes (3)

• docs/observability/core_metrics.md
• crates/dpa-manager/Cargo.toml
• crates/dpa-manager/src/metrics.rs

---

Summary by CodeRabbit

• Observability
  • Added event-driven monitoring for DPA monitor iteration completion, including per-iteration latency measurements and error details.
  • Successful iterations now emit metrics without warning logs; failed iterations emit warning logs alongside the same latency data.
  • Documented the new carbide_dpa_monitor_iteration_latency_milliseconds histogram metric.

Walkthrough

The DPA monitor now reports iteration completion through a carbide-instrument event, conditionally logs failures, preserves latency histogram output, simplifies counter instrumentation, and documents the latency metric. Tests cover success, failure, and exposition behavior.

Changes

DPA iteration observability

Layer / File(s)	Summary
Iteration event and instrumentation contract crates/dpa-manager/Cargo.toml, crates/dpa-manager/src/metrics.rs	Adds DpaMonitorIterationFinished, conditionally emits warnings for errors, moves latency handling into event instrumentation, and limits direct instruments to counters.
Monitor loop integration crates/dpa-manager/src/lib.rs	Emits iteration latency and error context from the monitor loop, while adjusting the interval only after successful iterations with changes.
Metric validation and documentation crates/dpa-manager/src/metrics.rs, docs/observability/core_metrics.md	Tests successful and failed event behavior, histogram exposition stability, and documents the iteration latency histogram.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• NVIDIA/infra-controller issue 3791 — Covers DPA iteration event instrumentation, failure logging, latency preservation, and tests.
• NVIDIA/infra-controller issue 3790 — Applies the corresponding event-backed latency and failure-logging pattern to the DPA monitor.
• NVIDIA/infra-controller issue 3792 — Connects iteration latency histograms with failure logging through completion events.

Possibly related PRs

• NVIDIA/infra-controller#3748 — Refactors iteration completion handling with event-backed latency metrics and conditional error logging.
• NVIDIA/infra-controller#3800 — Applies the analogous monitor iteration event and cadence changes to another monitor.
• NVIDIA/infra-controller#3802 — Uses completion events for latency metrics, warning logs, instrument simplification, and success-based cadence updates.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main refactor: coupling monitor latency reporting with failure logs.
Description check	✅ Passed	The description matches the changeset and clearly describes the metric/event refactor, tests, and unchanged behavior.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pull-request-3799.docs.buildwithfern.com/infra-controller

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/dpa-manager/src/metrics.rs`:
- Around line 248-277: Add a fractional-latency case to the iteration metrics
tests around the existing “successful iteration stays silent” and “failed
iteration retains the warning” cases, using Duration::from_micros(...) and
asserting the histogram_sum_delta preserves the fractional millisecond value.
Keep the expected logging behavior and histogram count consistent with the
corresponding iteration outcome.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 73fa455a-bee5-4131-ab31-b3e58a6376f4

📥 Commits

Reviewing files that changed from the base of the PR and between 9720d3c and aab093d.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

• crates/dpa-manager/Cargo.toml
• crates/dpa-manager/src/lib.rs
• crates/dpa-manager/src/metrics.rs
• docs/observability/core_metrics.md