Skip to content

refactor(site-explorer): pair iteration latency with failure logs#3748

Merged
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3737
Jul 21, 2026
Merged

refactor(site-explorer): pair iteration latency with failure logs#3748
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3737

Conversation

@chet

@chet chet commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Site Explorer recorded every iteration's latency and separately wrote the ERROR record for a failed pass.

SiteExplorerIterationFinished now owns the label-free histogram and emits the historical diagnostic only for post-lock failures. Success and the early lock-acquisition return remain metric-only, so normal replica contention keeps its existing caller-owned WARN without gaining a misleading ERROR.

Tests preserve the Prometheus family, description, unit, bucket layout, and one-sample-per-terminal-path contract.

Related issues

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

A check_values table covers successful and failed Event emissions. Separate tests pin the Prometheus family name, description, unit suffix, label-free shape, and the custom 30-second latency bucket. The existing lock-path test remains explicit.

Verified with:

  • cargo test -p carbide-site-explorer
  • cargo make format-nightly
  • cargo make clippy
  • cargo make carbide-lints
  • cargo make check-event-names
  • cargo xtask check-metric-docs

Additional Notes

CodeRabbit asked whether the Event's _milliseconds metric name would bypass the existing latency view. The framework removes that unit suffix before registering the OpenTelemetry instrument, so the registered carbide_site_explorer_iteration_latency name still matches the view; the exposition and 30-second bucket tests cover both sides of that contract. The thread was resolved without a code change.

Closes #3737

@chet
chet requested a review from a team as a code owner July 20, 2026 23:12
@chet

chet commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@chet I’ll perform a complete re-review of all changes in #3748. 🐇

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 4 minutes.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c50555fa-fcb9-4ede-a3cc-0fa9cf519607

📥 Commits

Reviewing files that changed from the base of the PR and between 4ad44c6 and 7cd3006.

📒 Files selected for processing (2)
  • crates/site-explorer/src/lib.rs
  • crates/site-explorer/src/metrics.rs

Summary by CodeRabbit

  • Monitoring

    • Improved site exploration iteration metrics with more consistent latency reporting.
    • Failure events now include clearer diagnostic details in monitoring logs.
    • Successful iterations continue recording latency without generating error logs.
  • Bug Fixes

    • Ensured iteration completion is recorded even when work cannot be acquired.
    • Preserved the established Prometheus metric naming and format.

Walkthrough

Site Explorer now emits a completion event for each iteration, recording latency and logging failures. The legacy iteration-latency instrument and duplicate failure log were removed, with tests covering metrics, logs, and Prometheus naming.

Changes

Site Explorer telemetry

Layer / File(s) Summary
Iteration-finished event and metric migration
crates/site-explorer/src/metrics.rs
Adds SiteExplorerIterationFinished, conditionally emits failure logs, replaces the legacy iteration-latency instrument, and tests metric and Prometheus behavior.
Iteration lifecycle emission
crates/site-explorer/src/lib.rs
Emits completion telemetry after lock-acquisition failures and exploration results, while removing the standalone exploration failure log.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant run_single_iteration
  participant SiteExplorer_emit_iteration_finished
  participant SiteExplorerIterationFinished
  participant carbide_instrument
  run_single_iteration->>SiteExplorer_emit_iteration_finished: pass elapsed latency and optional error
  SiteExplorer_emit_iteration_finished->>SiteExplorerIterationFinished: emit completion event
  SiteExplorerIterationFinished->>carbide_instrument: record latency histogram
  SiteExplorerIterationFinished->>carbide_instrument: emit ERROR log when error is present
Loading

Possibly related issues

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: pairing site-explorer iteration latency with failure logging.
Description check ✅ Passed The description is directly aligned with the refactor, metric ownership, logging behavior, and test coverage in this changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@chet

chet commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@chet I’ll perform a complete re-review of all changes in #3748.

🐇🔍

✅ Action performed

Full review finished.

@chet chet changed the title refactor(site-explorer): emit iteration outcomes refactor(site-explorer): pair iteration latency with real run failures Jul 20, 2026
@chet

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet I’ll perform a complete re-review of all changes in #3748.

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/site-explorer/src/metrics.rs`:
- Line 363: The metric selector assigned to metric_name must match the renamed
carbide_site_explorer_*_latency instrument view so custom histogram buckets are
used. Update the selector or instrument name consistently with the setup in
logging configuration, and add an end-to-end regression test covering the
exported metric path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fcf090d0-42e5-46e2-8586-a1010ee95594

📥 Commits

Reviewing files that changed from the base of the PR and between dcdd5da and c3e0f95.

📒 Files selected for processing (2)
  • crates/site-explorer/src/lib.rs
  • crates/site-explorer/src/metrics.rs

Comment thread crates/site-explorer/src/metrics.rs
@chet chet changed the title refactor(site-explorer): pair iteration latency with real run failures refactor(site-explorer): pair iteration latency with failure logs Jul 21, 2026
Site Explorer recorded every iteration's latency and separately wrote the ERROR record for a failed pass.

`SiteExplorerIterationFinished` now owns the label-free histogram and emits the historical diagnostic only for post-lock failures. Success and the early lock-acquisition return remain metric-only, so normal replica contention keeps its existing caller-owned WARN without gaining a misleading ERROR.

Tests preserve the Prometheus family, description, unit, bucket layout, and one-sample-per-terminal-path contract.

This supports NVIDIA#3737

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet I’ll perform a complete re-review of all changes in #3748.

🐇🔍

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 20 minutes.

@chet

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Hey @coderabbitai can you do a full_review when you get a chance? CICD isn't letting me continue because the CodeRabbit check "failed". Thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet I’ve triggered a complete re-review of #3748 to refresh the CodeRabbit check.

🐇🔍

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 12 minutes.

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet I’ve triggered a complete re-review of #3748 to refresh the CodeRabbit check.

🐇🔍

✅ Action performed

Full review finished.

@chet
chet merged commit 03fbea2 into NVIDIA:main Jul 21, 2026
62 checks passed
@chet
chet deleted the gh-issue-3737 branch July 21, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unify Site Explorer iteration latency and failure logs

2 participants