Skip to content

refactor(api,bmc-proxy): pair TLS failure records with their counters#3741

Merged
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3728
Jul 21, 2026
Merged

refactor(api,bmc-proxy): pair TLS failure records with their counters#3741
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3728

Conversation

@chet

@chet chet commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

The API and BMC proxy TLS listeners incremented their existing failure counters separately from the ERROR records that explained those failures.

Typed sibling Events now pair TCP accept, certificate reload, and handshake failures with their historical records. Counter names, reason labels, messages, peer addresses, and error context remain unchanged; connection attempts and successes stay metric-only rather than adding new log volume.

The branch also moves the merged #3743 firmware-upload outcome cases into a table-driven test. Production behavior in that area is unchanged.

Related issues

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

The API and BMC proxy failure matrices use check_values to verify that every reason produces one counter increment and one historical ERROR record. Since #3743 merged while this branch was open, it also converts that PR's seven firmware-upload outcome cases to check_values; the multipart fallback remains explicit because the ordered pair of attempts is the behavior under test.

Verified with:

  • the focused carbide-api-core TLS failure Event test
  • the focused carbide-bmc-proxy TLS failure Event test
  • the focused carbide-preingestion-manager firmware-upload outcome test
  • cargo make format-nightly
  • cargo make clippy
  • cargo make carbide-lints
  • cargo make check-event-names
  • cargo xtask check-metric-docs

Additional Notes

The firmware-upload changes are test-only; production behavior in that area remains the version merged through #3743.

Closes #3728

@chet
chet requested a review from a team as a code owner July 20, 2026 22:17
@chet

chet commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@chet: Starting a complete review of PR #3741.

🐇🔍

✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Summary by CodeRabbit

  • Monitoring Improvements
    • Added dedicated ERROR-level events for inbound connection failures, including listener accept errors, TLS handshake/connection failures, and TLS certificate reload failures.
    • Standardized failure tracking with shared “reason” labels and ensured error details (and peer address where applicable) are emitted as structured diagnostics.
  • Tests
    • Updated assertions to validate ERROR log event names/messages, structured fields, and expected metric counter increments for each failure scenario.
    • Refined firmware upload metric verification to use table-driven scenarios with expected logs and counter deltas.

Walkthrough

The API listener and BMC proxy now emit structured events for TCP accept, TLS handshake, and certificate reload failures. Events include reason labels and diagnostic context, while tests validate logs and metric counter increments. Firmware upload assertions now use structured observations.

Changes

Failure instrumentation

Layer / File(s) Summary
API listener failure events
crates/api-core/src/listener.rs
Adds TcpAcceptFailed, enriches TlsConnectionFailed, and updates TCP accept and TLS handshake error paths. Tests validate structured logs and metric increments.
BMC proxy failure events
crates/bmc-proxy/src/bmc_proxy.rs
Adds TCP accept and certificate reload events, and records error and peer address context for TLS failures. Tests validate event fields and counters.
Firmware upload observation tests
crates/preingestion-manager/src/metrics.rs
Refactors firmware upload metric and historical-log assertions around structured scenario inputs and observations.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main refactor: pairing TLS failure records with their counters in api and bmc-proxy.
Description check ✅ Passed The description accurately matches the TLS instrumentation refactor and the test-only firmware-upload table-driven changes.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@chet chet changed the title refactor(tls): emit connection failure events refactor(api,bmc-proxy): keep accept, reload, and handshake failures in sync Jul 20, 2026
@github-actions

Copy link
Copy Markdown

@chet chet changed the title refactor(api,bmc-proxy): keep accept, reload, and handshake failures in sync refactor(api,bmc-proxy): pair TLS failure records with their counters Jul 21, 2026

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Starting a complete review of PR #3741.

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 21 minutes.

@chet

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Hey @coderabbitai can you do a full_review when you get a chance? CICD isn't letting me continue because the CodeRabbit check "failed". Thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Triggering a complete re-review of #3741 now.

🐇🔍

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 15 minutes.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Starting a complete review of PR #3741.

🐇🔍

✅ Action performed

Full review finished.


Your plan includes PR reviews subject to rate limits. More reviews will be available in 15 minutes.

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Triggering a complete re-review of #3741 now. 🐇🔍

✅ Action performed

Full review finished.

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Triggering a complete re-review of #3741 now.

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/preingestion-manager/src/metrics.rs`:
- Around line 691-698: Remove the redundant method_label and outcome_label
fields from FirmwareUploadInput, and update expected_firmware_upload and all
callers to derive labels directly from the method and outcome enum values via
their existing label conversion (such as label_value()). Eliminate the
corresponding raw string parameters and initializers while preserving the
current metric labels and expected behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 83e53cb6-5abf-494c-abf8-afa125541912

📥 Commits

Reviewing files that changed from the base of the PR and between 4f01182 and e15008c.

📒 Files selected for processing (3)
  • crates/api-core/src/listener.rs
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/preingestion-manager/src/metrics.rs

Comment thread crates/preingestion-manager/src/metrics.rs
The API and BMC proxy TLS listeners incremented their existing failure counters separately from the ERROR records that explained those failures.

Typed sibling `Event`s now pair TCP accept, certificate reload, and handshake failures with their historical records. Counter names, reason labels, messages, peer addresses, and error context remain unchanged; connection attempts and successes stay metric-only rather than adding new log volume.

The branch also moves the merged NVIDIA#3743 firmware-upload outcome cases into a table-driven test. Production behavior in that area is unchanged.

This supports NVIDIA#3728

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

chet commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chet: Triggering a complete re-review of #3741 now. 🐇🔍

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/preingestion-manager/src/metrics.rs (1)

772-932: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Model expected historical logs as a typed variant.

Option<(Level, &str, &str)> permits invalid level/event-name/message combinations for a closed set of records. Replace it with an ExpectedFirmwareUploadLog enum that derives those fields, and use that enum in the cases.

As per coding guidelines, “avoid stringly-typed values when a value has a known finite set of possibilities.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/preingestion-manager/src/metrics.rs` around lines 772 - 932, Replace
the tuple-based log parameter in expected_firmware_upload with an
ExpectedFirmwareUploadLog enum representing each valid firmware-upload
historical log variant. Derive the tracing level, event name, and message from
the enum inside expected_firmware_upload, and update all test cases to pass the
corresponding typed variant instead of raw tuples.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/preingestion-manager/src/metrics.rs`:
- Around line 772-932: Replace the tuple-based log parameter in
expected_firmware_upload with an ExpectedFirmwareUploadLog enum representing
each valid firmware-upload historical log variant. Derive the tracing level,
event name, and message from the enum inside expected_firmware_upload, and
update all test cases to pass the corresponding typed variant instead of raw
tuples.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6eceef00-cd1e-47dd-99c1-262bbc8a159f

📥 Commits

Reviewing files that changed from the base of the PR and between 4bf329d and b1b03b4.

📒 Files selected for processing (3)
  • crates/api-core/src/listener.rs
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/preingestion-manager/src/metrics.rs

@chet
chet enabled auto-merge (squash) July 21, 2026 17:09
@chet
chet merged commit 9984e44 into NVIDIA:main Jul 21, 2026
61 checks passed
@chet
chet deleted the gh-issue-3728 branch July 21, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unify TLS connection failure logs and metrics behind Events

4 participants