Skip to content

fix(nvca): render only the selected OTel collector auth extension#233

Open
cr7258 wants to merge 2 commits into
NVIDIA:mainfrom
cr7258:cr7258/fix/nvca-otel-sak-auth
Open

fix(nvca): render only the selected OTel collector auth extension#233
cr7258 wants to merge 2 commits into
NVIDIA:mainfrom
cr7258:cr7258/fix/nvca-otel-sak-auth

Conversation

@cr7258

@cr7258 cr7258 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

TL;DR

When the NVCA OTel Collector uses SAK-based bearer-token authentication, nvca-operator still includes the unused oauth2client extension in the generated configuration. Because the OAuth2 Token URL is empty in SAK mode, configuration validation fails, and the nvca-otel-collector container enters CrashLoopBackOff, preventing Kubernetes events from being exported to the Event Ledger.

Additional Details

The operator already determines the OTel Collector authentication mode from the NVCFBackend configuration:

  • OAuth2 is selected when Vault is enabled and an OAuth client ID is configured.
  • Otherwise, bearertokenauth is selected for SAK-based authentication.

However, otel_collector_config.yaml previously rendered both authentication extensions regardless of the selected mode:

extensions:
  bearertokenauth:
    filename: "${env:NGC_SERVICE_API_KEY_FILE}"

  oauth2client:
    client_id: ${env:NVCA_OTEL_COLLECTOR_OAUTH_CLIENT_ID}
    client_secret_file: ${env:NVCA_OTEL_COLLECTOR_OAUTH_CLIENT_SECRET_FILE}
    token_url: "${env:NVCA_OTEL_COLLECTOR_OAUTH_TOKEN_URL}"
    scopes: ["write"]

In SAK mode, the Collector is configured to authenticate to the Event Ledger using the NGC service API key through bearertokenauth. OAuth2 is not selected, so NVCA_OTEL_COLLECTOR_OAUTH_TOKEN_URL is populated with an empty value.

Although the exporter selects bearertokenauth, the OTel Collector validates every configured extension during startup. Validation of the unused oauth2client extension therefore fails with:

Configuration references empty environment variable
{"name": "NVCA_OTEL_COLLECTOR_OAUTH_TOKEN_URL"}

Error: invalid configuration: extensions::oauth2client:
no TokenURL provided in OAuth Client Credentials configuration

As a result, the nvca-otel-collector container enters CrashLoopBackOff and never becomes ready.

This PR changes the embedded Collector configuration into a mode-aware template:

  • OAuth2 mode renders only oauth2client.
  • SAK mode renders only bearertokenauth.
  • otelCollectorConfigTemplateData provides the selected authentication mode to the template.
  • ConfigMap reconciliation renders the configuration before creating or updating the ConfigMap.
  • Template parsing and rendering errors are returned through the reconciliation path.
  • Tests parse the rendered YAML and verify that each authentication mode contains only its expected extension and environment-variable references.

Issues

NO-REF
Nvbug: https://nvbugspro.nvidia.com/bug/6470916

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Summary by CodeRabbit

  • New Features
    • OpenTelemetry collector configuration now conditionally enables either OAuth2 or bearer-token authentication based on backend settings.
    • Rendered collector configuration automatically reflects the selected authentication method.
  • Bug Fixes
    • Improved collector ConfigMap generation by rendering the embedded YAML template and returning clearer errors on parse/render failures.
  • Tests
    • Expanded/updated coverage to validate both auth modes by parsing the rendered config.yaml, including correct handling of empty OAuth client ID when vault is disabled.

@cr7258
cr7258 requested a review from a team as a code owner July 17, 2026 09:59
@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a58c80f4-4e3d-48c1-a7bd-c5f70dda1285

📥 Commits

Reviewing files that changed from the base of the PR and between 243d052 and 1e2e4bf.

📒 Files selected for processing (3)
  • src/compute-plane-services/nvca/pkg/operator/reconcile/nvcaagent_reconcile_test.go
  • src/compute-plane-services/nvca/pkg/operator/reconcile/otel_reconcile.go
  • src/compute-plane-services/nvca/pkg/operator/reconcile/otel_reconcile_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/compute-plane-services/nvca/pkg/operator/reconcile/otel_reconcile_test.go

📝 Walkthrough

Walkthrough

The OTel collector configuration is rendered per backend, selecting OAuth2 or bearer-token authentication. ConfigMap reconciliation propagates rendering errors, and tests validate authentication modes, client IDs, and collector sections.

Changes

OTel authenticator configuration

Layer / File(s) Summary
Backend-driven template selection
src/compute-plane-services/nvca/pkg/operator/reconcile/otel_reconcile.go, src/compute-plane-services/nvca/pkg/operator/reconcile/manifests/otel_collector_config.yaml
The embedded collector template is rendered with backend OAuth2 eligibility and emits either oauth2client or bearertokenauth.
ConfigMap wiring and authentication coverage
src/compute-plane-services/nvca/pkg/operator/reconcile/nvcaagent_reconcile.go, src/compute-plane-services/nvca/pkg/operator/reconcile/otel_reconcile_test.go, src/compute-plane-services/nvca/pkg/operator/reconcile/nvcaagent_reconcile_test.go
ConfigMap setup stores rendered data and wraps rendering errors; tests validate both authentication modes, empty OAuth client IDs, and required collector sections.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant BackendReconciler
  participant OTelTemplate
  participant ConfigMap
  BackendReconciler->>OTelTemplate: Render with backend OAuth2 eligibility
  OTelTemplate-->>BackendReconciler: Return config.yaml data
  BackendReconciler->>ConfigMap: Store rendered configuration
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: rendering only the selected OTel Collector auth extension.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"


Comment @coderabbitai help to get the list of available commands.

@cr7258 cr7258 changed the title fix(nvca): render OTel auth extension by mode fix(nvca): render only the selected OTel collector auth extension Jul 17, 2026
@cr7258
cr7258 force-pushed the cr7258/fix/nvca-otel-sak-auth branch from 4d58023 to dc756fa Compare July 17, 2026 10:03
@cr7258

cr7258 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

The Go and Rust CodeQL CI failures on this PR were caused by the fork workflow attempting to post PR comments with a read-only GITHUB_TOKEN. The scans themselves completed successfully; only the comment step failed. This is addressed by #234. Once #234 is merged, rerunning the CodeQL checks should resolve the failures.

Signed-off-by: Seven Cheng <sevenc@nvidia.com>
@cr7258
cr7258 force-pushed the cr7258/fix/nvca-otel-sak-auth branch from dc756fa to 243d052 Compare July 17, 2026 15:15

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please also clean this client ID placeholder? keep it as empty.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed client ID placeholder: 1e2e4bf

Signed-off-by: Seven Cheng <sevenc@nvidia.com>
@cr7258
cr7258 requested a review from along-2017 July 18, 2026 01:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants