fix(credentials): harden credential storage drivers - #3
Open
varshaprasad96 wants to merge 6 commits into
Open
Conversation
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
… performance Address review findings from the credential storage drivers PR: - Route additional_credentials through the driver on refresh to prevent silent data loss for multi-credential providers (e.g. AWS STS) - Clean up stored credential handles on CAS failure during refresh to prevent orphaned secrets in external backends - Enforce namespace validation in the Kubernetes Secrets driver to prevent cross-namespace credential access when allow_reference_namespace is not enabled - Cache Vault Kubernetes auth tokens with 80% TTL to avoid re-authenticating on every credential operation - Parallelize resolve_credentials in all three drivers using try_join_all for faster sandbox startup - Add existingSecret support for the KEK Secret to fix helm template/GitOps workflows where lookup returns empty and regenerates the key - Document RBAC blast radius for the Kubernetes Secrets credential driver and recommend a dedicated namespace Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
…rallelize operations Use resourceVersion optimistic concurrency with retry loop for K8s Secret ownership checks to prevent TOCTOU races. Switch Vault token cache from RwLock to Mutex with double-check pattern to prevent thundering herd on cache miss. Parallelize credential store and delete operations across independent keys using try_join_all. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
…ate cleanup Replace try_join_all with join_all in credential store/delete operations to handle partial failures — successfully-stored handles are cleaned up when another key fails. Add retry loop with conflict detection to db-credstore delete_credential, matching the K8s driver pattern. Consolidate 4 manual cleanup_pre_stored_provider_credentials call sites into a single error handler using an async block. Remove inconsistent .trim() from db-credstore validate_handle_owner. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Remove attempt-count guard from 409/Aborted match arms in retry loops so the post-loop Status::aborted error is reachable after exhausting retries. Previously, last-attempt conflicts fell through to the catch-all error arm, producing misleading Status::unavailable errors. Remove duplicate validation calls that ran after prepare_provider_credential_update but outside the cleanup-protected async block, which would leak pre-stored handles on failure. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
sjenning
force-pushed
the
1931-credential-drivers-implementation
branch
from
July 27, 2026 18:58
d1e10d1 to
052540f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardening fixes for the credential storage drivers (NVIDIA#2437), addressing correctness, performance, and operational issues identified during review.
Preconditions(uid + resourceVersion) on delete and resourceVersion on patch to prevent silent overwrites from concurrent modificationsRwLocktoMutexwith double-check pattern, preventing simultaneous login stormsjoin_all(replacing sequential loops), with partial failure cleanup of successfully-stored handlesapply_minted_credentialnow cleans up stored handles on CAS failureattempt + 1 < CONFLICT_RETRY_LIMITguard that made post-loop error unreachabledefaultKeyEncryptionKeyHelm value for deterministic key encryption in GitOps workflowsRelated Issue
Hardens PR NVIDIA#2437 — addresses 8 of 11 gaps identified during review.
Changes
openshell-driver-kubernetes-secrets/src/lib.rsopenshell-driver-vault/src/lib.rsopenshell-driver-db-credstore/src/lib.rs.trim()openshell-server/src/credentials.rsjoin_allwith partial failure cleanup, parallel deletesopenshell-server/src/grpc/provider.rsopenshell-server/src/provider_refresh.rsdeploy/helm/docs/reference/gateway-config.mdxTesting
mise run pre-commitpassesmise run testpasses58 > 19chars — unrelated to these changes)Known remaining gaps (not addressed here)
additional_credentialsfrom refresh bypass the driver (AWS STS session tokens silently dropped)allow_reference_namespaceparsed but never enforced at runtime