test(provisioner): real-backend integration round-trips for Deprovision/Regrade DDL (toward 100% integration)#45
Merged
mastermanas805 merged 1 commit intoJun 4, 2026
Conversation
…on/Regrade DDL (toward 100% integration) Adds env-gated real-backend integration tests that drive the gRPC server layer's Provision → Regrade → Deprovision lifecycle against a REAL Postgres and a REAL Redis, through the actual RPC handlers (not injected fakes). Motivation: the truehomie-db DROP incident class (2026-06-03). Every existing server test injects a fake backend, so the real DROP DATABASE / DROP USER / ALTER ROLE DDL had never executed through the gRPC handler path (breaker wrapping, tier→connLimit routing, mapError, response shaping, idempotent re-deprovision). High statement coverage from mocks did not prove the destroy/regrade DDL was correct end-to-end. Tests (all skip cleanly when the backend URL is unset): - TestServer_Postgres_Provision_Regrade_Deprovision_LiveRoundTrip: asserts db_/usr_ created, role CONNECTION LIMIT actually ALTERed (pg_roles cross- check), db_/usr_ DROPped, and a second Deprovision is a clean idempotent no-op (#9 DROP IF EXISTS). - TestServer_Postgres_Reprovision_AfterDeprovision_LiveRoundTrip: re-provisions the same token after teardown — guards a partial-DROP leak that would block reuse. - TestServer_Redis_Provision_Deprovision_LiveRoundTrip: ACL user created, ACL user + namespace keys reaped, idempotent second Deprovision. Coverage delta (integration-only, server test pkg vs ./... per the §1.4 mechanism): server package 18.9% → 24.8% (+5.9pp). The backend DDL methods, as exercised FROM the gRPC layer, go 0% → 60-83% (postgres Provision 0→68.3%, Deprovision 0→62.5%, Regrade 0→73.3%; redis Provision 0→70.0%, Deprovision 0→83.3%). Gate: go build/vet/test ./... -short green; gofmt + golangci-lint clean. Env-blocked: mongo/queue(NATS)/storage backend round-trips remain GAP (no local TEST_MONGO_URL/TEST_NATS_URL) — out of scope for this increment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mastermanas805
added a commit
that referenced
this pull request
Jun 4, 2026
…omie DROP-incident class) (#46) PR #45 proved single-tenant Provision/Regrade/Deprovision/idempotency round-trips for postgres + redis through the real gRPC handlers. It did NOT prove SCOPING: that a deprovision is confined to the target tenant. That is the gap the 2026-06-03 truehomie-db DROP incident exposed (an active Pro customer's db+role dropped while a co-resident tenant shared the cluster). Adds server_multitenant_scoping_test.go: provisions TWO co-resident tenants A+B through the genuine gRPC ProvisionResource handler against a real Postgres / real Redis, seeds data into each, deprovisions ONLY A, and asserts B fully survives. - Postgres: after Deprovision(A), A's db_/usr_ are gone (DROP ran) AND B's database + role still exist, B's seeded row is intact, and B can still CONNECT with its own ConnectionUrl credentials. - Redis: after Deprovision(A), A's ACL user + namespace key are reaped AND B's ACL user + namespace key + value survive. Env-gated identically to server_live_roundtrip_test.go (skips clean under -short / no backend; runs for real in coverage.yml's pg+redis services and local dev backends). Verified PASS locally against Postgres 16 + Redis 7. Coverage block: Symptom: unscoped DROP DATABASE/DROP USER (or ACL DELUSER/SCAN+DEL) on deprovision takes out a co-resident tenant (truehomie 2026-06-03) Enumeration: gRPC DeprovisionResource handler path, postgres+redis LocalBackend Sites found: 2 (postgres deprovision, redis deprovision) Sites touched: 2 (both have a co-resident-survival regression test) Coverage test: TestServer_{Postgres,Redis}_Deprovision_IsScopedToTargetTenant Live verified: PASS vs local Postgres 16 + Redis 7 (real backends); B survives A Co-authored-by: Manas Srivastava <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mastermanas805
added a commit
that referenced
this pull request
Jun 4, 2026
…age (Wave 4) (#47) * test(provisioner): real-backend gRPC round-trips for Mongo/Queue/Storage (Wave 4) Closes the remaining cells of the provisioner gRPC × backend matrix (INTEGRATION-COVERAGE-PLAN §2.3 / Wave 4). Postgres + Redis server-layer round-trips already shipped (#45, #46); this adds the real-backend Provision → assert artifact → Deprovision → assert-gone lifecycle for the three remaining backends, all driven through the genuine gRPC handlers (breaker wrapping, tier routing, mapError, response shaping): - Mongo: ProvisionResource creates usr_/db_ on a real MongoDB, GetStorageBytes reads real dbStats (>0 after seeding), DeprovisionResource runs the real dropUser/dropDatabase (truehomie DROP-incident class), second Deprovision is a clean idempotent no-op, and Regrade(mongo) asserts the documented skip path. - Queue (NATS): ProvisionResource passes the real NATS monitor health check and returns nats:// URL + subject prefix, GetStorageBytes(queue)=0 (message-metered), Deprovision is the shared-backend no-op, idempotent. - Storage (MinIO/S3): GetStorageBytes object-walk — empty prefix=0, after a real PutObject=exact byte count, after delete=0. (Storage Provision/Deprovision are API-side; provisioner only meters.) All tests env-gated (skip cleanly under `go test -short`, the deploy gate; run for real when the backend env is present). CI: added NATS service container + MinIO docker-run step + the TEST_NATS_HOST / TEST_MINIO_* / CUSTOMER_MONGO_AUTH_URL env wiring to coverage.yml so they execute (mongo was already provided). Verified locally against real mongo/nats/minio containers: all 8 server round-trip tests PASS; integration-only coverage for internal/server = 99.2% (provisionMongo/provisionQueue/GetStorageBytes/DeprovisionResource/RegradeResource all 100%). No bug found in the destroy/regrade paths. Added INTEGRATION-COVERAGE-EXCLUSIONS.md documenting the ≥80 floor method + the only genuinely-unreachable lines (k8s dedicated-backend boot wiring, cmd/ entrypoints). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(provisioner): start NATS via docker run, not a services container The minimal nats:2 image has no wget/curl/nc, so the service-container --health-cmd ('wget ... :8222/healthz') could never pass — GitHub Actions marked the container unhealthy and aborted the coverage job before any test ran (NATS logged 'Server is ready'). Mirror the MinIO pattern: docker run nats:2 -js -m 8222 + a runner-side curl wait on /healthz. Unblocks #47. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Manas Srivastava <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Env-gated real-backend integration tests that drive the gRPC server layer's Provision → Regrade → Deprovision lifecycle against a real Postgres and a real Redis, through the actual RPC handlers (
server.ProvisionResource/RegradeResource/DeprovisionResource) — not injected fakes.Why (truehomie-db DROP incident class, 2026-06-03)
Every existing
server_test.go/server_coverage_test.gotest injects a fake backend, so the realDROP DATABASE/DROP USER/ALTER ROLEDDL had never executed through the gRPC handler path (breaker wrapping, tier→connLimit routing,mapError, response shaping, idempotent re-deprovision). 98.9% statement coverage from mocks did not prove the destroy/regrade DDL is correct end-to-end — exactly the regression class where a real DROP path hides.Tests (all
t.Skipcleanly when the backend URL is unset → CI without a backend stays green)TestServer_Postgres_Provision_Regrade_Deprovision_LiveRoundTrip— assertsdb_/usr_created, roleCONNECTION LIMITactually ALTERed (cross-checked againstpg_roles.rolconnlimit),db_/usr_DROPped, and a second Deprovision is a clean idempotent no-op (the fix(bugbash 2026-05-21): Provisioner P1 wave-1 — Redis k8s sizingForTier yearly + plus tiers #9DROP IF EXISTSfix).TestServer_Postgres_Reprovision_AfterDeprovision_LiveRoundTrip— re-provisions the same token after teardown; guards a partial-DROP leak that would block reuse.TestServer_Redis_Provision_Deprovision_LiveRoundTrip— ACL user created, ACL user + namespace keys reaped, idempotent second Deprovision. (Redis LocalBackend has no Regrade — only the k8s backend implementsredis.Regrader.)Coverage delta (integration-only, per §1.4 mechanism —
-coverpkg=./...)Measured from the server test package (the gRPC layer):
./...backend/postgres/local.goProvision (from gRPC layer)backend/postgres/local.goDeprovisionbackend/postgres/local.goRegradebackend/redis/local.goProvisionbackend/redis/local.goDeprovisionThe backend DDL methods, as exercised FROM the gRPC handlers, go from 0% to 60–83% — the real destroy/regrade path is now integration-covered, not just unit-covered in isolation.
Env-blocked / out of scope (this increment)
TEST_MONGO_URL/TEST_NATS_URL. Postgres + Redis prioritized per the bounded scope.Gate
go build ./... && go vet ./... && go test ./... -short— green (exit 0, no FAIL).gofmt(via$(go env GOROOT)/bin/gofmt) +golangci-lint— clean (0 issues).localhost:5432) + Redis (localhost:6379), and SKIP cleanly with backends unset.🤖 Generated with Claude Code