test: add proxy burn-in tooling

[freshtonic]

Summary

• add a dedicated cipherstash-proxy-burn-in workspace package with deterministic conformance coverage
• add a timed soak workload with configurable duration, concurrent CRUD, release-profile Proxy build and launch, and one-second RSS sampling
• adapt the pg-proto type-lab and commerce fixtures into uniquely named public tables with representative EQL v3 integer, text, and JSON domains
• install EQL when needed, apply fixture DDL through Proxy, and seed encrypted values on a fresh connection after schema and encrypt-config reload
• use unqualified fixture names so EQL Mapper resolves every workload statement
• fail conformance unless direct PostgreSQL reads prove ciphertext is stored at rest, then verify typed plaintext is returned through Proxy
• run the encrypted burn-in in a dedicated PostgreSQL 17 CI job and upload its RSS report
• document credentials, EQL setup, module responsibilities, encryption-path invariants, conformance runs, soak runs, and generated reports

Soak reliability

• discover and execute Cargo's exact release artifact, configure it from the direct database target, and reject occupied listeners
• retain and monitor the owned child process so an unrelated Proxy cannot satisfy readiness
• bound readiness, operations, and worker shutdown; terminate the child on interruption or drop
• redact connection credentials from help, diagnostics, and reports
• serialize fixture mutation with a PostgreSQL advisory lock
• write reports atomically with terminal status, errors, elapsed time, artifact provenance, operation counts, and non-zero RSS evidence

Verification

• cargo fmt --all -- --check
• RUSTC_WRAPPER= cargo test -p cipherstash-proxy-burn-in
• RUSTC_WRAPPER= cargo clippy -p cipherstash-proxy-burn-in --all-targets -- -D warnings
• conformance run against the encrypted fixtures
• isolated two-second release soak using a custom CARGO_TARGET_DIR: 15 encrypted CRUD cycles, zero errors, live RSS report generated
• occupied-listener regression: soak failed before spawning instead of attaching to the existing listener
• GitHub reports all six commits as validly signed with james@cipherstash.com and one DCO sign-off each

[tobyhede]

Proxy already has benchmark setup that might be worth extending if it doesn't do what you need

tests/benchmark/ (in main, CI-wired via mise run benchmark/benchmark:continuous) — pgbench-driven, black-box, comparative: Proxy vs raw Postgres vs pgbouncer vs pgcat, plaintext vs encrypted.
Plots results to PNG/CSV, CI handles as regression
Protocol-level throughput/latency comparison.

[tobyhede]

Review: Standards & Spec (vs origin/main, single commit 179b9dc)

Two-axis review — does the code follow the repo's documented standards, and does it match what the PR description asked for — with every finding independently re-verified against the actual code before reporting.

Worth fixing

1. benches/proxy_crud.rs reimplements what cipherstash-proxy-burn-in already provides

proxy_crud.rs:6-7 re-includes the migration SQL via a relative path:

const SCHEMA: &str = include_str!("../../cipherstash-proxy-burn-in/migrations/0001_schema.sql");
const SEED: &str = include_str!("../../cipherstash-proxy-burn-in/migrations/0002_seed.sql");

but cipherstash-proxy-burn-in::lib already exports these as pub const SCHEMA_MIGRATION / SEED_MIGRATION. Same story for connect() (near-duplicate of database::connect), the CRUD shape (realistic_crud mirrors soak::crud_cycle's insert → read → update → cascading-delete skeleton), and the connection-string defaults (duplicated verbatim between main.rs and proxy_crud.rs).

Adding cipherstash-proxy-burn-in as a dev-dependency of cipherstash-proxy would fix all of these at once — no dependency cycle results, and nearly all of burn-in's deps (clap, serde, tokio, tokio-postgres) are already direct deps of cipherstash-proxy, so it shouldn't meaningfully affect bench build time.

2. README doesn't document the credentials the release Proxy binary needs

The soak workload spawns a real release-profile Proxy binary that inherits the parent process's environment, but the README only says credentials "must already be available in the environment" — no variable names, no pointer to mise.local.toml. packages/showcase/README.md:438-445 sets a good precedent here (lists CS_WORKSPACE_CRN, CS_CLIENT_ACCESS_KEY, CS_DEFAULT_KEYSET_ID, CS_CLIENT_ID, CS_CLIENT_KEY explicitly) — worth matching that so a new contributor can actually get soak running from the README alone.

Minor, not blocking

• conformance.rs repeats the literal 900_001_i32 nine times rather than binding it once (the repo's convention elsewhere — e.g. random_id() in the integration suite — binds once and reuses).

Checked and cleared (no action needed)

• The RSS sampler's first tick landing at t≈0 rather than t≈1s is expected tokio::time::interval behavior, and it's actually useful here — it becomes initial_rss_bytes, a genuine baseline reading.
• --max-rss-growth-mib isn't called out in the PR description's bullets, but it's a natural, opt-in extension for a burn-in/soak tool (the whole point is catching leaks) — not scope creep worth flagging.
• The migrations' "copied from pg-proto" provenance checks out — pg-proto is a real repo by the same author, already a workspace dependency of this codebase.

[freshtonic]

Proxy already has benchmark setup that might be worth extending if it doesn't do what you need

Ah, I just looked for a benches dir and missed that. I'll remove mine.

[freshtonic]

Addressed in f517e6db (with the duplicate CRUD benchmark already removed in af008d7b):

• documented CS_WORKSPACE_CRN, CS_CLIENT_ACCESS_KEY, CS_DEFAULT_KEYSET_ID, CS_CLIENT_ID, and CS_CLIENT_KEY in the burn-in README, with mise.local.toml setup
• removed the remaining stale Criterion benchmark documentation
• bound the conformance fixture ID once and reused it

Verified with formatting, the burn-in package tests, and Clippy with warnings denied.

[tobyhede]

Review: Correctness at ff0243ba

The two follow-up commits fix the original aggregate type mismatch, add an automated encrypted soak, and verify ciphertext at rest. I rechecked the remaining findings against the new head; these still need attention.

Blocking

1. Release artifact discovery still ignores Cargo configuration — src/soak.rs:223-253 [repro]

cargo build respects CARGO_TARGET_DIR, build.target-dir, and configured build targets, but spawn_release_proxy() always opens workspace/target/release/cipherstash-proxy. On a shared-target setup, the run either fails after building or launches a stale workspace binary.

Consume Cargo’s JSON compiler-artifact.executable instead of reconstructing the path. That identifies the artifact produced by this exact build and handles target triples as well as custom target directories.

2. An existing listener on 6432 still makes soak measure a dead process — src/soak.rs:63-116, src/database.rs:183-193 [repro]

The child is reduced to a PID before readiness and is never checked again. With another Proxy already listening, the newly built child exits with Address already in use, readiness and fixture setup use the existing Proxy, and RSS sampling targets the dead child. I reproduced a successful run with 1,701 CRUD cycles, zero final RSS, and zero reported growth.

Preflight the configured bind address, but do not treat that as proof of identity—it remains racy. Pass the Child into the run loop, check try_wait() throughout startup, migration, sampling, and worker completion, and reject zero RSS samples.

3. A timed soak can run indefinitely and runtime failures can erase its report — src/soak.rs:89-136, src/database.rs:183-193

Connection, startup-handshake, readiness-query, and CRUD futures have no deadlines. Once the sampling deadline passes, join_next() can wait forever for a wedged worker.

Sampling and worker errors also propagate before the report is written. Add bounded readiness attempts, per-operation timeouts, and a bounded worker-shutdown period. Preserve partial samples and record the terminal error before returning failure.

4. Database credentials are exposed in help and errors — src/main.rs:31-50, src/database.rs:17-20,183-190 [repro]

Clap prints the complete BURN_IN_*_DATABASE_URL environment value in --help, and connection/readiness errors interpolate the same URL.

Use hide_env_values = true and a parsed connection type with a redacted display form. Never include raw database URLs in diagnostics or reports.

Should fix

5. The aggregate assertion still panics on its intended failure case — src/conformance.rs:86-97

The new ::bigint cast fixes the unconditional numeric decoding panic. However, if the join loses every row, sum(...)::bigint is NULL and get::<_, i64>() still panics before "joined CRUD result was corrupted" can fire. Decode with try_get::<_, Option<i64>>() and assert Some(4_998).

6. Concurrent runs can deadlock or invalidate one another — migrations/0001_schema.sql, migrations/0002_seed.sql:3-5, src/soak.rs:156-219

Every run drops and recreates the public fixture tables. The remaining TRUNCATE order is also the reverse of the CRUD insert order. Reordering prevents that specific lock inversion, but concurrent runs would still destroy or contaminate each other’s fixtures and measurements.

Acquire a run-level advisory lock and retain its connection for the entire conformance or soak run; a migration-only lock is insufficient.

7. Report and gate semantics remain inconsistent — src/soak.rs:113-153

A worker error already short-circuits at join_next(), so ensure!(report.errors == 0) cannot observe one. Zero completed cycles can pass, output-path errors are discovered only after the workload, and soak passed prints before the RSS gate—I reproduced it printing success immediately before exiting 1.

Require at least one completed cycle, include terminal status in partial reports, preflight output with a sibling temporary file and atomic rename, and print success only after all gates pass.

8. The migrated database may not be the spawned Proxy’s upstream — src/soak.rs:61-74, src/soak.rs:247-253

--direct-database-url selects the database where EQL is installed and ciphertext is inspected, but the child Proxy is spawned without database arguments and reads ambient CS_DATABASE__*. An override can therefore migrate one database while the child serves another.

Pass or validate the child’s upstream configuration. Record sanitized provenance in the report: artifact hash or commit, concurrency, timestamp, actual elapsed duration, and a redacted database identity.

Additional notes

• The new CI job closes the encryption-path and execution gaps. It runs only soak and omits --max-rss-growth-mib, so deterministic conformance and retained-growth gating remain local-only. Adding them would strengthen coverage once a stable threshold is established.
• First-to-last RSS delta matches the README’s “retained growth” wording. The immediate first tick is still a cold baseline; add a warm-up or delayed first sample. Trend fitting is optional rather than a correctness requirement.
• Add kill_on_drop(true) and signal handling so panics and interrupts do not leave an owned child or discard all samples.
• Make the wide-text assertion exact, use checked multiplication for --max-rss-growth-mib, and establish a fixture migration/version strategy before the schema evolves again.
• The updated package’s three unit tests pass. The live subcommands remain the only tests of lifecycle, encryption, and report behavior.

Separately, commits 5c72f021, 31836df6, d64fcba4, and ff0243ba lack the repository-required DCO sign-off, and src/main.rs:61 should say “connections,” not “sessions.”

[freshtonic]

Addressed the latest review in c9dbd6f9:

• Cargo JSON now supplies the exact release executable; custom target directories are supported.
• The soak rejects occupied listeners, retains its spawned child, checks try_wait() throughout, and rejects zero RSS samples.
• Readiness, individual operations, migration, and worker shutdown are bounded; interruption/drop terminates the child.
• Reports are written atomically and include terminal status/error, actual elapsed time, artifact/source/database provenance, operations, errors, and partial RSS evidence.
• Database arguments are parsed into a credential-redacting type, hidden from environment-backed help text, and the spawned Proxy is configured from the direct target.
• A run-wide PostgreSQL advisory lock prevents concurrent fixture mutation; truncation follows dependency order.
• The aggregate assertion handles SQL NULL explicitly, the wide-text assertion is exact, RSS conversion is checked, and CLI wording says connections.

Regression coverage includes credential redaction, custom Cargo artifact discovery, report validity, occupied-listener rejection, and an end-to-end custom-target soak. The burn-in package tests and Clippy pass. I also rewrote the stack: GitHub reports every commit as validly signed by james@cipherstash.com, with exactly one matching DCO sign-off.