clusterd-test-driver: headless frontend to clusterd for scripted compute tests

[antiguru]

Motivation

environmentd couples the compute protocol to the full SQL and catalog stack, which makes targeted compute experiments slow to set up and hard to control.
This adds a generic headless frontend to clusterd that drives the compute protocol directly from a script, so a test controls the exact persist state, the exact commands the replica receives, and the exact timestamps.
Design doc: doc/developer/design/20260612_headless_clusterd_test_driver.md.

Description

New crate mz-clusterd-test-driver, split into a generic mechanism, a dataflow builder, and a text scripting layer.

The mechanism hosts the persist PubSub server, connects over CTP (sending only Hello), and exposes a Driver that sends any ComputeCommand, submits dataflows without auto-scheduling, observes merged frontiers and the raw response stream, and peeks.
DataflowBuilder assembles dataflows from generic parts (import_persist, import_index, build over MIR, export_index), owning the MIR-to-LIR lowering, the RenderPlan conversion, the CollectionMetadata attachment, and the SqlRelationType-versus-ReprRelationType bookkeeping; a RenderPlan cannot be hand-built outside mz-compute-types, so it uses the real lowering pipeline.

The driver runs a hand-writable, datadriven-style text script: each stanza is a command (a directive line plus an optional indentation-structured body) followed by a ---- expected-output block that is the assertion, regenerable with REWRITE=1. JSON was awkward to author by hand, especially the MIR spec embedded in a string.
The controller handshake is explicit: a script opens with create-instance, update-configuration (toggles the peek-response stash, off by default so peeks return inline), and initialization-complete. It then declares relations with define-schema, writes synthetic or explicit rows, submits and schedules dataflows, drives allow-compaction, awaits frontiers, and reads results with count/peek — whose output the ---- block asserts. A command that fails renders as error: <message>, so an expected failure is asserted by its golden block rather than a special command.
create-dataflow builds a dataflow from generic MIR — the abstraction behind index / materialized-view / subscribe / copy-to, with define-index as sugar over it (only the index export kind is implemented; the sink kinds are scaffolded). Its build expressions are authored as mz-expr-parser specs (the readable MIR-from-text syntax used by the transform .spec tests, MzReflect-free) rather than a bespoke vocabulary, since MirRelationExpr's serde is not hand-authorable (Row literals are opaque bytes, Get carries a full ReprRelationType). Index import lets a computation read an arrangement the replica already holds, which the lowering picks up automatically with no optimizer.
Assertions run through compute: peek peeks an index and emits its rows, and count is sugar that index-imports the target, computes a count(*) Reduce over it, and peeks the single-row result, so a count runs through a real reduce operator rather than being tallied in the driver.
Because assertions are level-triggered waits on monotonic frontiers, a single sequential script is deterministic regardless of how the dataflows interleave; await-frontier --allow-timeout emits a fixed awaited token so the multi_dataflow repro, whose hydration is nondeterministic, still has stable golden output.
reconnect re-sends only Hello; the script then re-issues create-instance, replays the reconciliation set, and sends initialization-complete to close the window. The builder returns errors instead of panicking on malformed input, so bad script input is asserted as an error: line.
Explicit row values are written as plain tokens, typed against the schema and packed via cell_from_json (JSON value + ColumnType -> Datum).
The scenarios — index, deep-history, side-effects, multi-dataflow, reconciliation, error-behavior, and reduce — are text scripts under test/clusterd-test-driver/scripts/, not compiled Rust.

The handshake version comes from mz_persist_client::BUILD_INFO so it matches the connected clusterd. In mzcompose, persist consensus lives in the environment-selected metadata store (metadata_store_companions(), default Postgres) rather than a hardcoded Cockroach.

Verification

mzcompose runs each scenario script against a real clusterd, with the composition directory mounted at /workdir so the scripts are readable in the container; each command's output is compared to its golden ---- block and a mismatch fails the run.
Count assertions go through a count(*) Reduce, reduce asserts a reduce's output via peek, error-behavior asserts a set of error messages, and multi-dataflow reproduces a current limitation while staying deterministic via allow-timeout.
Crate unit tests cover the direct-write round trip, the frontier merge, the lowered dataflow structure (including count-over-index), schema and datum parsing, and the text script parser.
The local runner run-local.py drives the same scripts on the host (with REWRITE=1 to regenerate goldens) for profiling.

[antiguru]

What we're missing is a specific language to script the headless driver. Encoding the interactions in Rust is a good first step, but ultimately we want something that is easier to iterate on. I could even imagine Python scripts that we load through pyo3 or so.

[antiguru]

Agreed on a scripting language being the goal. I captured it under "Future work" in the design doc: the mechanism is already a thin scriptable surface (send, submit_dataflow, schedule, expect_frontier, peek, subscribe_raw), so a declarative script or Python-via-pyo3 layer would bind to it rather than replace it. Left it out of this PR to keep the first step focused.

[def-]

I have once built something very similar at a previous company, also using json inputs to test an execution engine directly instead of going through the entire database stack. Back then the test framework turned out to not be very useful since it was easier to write unit tests and full system tests instead of the middle-ground headless json tests.

Do we have some concrete examples of regression tests we've wanted to build and failed before, which would be possible with clusterd-test-driver?

I still think it's a worthy experiment, and I can especially imagine that building a fuzzer for clusterd-test-driver with some invariants could lead us to interesting bugs.

[antiguru]

Can we use the metadata store that automatically dispatches to the DB configured through the environment?

[antiguru]

Done in d8bf972. The composition now pulls the store in via metadata_store_companions() (dispatches to the env-configured DB, default postgres-metadata) and derives the consensus URI from METADATA_STORE instead of hardcoding Cockroach. Guarded to a SQL store, since persist consensus needs a Postgres-wire backend (FoundationDB wouldn't work).