Bradley/stackbench test - #5766
Draft
bradleyshep wants to merge 115 commits into
Draft
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
bradleyshep
force-pushed
the
bradley/stackbench-test
branch
from
August 31, 2026 17:38
6a1eb7d to
dc1a416
Compare
The overview carried plans, operations and every campaign's attempts in one body, and the client polled it every five seconds. Give each view its own route: the overview returns summaries, plans move to /api/plans, a campaign returns its sheet, and an attempt's checks, evidence package and log bytes are fetched per tab, with the log answering from a byte offset. Add /api/campaigns/:key/progression, the catalog subgraph plus one node-status snapshot per progression event replayed through the engine, cached per progression-state write. Add /api/events, a change stream fed by a recursive watch per campaign directory with a five second fingerprint poll where the platform has no recursive watch, so the client subscribes instead of polling, and serve the client paths from the same shell.
A completed repair whose grade did not finish is still charged, but its source is now kept beside the run and graded on resume instead of being discarded and replaced by another paid session. The engine no longer changes feature status on an inconclusive result, so a grader failure cannot finalize a feature or open its children. Exhaustion reasons name the limit that ran out (feature, depth, total) or the stall detector. The stall detector seeds on the first observation of a failure again and counts only completed repairs. Features whose shared budget ran out are finalized after any conclusive result rather than being offered a repair with nothing left. Sequential and dependency runs share one repair allowance. The initial grade retries once on a grader failure. The dashboard parses feature repair markers, inspection counts charged repairs from events, and the appliance guide documents every budget field. Dependency mode 4.1.0, mode schema 6. Tests restored for result coverage, definition validators, replay equality, the total cap, carry false, and rolled-back repairs; stale tests from the budget refactor fixed.
Compiled node order is dependency depth, then the order the catalog declares its features, instead of depth then alphabetical id. Within a depth the next repair goes to the first failed feature in that order, so the priority is authored, part of the catalog identity every campaign binds, and identical for every stack. The engine sorts by compiled index and no longer compares ids. The appliance guide states the rule.
A campaign's repair plan takes an order: declared, the catalog's own order, or shuffled, a permutation within each dependency depth drawn from the campaign's ordering seed when the plan compiles. The permutation is frozen in the dependency policy as nodeOrder and used by every stack in the campaign, so no stack sees a different queue and the order is part of the plan's identity. The feature catalog and its qualification are unchanged; the policy identity carries the order. A declared plan must repeat the catalog order; sequential mode rejects shuffled. Dependency mode 4.2.0. The runtime definition and the policy compiler order nodes from the policy and re-derive questline membership.
The dashboard test moves to its own tier under tests/dashboard, writes its fixture evidence once, and compiles each campaign plan once; the unit tier no longer carries it. Lease tests that spawn a second controller and hold lock files move to the integration tier. The two one-assertion command tests merge into cli-entrypoints. Nine per-pack contract files that restated pack ids, points, paths and hook names become one contract that loops over every feature pack the graph references and asserts only rules: one whole request and interface that composes as an upgrade, unique paths, neutral request text, criteria that exist, distinct products across shopping criteria, one authorization owner, declared actors on every replay, a wire or server proof on privacy checks, and datetime-local promotion values. The two graph-shape files merge into the progression contract. A duplicate governance-text test is deleted and the reference campaign compiles once per file. The preflight capacity test derives its floors from the product configuration and the run-suite promotion rig qualifies the recipe's fixture and packs with the recipe, so the eight stale failures are gone. Unit runs in about 19 seconds instead of 246; contracts in 77 instead of 97; the dashboard tier in 153 on its own.
The campaign report and dashboard read the dependency-mode final score as passed points over every selected point, the same scale as the first build, and carry the questline average as a secondary metric. First-try history uses the same scale. Dispersion is reported only from three completed attempts, and every centre value prints its sample size. The cost line is labelled as API-equivalent usage with tokens beside it, and reported duration excludes recorded provider throttle waits. Session metadata records the model ids the provider actually served. LEVELS.md states the scoring rule, prompting.md states the repair-report disclosure symmetry, and the grader README asks for a points rationale in each criterion's note.
Each stack adapter now declares the grader capabilities it provides and whether named application actions travel as HTTP routes or reducer calls. The action registry accepts only the closed capability list. The campaign compiler walks every selected check's setup and criterion steps for every selected stack, requires each action's capabilities, requires the binding the stack's transport can issue, and refuses a campaign while a stack could not measure a selected check. A parity contract compiles the reference graph per stack and checks identical checks, points, and questline denominators. The grader decodes SpacetimeDB's frame compression envelope before searching received transport text, so message-delivery checks measure on that stack. The two replay checks that re-issued a captured HTTP write now carry named actions; the automatic-reorder contract names the saveReorderRule action and requires an entity id on each rule row.
A stored check outcome is now pass, fail, or null. An application abort leaves non-current checks at their prior value, and a check that was never graded stays null and counts as ungraded, so a score has passed, failed, blocked, and ungraded points only. The needs-work rule prompts a working node only for a repairable failure. The accepted-source regrade in the runner keeps its role for re-prompted nodes whose checks are already graded, under the stop reason accepted-source-regrade. Dependency mode is 4.3.0 with state schema 7; the reference campaign and fixtures pin the new values.
Coding containers share the runner's network until the cross-run isolation test passes. Campaign admission now refuses a plan whose parallelism is above 1 and refuses to start while any running container carries the Stack Bench creation label, so one attempt is active on the runner from its first build through its final grade. Campaigns that start no coding container are exempt. The reference campaign runs one attempt at a time. The leak audit extracts every URL and raw socket target from shell commands. A local target outside the run's own web, database, and SpacetimeDB ports contaminates the build; a blocked one is recorded as refused; internet targets are recorded. The runner passes its own ports to the audit.
…acts The contamination audit now reports a harness failure when it examined no session transcript, so a pruned or unmounted transcript store invalidates the attempt instead of passing as clean. Adapters that run no provider session are not audited. Runner facts record the hostname and the number of running containers on the Docker daemon, and preflight warns when containers that are not part of Stack Bench share the runner. Qualification evidence keeps its existing required observations.
An executor no longer fails with a sentence. It fails with a finding from a closed catalog: a kind and typed fields, where a field is a contract control name, an action id, an actor label, a number, a count, or an HTTP status. The rendered sentence is the error message, so every reader shows the same text. The browser boundary classifies a Playwright error into a kind, and raw text travels only as a detail no template reads. Action and check evidence carry the finding; the grader records each criterion's authored sentence. The repair report is assembled from three sources only: the sentence the agent was already given for the behavior, the rendered finding, and the application's console errors. The prose humaniser, the vague metric, the quality artifact, and the scenario-value withholding are gone. The repair policy is scored-only 1.1.0, which declares that scenario values are withheld. An assertion whose action never ran is now inconclusive.
Other systems write the stock tables directly, and the grader's direct write depends on their names, so the names are an interface, not a production expectation. The warehouse-administration contract now names the item, warehouse, and stock tables and their columns; the withheld specification keeps only the convergence behavior. The draft external-data-sync pack reads its own copy of the specification text so the qualified pack is unchanged. A direct write that cannot find the stock interface is now graded as the application failing that interface on every stack, with the writer's diagnostic kept as detail, instead of an inconclusive check. A contract test walks every scored check in the reference graph and fails if a control, attribute, route, reducer, or table it needs is absent from the text the coding agent receives.
…app server after each criterion An estimated request is one the credential broker charged its cost ceiling because the provider response carried no complete usage, broke off, or never arrived. It makes the session cost an upper bound, not unknown, so the receipt now records exact=false with the estimated count and reasons instead of invalidating a completed session. The ledger names each estimate's reason; the cost ledger carries exactness through to the run. A criterion that stops the application server now has it started again before the next criterion, whatever the outcome. A restore the harness cannot complete is recorded as a cleanup failure, the feature's remaining criteria are harness failures rather than application failures, and a later readiness abort is attributed to the harness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
This PR adds Stack Bench, a reproducible benchmark for coding agents building applications against SpacetimeDB, PostgreSQL, and MongoDB.
Benchmark definitions and grading
passed,failed,inconclusive, orharness_failure.Ecommerce benchmark
Runs, repairs, and campaigns
CLI, dashboard, and appliance
SpacetimeDB TypeScript SDK
TableReffrom an anonymous readonly type alias to a readonly interface with the same members.TableRefis used incorrectly.ConnectionManagerreconnects to reuse the identity token previously issued by the server.TableRefdiagnostics.SpacetimeDB CLI
spacetime devto retain configuration changes made during direct-login publishing.SpacetimeDB skills
API and ABI breaking changes
This PR changes the public Rust signature of:
spacetimedb_cli::publish::exec_from_entryThe function now accepts
&mut Configinstead of takingConfigby value. External Rust callers must pass a mutable reference. The API-breaking label should be applied unless the previous entry point is preserved through a compatibility wrapper.The exported TypeScript
TableRefdeclaration changes from a type alias to an interface while retaining the same readonly member contract.Automatic TypeScript client reconnection also changes behavior by presenting the identity token from the previous successful connection.
Expected complexity level and risk
3 / 5
Most of the implementation is isolated under
tools/stack-bench. The principal risks are benchmark correctness, evidence integrity, concurrent resource isolation, interrupted-run recovery, Docker cleanup, and credential handling.The TypeScript reconnect change and
spacetimedb-clihelper signature are the only changes outside the benchmark and skill directories.Testing
TableRefregression tests: 4/4 passed.cargo check -p spacetimedb-cli.exec_from_entryRust signature.