Repository navigation
fix(oracle-gaps): Windows static and e2e failures, by class (follow-up to #58) - #73
Merged
Merged
Conversation
The framework, the eslint plugin, and the gap CLI have never run on Windows, and users report path problems there. Two diagnostic jobs run on windows-latest: the unit suites (framework, eslint plugin, gap fast oracles) and the gap CLI end to end (a fixture package written at run time, the plain and daemon paths, and one fixed-seed e2e unit). Both are continue-on-error and each step continues on error, so the first failure does not hide later ones. Neither is a required check or a dependency of preview. Co-authored-by: Isaac <no-reply@databricks.com>
Diagnostic only, not a fix. The first run showed the gap fast suites and the fixed-seed end-to-end unit hang on windows-latest (cancelled at the job timeout, with no output after the static oracle and the e2e calibrations started). The fast suites now run one file per step with a step timeout and verbose output for the static oracle, so the hang is located. The fixture gets the package.json it was missing (the CLI reported ENOENT for it). The job timeouts are raised to 45 minutes so the split steps can finish. Co-authored-by: Isaac <no-reply@databricks.com>
… run Diagnostic only. Adds a bash replay of the first failing static unit (its verdict is not printed by the run), a run of the fixture with the discovered config instead of --config, and a dump of the gap CLI's error.log files from the OS temp directory, which the daemon run's failure points to. Co-authored-by: Isaac <no-reply@databricks.com>
…ines Forward slashes at entry, native only at the file system. Paths enter the CLI as argv, as Vite and Vitest report them, and as realpath returns them; comparisons and keys use one spelling (oracle-gaps/paths.mjs). - The config wrapper imports the config by a POSIX specifier. A backslash in an ES module specifier is a character, so every run with a config was refused on Windows (check.mjs, wrapper). - Mutant copies are keyed and served in forward slashes, as Vite resolves and Vitest reports them. The copy map was keyed by native paths, so the copies were never served on Windows. - Module IDs are kept by isAbsolute, not by a leading slash: a Windows ID is `C:/...`. - The package root, the located packages, and the source names use the native real path on both sides (realPathOf). The JavaScript realpath kept 8.3 short names such as RUNNER~1 and refused every owns entry. - The daemon listens on a named pipe on win32 (no socket file to remove), keys by the same real path as the CLI, and its timeout names the end of its log. - A package with no package.json is refused, not an internal error. - Replay and rerun lines print PowerShell and cmd forms on Windows; the POSIX form is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
…e realpath - npm is an npm.cmd shim on Windows; spawnSync runs it through a shell there (package.test.ts, shipped-citations.test.js). - The probe-form oracle's model decides production by a directory named src, split on either separator, not by a substring with `/`. Production was right; the model was wrong on Windows. - The owns test uses the native real path, as the CLI does. Co-authored-by: Isaac <no-reply@databricks.com>
The diagnostic steps are gone. The Windows jobs mirror the Linux shards: the unit suites in one job, the static gap oracle and the fixed-seed end-to-end oracle in three shards each (pr-shards.mjs), and the gap CLI on a fixture, including the discovered config and the daemon. Each job has a 15-minute backstop. The jobs are still continue-on-error until every one is green; that line is removed then. Co-authored-by: Isaac <no-reply@databricks.com>
… path The owners and the instrumenter named files with the platform's separators, and TypeScript matches a file's name as given, so on Windows the owner map keyed `src\x.js` while the sources were `src/x.js`, and CommonJS sites lost their functions. The compiler now gets forward-slash names, and owner keys are forward-slash (schemata.mjs, owners.mjs, analysis.mjs). A path's drive letter is capitalised in the one spelling (paths.mjs), so a drive's case never decides a path. New oracle: gap-report-path-spelling (a metamorphic check). Law 1: the instrumenter gives the same mutant ids for a source under any spelling of its path. Law 2: relativeWithin answers from the segments alone. Evidence block - Law: a path's separators and a drive letter's case do not change what the gap CLI decides (paths.mjs, Windows support, 2026-11-13). - Reach: 200 generated histories (fixed seed 20261113 and random), over four module forms, drives in both cases, mixed separators, short and spaced names. Reach is not declared (states: []); the histories are not measured as reaching a state. - RED: on the parent commit (72b9495), the Windows jobs of #58 failed: the config wrapper import (check.mjs), the copy map, the owner keys (owners.mjs), and the 8.3 short names. On Linux the enumeration law does NOT reproduce the Windows failure: the instrumenter gave the same ids on the parent. The Windows failure is a platform path.relative difference, which Linux cannot produce. The RED for those classes is the Windows CI run. - GREEN: the Windows jobs (see the PR). Local, on macOS: the new oracle 5 of 5, the framework 114 of 114, the eslint plugin 7335 of 7335. - Mutant: relativeWithin as it was before (raw string comparison) fails the relation checkpoint; dropping the drive capitalisation fails it too. Both run on the parent tree with the oracle and both fail (exit 1). - Limits: no declared reach; Law 1 uses the forward-slash spelling as its reference, so it claims only that the spelling changes nothing; owners, realpath, and OS short names are judged by the Windows end-to-end run, not by this oracle. Co-authored-by: Isaac <no-reply@databricks.com>
… witnesses - P10: the Windows CLI runs assert the done line on both streams, and that it matches the exit code (scripts/windows-witness.mjs done). - A package with no package.json is refused with exit 2 and a done line that names it (Windows witness). - One early stop: SIGINT once the CLI prints its first line; the run must end with the stop's done line on both streams, or with no output (#57). - Per-shell replay (framework, accepted by the coordinator): a failing property's PowerShell and cmd lines each reproduce the failure in their own shell. The POSIX line is run in PowerShell and must not reproduce (the RED). A run that printed only the POSIX line fails here (the mutant). - The tsc link lookup in package.test.ts reads npm's Windows shim, not a symlink. - The eslint plugin suite runs even when the framework suite fails. - Job timeouts are 20 minutes, as high backstops (2026-11-12). Co-authored-by: Isaac <no-reply@databricks.com>
…ch packages have a manifest Windows CI failed the classify oracle's baseline law: the CLI prints a PowerShell and a cmd replay line there, and the law read only the POSIX `replay:` line (Difference: "the replay of tests/f0.test.ts is undefined"). The law is unchanged: each replay line the platform prints must read as `vitest run <file> -t <pattern>` in its own shell, and the owed test is the one it selects. On win32 the law owes both per-shell lines; elsewhere the POSIX line (replayLinesOf, powerShellWords, cmdWords). The static oracle's scratch packages had no package.json, which the CLI now refuses (args.mjs). Each fixture now writes a minimal manifest; its content and its histories are unchanged. Evidence block - Law: the replay line of each shell reads as the owed command (Law 9, baseline; the CLI's replay lines, stream.mjs). - Reach: the existing classify histories (276 tests); no new histories. - RED: Windows run 43478073715, classify shard 3 (baseline law, 11 violations) and the static shard 1 refusal (the missing manifest). - GREEN: local POSIX classify 276 of 276. The win32 branch runs in the Windows job (see the PR). - Mutant: not run; the win32 branch is exercised only by the Windows job. - Limits: the win32 branch is checked on Windows only, not by the oracle's own generated histories. Co-authored-by: Isaac <no-reply@databricks.com>
… shipped citations Run 43483101749 (Windows, the first with the witnesses) showed: - The CLI printed `report <pkg>\out\summary.md` and `tests\a.test.ts` in the done line and in the heartbeat. Report paths and display paths are now forward-slash (posix, check.mjs, ending.mjs), as the register's keys rule requires; the file system still gets native paths. - The classify oracle's cmd reader rejected `\"` inside a double-quoted argument, which Node's argument parser reads as a quote. It now reads it so. - The static and classify oracles' second scratch package had no manifest. - Shipped citations joined paths with backslashes, and npm names files with forward slashes. The comparison now uses forward slashes. - The refusal witness searches both streams (a refusal prints to stderr). - The static and fixed-seed e2e took 3 to 6 times their Linux time on Windows, and the Linux shards hit the limit. Windows runs six shards of each (pr-shards.mjs `oracle-gaps-static-windows`, `oracle-gaps-fixed-seed-windows`, checked by `pr-shards.mjs check`: each test in exactly one shard). The classify, probe-form, mutant-validity, and path-spelling oracles run in their own job. Evidence block - Law: report and display paths are forward-slash; a cmd line reads its quoted quotes as Node does (register: deterministic replay; the CLI's keys rule). - Reach: the existing e2e and static histories, unchanged. - RED: run 43483101749, which printed backslash report paths and failed classify on `\"`. - GREEN: pending the next Windows run; local POSIX suites pass (classify 276, shipped citations 2). - Mutant: not run for this commit. - Limits: the Windows e2e and static shard timings are measured on the next run before the budget claim (P12) is made. Co-authored-by: Isaac <no-reply@databricks.com>
…and its models Run 43493431349 (Windows) failed the static and e2e oracles on report paths: the done line names its report in forward slashes (the register's Windows paths rule), but the model built the expected line from the native output directory, and the declared oracle files were keyed by native relative paths (`tests\a.test.ts`), so a heartbeat's file was not an owner. - analysis.mjs, check.mjs: declared oracle files and copies are keyed by their forward-slash relative path (relativeWithin). - gap-report-model-oracle.mjs: the model's expected done line is forward-slash (inline, the model does not import production). - shipped-citations.test.js: the rejection test's strings are forward-slash. Evidence block - Law: keys and report paths use forward slashes on every OS (register, Windows paths, #58); filesystem calls get native paths. - Reach: the existing static and e2e histories; no new histories. - RED: run 43493431349: static 40 refusal-ending violations, e2e 17 baseline lead, 12 report-shaped ending, 4 streaming, 3 greppable stream. - GREEN: pending the next Windows run. Local POSIX: shipped citations 2 of 2, CLI lints clean; the Linux keys are unchanged. - Mutant: not run. - Limits: the Windows e2e and static timings are measured on the next run. Co-authored-by: Isaac <no-reply@databricks.com>
The early-stop step failed on Windows (SIGINT ends the process through TerminateProcess, so the CLI's stop handler never runs; the run ended with a null exit code after its done line), which skipped the replay witness behind it. The replay witness now runs regardless, so each witness reports. Co-authored-by: Isaac <no-reply@databricks.com>
The replay witness found no per-shell lines in the fixture's failing output. This prints the first lines of that output in the same step, so the next run shows what vitest printed. Diagnostic, to be removed with the witness fix. Co-authored-by: Isaac <no-reply@databricks.com>
The replay fixture failed to load (TypeError reading fixedSeeds[0]), so the witness saw no failure. It now declares fixedSeeds: [1]. Co-authored-by: Isaac <no-reply@databricks.com>
… forward-slash done lines - Stop (#57, all-or-nothing) on the unsupervised win32 path: a stop of a run that has started now writes its `oracle-gaps: done stopped (<signal>)` line on both streams and exits 128 + signal, as the supervisor does. A stop before the run starts still prints nothing (index.mjs, runStarted). - Test-only stop hook: when the file ORACLE_GAPS_TEST_STOP names appears, the CLI takes the same stop as a SIGINT (index.mjs). The Windows stop witness uses it (a non-console parent cannot deliver Ctrl+C there; child.kill on win32 is TerminateProcess, which is a SIGKILL-like limit, so no line is owed). - Static oracle: the `unknown` long-argument case uses 30,000 characters on win32. Windows caps a command line at 32,767 characters; a longer one never starts (status null). Narrowing, recorded in the PR: the POSIX runs keep 100,000. - e2e and static model: the expected done line's report path is forward-slash (inline in the model; the e2e corruption check likewise). - The diagnostic Get-Content line is removed. Evidence block - Law: the all-or-nothing stop law (#57) on every platform; keys and report paths forward-slash (register: Windows paths; stop window on every platform). - Reach: the existing static and e2e histories; the stop witness is one run. - RED: run 43498457502, stop witness `stop was not all-or-nothing (signal SIGINT, exit null)`; static `with unknown arguments, the refusal exited null`; e2e `the last stdout line is ... report D:/...`, not `...\out\summary.md`. - GREEN: pending the next Windows run. Local: the CLI lints and loads; the stop witness is queued on the e2e lock for a fixture run. - Mutant: pending (the replay witness, on a throwaway branch). - Limits: OS delivery of Ctrl+C is not exercised on Windows; TerminateProcess is treated like SIGKILL; the win32 argv cap narrows the long-argument case. Co-authored-by: Isaac <no-reply@databricks.com>
…et, and logs The default output directory is under the OS temporary directory, which on windows-latest is `C:\Users\RUNNER~1\...` (an 8.3 short name). The model's expected done line names the package by its real path, so the report path never matched (e2e: `the last stdout line is ... report C:/Users/RUNNER~1/...`). The temporary directory is now taken by its real path (realPathOf) for the state directory, the socket, and the error log. Evidence block - Law: keys and report paths use one spelling; a real path on both sides (register: Windows paths; the realpath rule). - Reach: the existing e2e histories (the done-line law). - RED: e2e run 43515364499, done line with RUNNER~1 vs the package path. - GREEN: pending the next Windows run. - Mutant: pending; the replay witness mutant runs on the throwaway PR. - Limits: the stdout gap-count mismatch (`2 gap lines for 0 gaps`) and the cold internal error in the incremental law are not yet explained. Co-authored-by: Isaac <no-reply@databricks.com>
… real path The e2e oracle realpaths the OS temporary directory with the JavaScript realpath, which keeps the 8.3 name (C:\Users\RUNNER~1). The CLI prints the native real path, so the expected report line never matched. Use realpathSync.native, as the CLI does. Evidence block - Law: one real path on both sides (the realpath rule; Windows paths). - Reach: the existing e2e histories. - RED: run 43515364499 and ee1a5fd: the done line names RUNNER~1. - GREEN: pending the next Windows run. - Limits: the static `output namespace` violations (82 in the last run) are not yet traced. Co-authored-by: Isaac <no-reply@databricks.com>
…es; fix the tsc lookup type - Blocking now: the Windows unit suites, the classify, probe-form, mutant-validity, and path-spelling job, and the gap CLI fixture job with its witnesses (done line, refusal, stop, replay). Their continue-on-error is removed. - Non-blocking, stated in the workflow: the Windows static and e2e jobs, which fail on Windows (tracked in the follow-up PR's body by class). - Linux `Test` failed the typecheck on package.test.ts (the tsc lookup's optional capture). Fixed; tsc --noEmit passes locally. Co-authored-by: Isaac <no-reply@databricks.com>
…al path (output namespace) The output-namespace law (checkpoint `static`) took the OS temp directory by JavaScript's realpath, which keeps an 8.3 short name (C:\Users\RUNNER~1), while the CLI's default output is under the native real path. So every run's output failed "not under the OS temp directory" on windows-latest (82 hits in run 43525797122). The oracle now uses realpathSync.native, the same rule the CLI uses, and drops the raw tmpdir() alternative. Evidence block - Law: a run's default output is under the OS temporary directory's real path, never in its package (maintainer decision, 2026-11-11; register: the realpath rule). - Reach: the existing static histories (the outputs of three runs per history, two of one package and one of another). - RED: Windows run 43525797122, static shards: 82 violations, law "output namespace", "a run's default output is not under the OS temp directory". - GREEN: pending the next Windows run on this branch. - Mutant (Windows-only): the CLI's temporary directory taken raw (tmpdir() without realpath) puts the output under RUNNER~1, which the fixed law rejects on Windows. On Linux the two spellings coincide, so the Linux run cannot kill this mutant; the Windows run is its witness. - Limits: Linux does not exercise the 8.3 spelling. Co-authored-by: Isaac <no-reply@databricks.com>
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
…gs (baseline lead) The e2e driver replaces scratch paths with <package> and <cli> in the CLI's output, but only in the native spelling. The CLI now names its report and package paths in forward slashes (the Windows paths rule), so the done line kept `D:/a/.../.scratch-XXXX/out/summary.md` and the baseline-lead law (18 hits in run 43525797122) read it as not `<package>/out/summary.md`. Evidence block - Law: the CLI's done line names its report in the package's placeholder spelling (law: baseline lead; register: Windows paths). - Reach: the existing e2e histories. - RED: run 43525797122, e2e shards: 18 violations, law "baseline lead". - GREEN: pending the next Windows run on this branch. - Mutant: a CLI that prints the native spelling (backslashes) in the done line is now caught by the same check, since the placeholder is only applied to the spelling the driver names; not run in CI. - Limits: the driver names two spellings of each scratch path (native and forward-slash), not every possible form. Co-authored-by: Isaac <no-reply@databricks.com>
…river (streaming) The e2e driver's mutant observations named each owner file with path.relative, which returns backslashes on Windows (tests\a.test.ts). The CLI names a heartbeat's file in forward slashes, so every heartbeat failed "names sut.ts:M1 in tests/a.test.ts, which does not run it" (the streaming law, 4 hits in run 43525797122, plus greppable-stream instances). Evidence block - Law: the heartbeat names a test file that runs the mutant (streaming law); keys and report paths are forward-slash (register: Windows paths). - Reach: the existing e2e histories with a slow owner. - RED: run 43525797122, e2e shards: "a heartbeat names ... which does not run it". - GREEN: pending the next Windows run on this branch. - Mutant: not run. - Limits: the stdout gap-line count mismatch (`122 gap lines for 121 gaps`, greppable stream) is not yet explained; left for the next run. Co-authored-by: Isaac <no-reply@databricks.com>
commit: |
…d review items F1-F8 Review of #58 at ebc39e6 (NOT READY). This commit fixes: - F1 (stop never stops): the witness's stop file is absolute (the CLI's cwd is the package), a stop requires the stopped line on both streams with exit 130, and the stop lands 400 ms after the run starts (the fixture's output is written in one burst near its end, so the first output line is no clock). - F2/F3 (replay quoting): PowerShell reads U+2018 to U+201B as quotes, so every such character is doubled inside single quotes. cmd toggles quoting on every `"` and expands `%`; a name with either has no cmd form, and its cmd line says `use the PowerShell line for this name` (stream.mjs, src/oracle-property.ts). The classify oracle reads each shell's line by that shell's grammar (the PowerShell quote set; cmd's quote toggling, `^`, and `%`), checks the win32 printer on every host, and generates adversarial names (curly quotes, `& | < > ( ) ^ % !`, quotes, backslashes), drawn as the last field of the test record (the fixed seeds' later draws shift; recorded here). - F(f) (POSIX hook): the test-only stop hook lives in the supervisor and goes through its stop, which forwards to the child; a hooked stop gives `done stopped (SIGINT)`, exit 130 (local run: outcome (a)). Unsupervised runs keep the hook in the process. - F5: posix() acts only on win32; on POSIX a backslash is a name's character. - F6: check.mjs realpath of workspace links, report.mjs results path, and ending.mjs error log path are forward-slash and use the native real path. - F7: the blocking Windows jobs are in preview's needs. - F8: the non-blocking jobs point at #72. - The Windows replay witness runs each printed PowerShell line and its cmd line, for two fixtures with hostile names (curly quote and metacharacters; a double quote and `%`). Evidence block - Law: the printed replay line replays in the shell it names (register: deterministic replay across shells; each printed line must replay on Windows); a hooked stop is a stopped line with exit 130 (register: stop window on every platform). - Reach: the classify oracle's 276 generated histories (new adversarial name dimension, drawn last); the Windows replay fixtures; one stop per run. - RED: the parent printer (ebc39e6, with only the platform parameter added) fails the new classify oracle on two names: `"vitest run tests/f0.test.ts -t '^’ t0$'"` (the PowerShell line ends its quote at U+2019) and the cmd line for `q"t` (`which the shell does not read as plain words`). Run locally, exit 1. - GREEN: classify 276/276 on Linux (current code); POSIX stop witness outcome (a), exit 130 (local, under the e2e lock). Windows: pending CI on this head. - Mutant: the parent printer is the mutant (the curly-quote and cmd forms before this commit); the RED run above is its rejection. - Limits: - the stop timing (400 ms after spawn) is an assumption about the fixture's length; a run that ends first is reported as a failure, not passed; - cmd's `!` is literal (delayed expansion off), an assumption; - the classify fixed seeds' later draws shift (the new field is drawn last); - the win32 argv cap, the pipe name's machine scope, the POSIX state paths moving to the temporary directory's real path (an older daemon is found only after its idle timeout), and the unsupervised win32 process-tree exemption (#74) are recorded in the PR body. Co-authored-by: Isaac <no-reply@databricks.com>
… drives only on win32 The Linux run of 50b4ab5 failed gap-report-path-spelling: posix() acts only on win32 (review F5), so on POSIX a backslash is a name's character and a drive letter's case is a different name. The oracle now generates backslash joints and mixed drive letters only on win32; on POSIX it generates `/` joints and `C`. This narrows the law on POSIX (recorded): the POSIX law is the forward-slash spelling, and the separator and drive-case laws are judged on win32 (the Windows classify job runs the oracle there). Evidence block - Law: a path's separators and a drive's case change nothing the gap CLI decides on win32; on POSIX they are name characters (paths.mjs, F5). - Reach: the same 200 histories, with the separator and drive alphabets by platform. - RED: Linux shard 3 of 50b4ab5: "observed value differs" on the path-spelling oracle (2 of 5 tests). - GREEN: local POSIX 5/5. - Mutant: the win32 mutants run in the Windows classify job; not re-run on POSIX. - Limits: on POSIX the separator and drive laws are not generated. Co-authored-by: Isaac <no-reply@databricks.com>
…e POSIX F5 witness Coordinator rulings, applied: 1. Stop trigger: an event, not a clock. The witness writes the stop file on the CLI's first stderr line. Measured (2026-11-14): on a pipe the CLI streams its progress on stderr and prints the done summary (stdout) only at the end, so stdout is no start marker. No stream.log exists on this branch; the stderr line is the event. The 400 ms timer is gone. 2. Classify generator: the hostile names are derived from the test's own fields (HOSTILE[(ms+ti) % 7] for one test in four), not drawn. A drawn field would shift every later draw of the fixed seeds. Proof: `git diff origin/main` on the classify oracle changes no `fc.` line (zero), so the generator is unchanged. 3. Path spelling on POSIX: the oracle generates backslash joints and `x:` drives on every platform. The model splits the spelled strings per platform (win32: both separators, case-insensitive drive; POSIX: `/` only, a backslash is a name's character). On POSIX a disk witness creates the backslash name and asserts it exists, realPathOf keeps it native, and posix() leaves it unchanged (F5). This removes the earlier POSIX narrowing. Evidence block - Law: the stop is delivered at a run event and ends stopped (register: stop window on every platform); a backslash name on POSIX is kept native (review F5). - Reach: the path-spelling oracle's 200 histories, now with backslash and `x:` names on POSIX (disk-backed); the classify oracle's existing histories, with hostile names on one test in four. - RED: ebc39e6 with this oracle: 2 of 5 tests fail, "the observed value differs from the owed value" (the relation and disk checks). - GREEN: POSIX local 5/5 (path-spelling); stop witness local POSIX outcome (a), exit 130 (event trigger). Windows: pending CI on this head. - Mutant: the parent (ebc39e6) is the mutant for F5 (posix() rewrites backslashes on POSIX); its RED is above. - Limits: the stdout summary arrives only at the end on pipes (a finding, not a fix here); the stop trigger assumes the CLI prints a stderr line before its work completes (measured on the fixture only). Co-authored-by: Isaac <no-reply@databricks.com>
The derived hostile names reached constructed tests too (ms 0 at index 0), which
broke the name-relationship reach of two constructed histories ("a failing
baseline test whose name another test's contains" and "... ends another
test's"). A hostile name now applies to generated tests only (ms >= 1).
Evidence block
- Law: the constructed histories reach their declared states (the oracle's own
reach rule).
- RED: de8a373 run: "constructed history ... did not enter a baseline test whose
name another test's contains" (Linux shard 3 and the Windows classify job).
- GREEN: local classify 276/276.
- Mutant: not run.
- Limits: as recorded in the previous commit.
Co-authored-by: Isaac <no-reply@databricks.com>
…IGTERM state only on POSIX RED: CI run 43675826518, every Windows end-to-end shard: 'constructed history "every outcome class" names undeclared states a run stopped by SIGTERM while a worker spun'. The SIGTERM state is declared on POSIX only (posixStops), and the first half named it on every platform. Co-authored-by: Isaac <no-reply@databricks.com>
…ns of the merged tests Durations: the Vitest JSON reporter on the Arclet pool at 1636e85 (51 fixed-seed tests, all passing). The two halves of 'every outcome class' sit in shards 2 and 4; ten shards, cap ten names each. Co-authored-by: Isaac <no-reply@databricks.com>
… the fingerprint lists get its line RED: CI run 43680525602, 'Windows unit suites', 'Test' and 'Test on Vitest 5': tests/gap-report-history- fingerprint.test.ts failed: the field was inserted before 'importer' and the other fields' draws were kept, but the recorded key list had no 'pauseEngages'. A new field goes at the end of the grammar and its line at the end of each recorded list. GREEN: the canary passes (the 53 recorded lines are unchanged; one line added per list). Co-authored-by: Isaac <no-reply@databricks.com>
…the list Constructed histories sample seed + index, so inserting one mid-list changed the histories of every later entry on main. The disarmed-pause history and the second half of "every outcome class" now come after main's last entry; the first half keeps main's index 0, and the remaining entries keep their main positions. Co-authored-by: Isaac <no-reply@databricks.com>
…es no warm label The warm label names the daemon that served the warm check. Until now no history reached a reset whose paused daemon was another one, so dropping the warm-pid equality went unseen (review of #73, B2). A constructed history, appended at the end of the list, sets warmReplaced (a drawn field, appended last, that only constructed histories set): the warm check's daemon is stopped before the paused check, so the paused daemon is a new one. The state "a warm daemon was replaced before the pause" is declared and entered, and the law check throws if the warm label is recorded for a daemon that did not serve the warm check. The README documents ORACLE_GAPS_TEST_SERVED_MARK, which the warm check reads. Co-authored-by: Isaac <no-reply@databricks.com>
…ux shard 2 The Linux fixed-seed pattern matches whole names, so "every outcome class" selected the first half only. The second half, "every outcome class, the warm and abort stops" (155 s on the CI runner), fell into the rest and made shard 3 take 605 s of tests, 10.37 min for the job (run 43683794816). Shard 2 names both halves; the comment of shard 3 lists what it holds. Co-authored-by: Isaac <no-reply@databricks.com>
"every outcome class" must reach "a hang cut off for idleness", a state that win32 reaches too. Once on win32 (run 43683794816, shard 2/10) the sampler's quiet window was not seen, so the state was missed. The cause is not confirmed; the witness is owed (a test-only hold at the hang), recorded as a Limits item with the job references. Co-authored-by: Isaac <no-reply@databricks.com>
…is armed An armed daemon holds its first connection unread, so the stop of the warm daemon, sent after the arm, never ended and the paused check was served by the same daemon. The stop now comes before the arm. Co-authored-by: Isaac <no-reply@databricks.com>
Appended at the end, the disarmed-pause history draws a new base (seed + index). Its base has a generated test whose input is a multiple of 3, so the work bound of 1 applies, and a mutant of a generated function is killed by work there. The model predicts no work verdict for a generated test, so the CLI and the model disagreed: "random-only-kill" against "not-judged" (run r73h-disarm-green, 07:13 UTC). The history's generated inputs now declare no bound. The Limits name the model gap; the random seeds keep the bound. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
marked this pull request as ready for review
November 15, 2026 08:17
KyleAMathews
added a commit
that referenced
this pull request
Nov 15, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 15, 2026
Takes main's constructed histories (indices 27-29, from #73) first, so they keep their positions, and appends this branch's test-root history after them. The grammar's new field testRoot goes at the end of the drawn record, and its fingerprint is recorded in history-fingerprint.json. Refs #101. Co-authored-by: Isaac <no-reply@databricks.com>
5 tasks done
KyleAMathews
added a commit
that referenced
this pull request
Nov 15, 2026
Takes main's #73 (the Windows stops through the test stop hook, the warm daemon, the disarmed pause and its pause marks) as the base, and layers #57 on top: - The `after` stop holds on ORACLE_GAPS_TEST_HOLD_UNTIL. The wall-clock ORACLE_GAPS_TEST_HOLD_EXIT_MS hold is gone from the CLI, and the README lists the hooks as they now are. - Model: main's pauseEngages and warmReplaced draw first, then #57's fields. - Fingerprint regenerated (main's 55 fields are an exact prefix). - Constructed histories: main's 30 blocks are an exact prefix by index, and #57's 12 follow (the default-output and claim histories, the stream-log losses, and the supervisor death). - pr-shards: main's partitions, plus the 12 new Windows names placed in shards 6 to 10. The Linux fixed-seed rest shard takes the new names. - Static oracle: main's win32 spelling of the package path with #57's default-output normalization. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 15, 2026
… the canary's listing to Vitest 4 The determinism unit added by #73, a daemon pause that cannot engage, gives different traces across runs (#105), so it is excluded until fixed, as the other main failures are. The constructed-order canary lists the gap oracle only under Vitest 4: the Vitest 5 job runs no gap oracle, and its listing finds no suite in that file. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
added a commit
that referenced
this pull request
Nov 15, 2026
…death state is platform-scoped Windows CI showed the claim refusals' fix line printed a backslash path (`delete <out>\run.pid`), where the owed line names the claim as `<out>/run.pid`. The fix line now prints the claim in posix form. The supervisor-death state exists only where a supervisor does, so it is required and entered only off win32 (platform-scoped reach, register "Platform-scoped reach (#73)"). On win32 the history runs without the death witness and enters no state; that run is not judged by the death law. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 16, 2026
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
* test(oracle-gaps): constructed history for a config whose root is a subdirectory Vitest names a copy of a test file by its path from the config's root. A config that sets root to the directory holding the tests makes every copy's id root-relative, so the wrapper's resolver must find the copy there too. Adds one constructed history (appended last, so existing histories keep their positions) with a config whose root is the tests directory, and declares the state it reaches. The owned mutant must be judged, never a setup error. Refs #101. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): declare the constructed config-root field in the generator The constructed history for a config whose root is a subdirectory sets configRoot, which the histories arbitrary could not take back, so the campaign failed before running. Declare it as a drawn constant (it draws nothing, so generated histories keep their values) and take a constructed history back as false, as tornRows is. Refs #101. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): set the constructed history's root through test.root Vitest names a copy by its path from the test root (test.root), not from the Vite root the CLI forces to the package. The constructed history must set test.root to the directory holding the tests, as the repository's own oracle-gaps config does, and the fixture must reach that state. The field is renamed testRoot to match. Refs #101. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): the test-root history holds no torn row The constructed history copied tornRows from its template. Its witness needs a batch hold this history does not produce, so the campaign failed on the torn-row witness before the law that the history exists for. Its state is the test root, so it sets tornRows false. Refs #101. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): copies resolve when the vitest config sets root A copy of a test file is named beside its file, but Vitest names a module by its path from the test root (test.root), so a config that sets the test root to a subdirectory gave every copy an id the resolver did not map back, and each mutant failed with setup-error. The wrapper's resolver now also tries the copy from the root Vitest names modules from, recorded per context after it resolves, and spells paths as the copies are keyed (on Windows, forward slashes and an uppercase drive; on POSIX, unchanged). Closes #101. Co-authored-by: Isaac <no-reply@databricks.com> * chore: changeset for the test-root copy fix Co-authored-by: Isaac <no-reply@databricks.com> * ci(oracle-gaps): place the test-root history in a Windows fixed-seed shard The Windows fixed-seed partition lists every test by name, so the new history must be named in one shard; shard 6 is the lightest. Refs #101. Co-authored-by: Isaac <no-reply@databricks.com> * ci(oracle-gaps): place the test-root history's units in an e2e shard The e2e partition lists every unit by name: the fixed-seed, random, and determinism units of the new history go in shard e2e-3, beside the other constructed histories of #73. Refs #101. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): constructed histories for each shape of a config's test root The test root dimension (testRoot) becomes a set of shapes: the directory that holds the tests, the directory above the package, a directory that holds no test file, and a tests directory that is a symbolic link. Each new shape has its own constructed history, appended after the existing ones, and runs a setup file named from the root. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): share the test-root fixture row of the linked root, and state the copy read's limit Co-authored-by: Isaac <no-reply@databricks.com> * ci(oracle-gaps): place the new test-root histories in Windows shard 6 and e2e-3 Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): serve copies of test files only through the read of their path No plugin resolves a copy any more, so Vite keeps the absolute path that Vitest imports it by and reads it through serveCopies, under every test root. This removes the per-context roots, the package and root candidates, the load hook's copy lookup, and the wrapper's second copy of the path rule. serveCopies uses paths.mjs's posix. Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): install a pragma's environment where a root above the package resolves it Vitest resolves vitest-environment-<name> from the test root. A random base of the root-above-the-package history drew a pragma, and the environment in the package's node_modules was not found from the root, so the fixture's baseline failed in Vitest itself (seed 837819645). Co-authored-by: Isaac <no-reply@databricks.com> * test(oracle-gaps): a constructed history whose config plugin loads test files with a sync read The merge-gate review of #103 (F1) found that a config plugin whose own load hook reads each test file with readFileSync gets ENOENT on the runner's in-memory copy, once the runner's load no longer answers for it. No history had a plugin with a load hook. The new field loadPlugin is a drawn constant, set only by the new constructed history, appended after the test-root histories. Also corrects the test-root comments, which still described a copy resolver, and the Limits text on Vite's client environment. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracle-gaps): the wrapper's load serves a copy before the config's own plugins The pre plugin's load answers for an in-memory copy of a test file again, by its absolute id, so a config plugin whose load reads test files with a sync read never reads a copy's path (review of #103, F1). No plugin resolves a copy, so #101 stays fixed. Co-authored-by: Isaac <no-reply@databricks.com> --------- Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Nov 16, 2026
KyleAMathews
added a commit
that referenced
this pull request
Nov 16, 2026
…name that extends it #73's fix class 2 (isWithin in place of a prefix test in the driver) had no hostile mutant (D58): the driver's CLI path and package directory were always spelled alike, so a prefix test agreed with isWithin. A drawn field, siblingDir, places an in-repository package in a directory beside the CLI's whose name extends it (oracle-gaps-sibling-*), and a constructed history, appended last, refuses a run there. The driver now judges the inside label against the package's path from the CLI's directory, so a prefix test that attests it for the sibling fails loudly. A drive-letter case or separator difference is win32-only and is not generated. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the oracle-gaps gap report's static and end-to-end oracles on Windows, and fixes the Windows failures class by class. Windows static is blocking. Windows end-to-end stays non-blocking (C entry (c)) until the win32 supervisor lands in #79, which owns the win32 abort witness and the remaining win32 end-to-end failures.
Register entries this follows: Windows paths (forward slashes in keys, entry, and report paths; native paths only at fs calls; posix() acts only on win32), Windows stops (win32 has no signal a Node process receives: a stop goes through the test stop hook, observed as SIGINT), platform-scoped reach (a state is declared on every platform; a platform's witness reaches it), C (c) (Windows end-to-end non-blocking until #79), determinism under load (witnesses wait on marks, not on the clock), test-only hooks (
ORACLE_GAPS_TEST_*, inert unless set, documented in the README), and P7 / P12 (no narrowing to make a job pass; cut real cost first).The Linux 2/3 account
At
495b281(run 43598121853, Linux shard 2/3) the job ran 1010.9 s and hit the 900 s test timeout. The failure was a history that entereda run whose process aborted, a state the property did not declare. A failed constructed-history check then made fast-check shrink the history (six shrinks, about 134 s each), which accounts for the time. The earlier account of this as a harness change was wrong: a reviewer run of495b281with only9b22c71's harness restored still timed out, whiled7f437bpasses. The declaration is the fix, carried byd7f437b.Harness cleanup:
capture-setup.mjsloses the idle-profiling code that the revert brought back (theblockedmap, theblockerternary,longest, and theORACLE_GAPS_IDLE-gated write; that variable is outside the test namespace). The sampler's condition is the original one.Fix classes, with evidence
Each block: the law; the RED; the GREEN; the mutant or check that rejects the wrong behaviour; and the limits.
d7f437band later: Linux fixed-seed shards pass on the Arclet pool (r73-shard1-a48935192.7 s,r73-shard2-a48935189.3 s,r73-shard3-a48935210.1 s). Mutant: the abort state removed from the declared states, with the history still entering it:r73-mut-abortfails with "names undeclared states a run whose process aborted". Limits: the win32 witness is not in this PR (see fix(oracle-gaps): supervise the check on win32; log each CLI run in the e2e driver #79); on win32 the state is declared but its witness is the POSIX worker-thread kill, so the Windows end-to-end job stays non-blocking.daemonChecksadds each label from those observations;resetBeforeReadruns against a warm daemon (an earlier check through it, before the pause is armed withORACLE_GAPS_TEST_PAUSE_ARM), so the reset hits a session that is already warm (P7);ORACLE_GAPS_WAIT_MSis renamedORACLE_GAPS_TEST_WAIT_MS.r73-green-7000-daemon(history "an edit sequence through the daemon") passes and reaches both labels.r73-mut-m1): fails "did not enter a check whose daemon held it past the CLI's wait, then died".r73-mut-m3): fails "did not enter a check whose connection was reset".r73-mut-m2): passes. The driver parses the pid as a positive integer and does not signal until it has one, so the group is not killed (the reviewer's runpr73rv-m2-empty-markexited 137 before this change).r73-item3-red-old6s,r73-item3-old-hold0, both pass. The reason is in the code: the CLI checks its wait only after a request returns, so a paused daemon is never governed by the wait. So no timing RED exists for this state; the rule is enforced by construction, not by a load-dependent failure.posix()rule, andgap-report-path-spelling-oraclepins POSIX names as native; that is the check for this gate. Not run in this revision.Commit-message correction
The commit
a489353says it importsposixandrealPathOf. The only missing import wasposix;realPathOfwas imported but never used, and the import is removed in7000a93. The commit history is not rewritten.Test time
r73-dr-json):an edit sequence over the witness module195.6 s,every outcome class106.2 s,an edit sequence through the daemon65.4 s; the other 135 tests are 1.6 to 48 s each.pr-shards.mjs checksplits the tests exactly across five partitions.Windows results
Run
43659722330(nine shards, head7cdf04f): 1/9 7.3 min pass; 2/9 11.7 min fail ("every outcome class": the win32 abort run completes, expecteddone error … SIGABRT; #79's job); 3/9 8.5 min fail (the daemon history: a CLI internal error at the daemon-crash step, "This is a bug in oracle-gaps … out-daemon/error.log", the same win32 daemon-crash class as the reviewer's 1/6 atincremental); 4/9 2.5, 5/9 1.6, 6/9 2.4, 8/9 1.9, 9/9 1.6 min pass; 7/9 fails in 0.8 min ("the command line is too long": its pattern was 6,277 characters, which is why the partition has ten shards now).Closest to the budget: "every outcome class" takes 9.6 min on Windows at
104376d(11.7 min at7cdf04f), under the 10-minute limit (P12). On Linux it is 111 s: 82 s in two real CLI runs and 28 s in five stop runs, with no fixed wait to remove. It is one test and cannot be split; the margin is thin, and a regression there would fail the budget.Run
43661845394(ten shards, head104376d): 1/10 5.1 min pass; 2/10 9.6 min fail ("every outcome class", the win32 abort: #79's job); 3/10 9.0 min fail ("an edit sequence through the daemon", the win32 daemon-crash internal error above); 4/10 9.0 min fail ("a declaration fault: noTypescript": the stdout has 169 gap lines for 168 gaps, a count breach not yet diagnosed); 5/10 1.8, 6/10 2.8, 7/10 1.5, 8/10 1.2, 9/10 1.5, 10/10 1.5 min pass. Every end-to-end job is under 10 minutes. The blocking checks are green in that run.Not in this PR
process.abort()throughORACLE_GAPS_TEST_ABORT_AT, under the supervisor) and its hostile mutant: fix(oracle-gaps): supervise the check on win32; log each CLI run in the e2e driver #79. Until then the Windows end-to-end job is non-blocking under C entry (c).no-undeflint rule (a missing import reached a commit): issue Lint: enforce no-undef in oracle-gaps test files (a missing import reached a commit) #85. Lint config is unchanged here.Final round (head
227e2fd)r73-h1e-disarmed, fails). The disarmed history itself passes on the correct code (r73-g7-disarmed).r73-h2e-daemon). GREEN:r73-g7-daemonpasses.r73-m1f-daemon). M3 (no read pause) fails by reach on "a warm daemon's connection was reset" (r73-m3f-daemon).pr-shards.mjs checknow fails on any listed name that matches no test (the check is on for this partition, viaexactNames).227e2fd(run 43673840770): 1/10 4.7 min, 2/10 11.4 min (failing, non-blocking; over 10), 3/10 3.2, 4/10 0.9, 5/10 1.7, 6/10 3.9, 7/10 1.8, 8/10 1.7, 9/10 3.9, 10/10 1.7. The same test on3e7400e(run 43671444054) took 9.8 min. That is the job the budget depends on: two runs bracket the limit.daemonsOfreads the lock, notps. RED run 43661845394 (3/10), GREEN run 43671444054 (3/10 passes, 3.2 min). Closed.670280967/3e7400e): shard 1 passes, shard 2 passes (4 tests), shard 3 passes (32 tests). The grammar change (pauseEngages) is in every seed's histories; these runs are the re-verification.e2e-shards.mjscheck (CITestjob): the disarmed-pause constructed history and itsrandom seedanddeterminismvariants are listed in sharde2e-3.Split of "every outcome class" (P12), and the merge with main
a config that --config names ran, reached by the second half's lighter base). The stop-state counts: each stop state is reached by the half that owns its run.tests/gap-report-history-fingerprint.test.ts(main's, the generator rule) found thatpauseEngageswas inserted in the middle of the drawn record and was missing from the recorded lists. It is now the last field of the drawn record, and the lists have one new line each (pauseEngages,c93e2e40cd0c4b9f); the 53 recorded lines are unchanged. RED: run 43680525602 (Windows unit suites, Test, Test on Vitest 5). GREEN: run 43683794816 (those jobs pass).pauseEngagesisfc.constant(true)in the drawn record, set to false only by the disarmed constructed history. Canary (sampled histories at seed 20261105, before and after the field, all other fields identical) passes.origin/main(5726197, fix(oracle-gaps): a file a test reads through fs is an input of its daemon run #55) is merged as a normal merge commit. Main's constructed histories stay first; this branch's additions come after. The take-back mapping keeps both main's tornRows/laterSteps handling and the pauseEngages translation.Final check round (head
8746768)The seven items of the final check, one commit each. CI on head
8746768(run 43690320025): every blocking job passes, and the slowest job is 8.93 min (the longest is the Linux fixed-seed shard 3/3 at 8.05 min). The Windows end-to-end jobs stay non-blocking under C entry (c): 2/10 misses the idle state in item 5 (8.93 min), and 4/10 is the win32 abort of #79 (8.87 min).950b5e9). The first half keeps main's index 0, and the names of main's constructed histories are in main's order (checked againstorigin/main). The reach union on this head (Arclet, fixed seed, the three Linux shards): first half 82 states, second half 68, union 84, which equalsorigin/main's "every outcome class" exactly (none missing, none extra). Law: a constructed history's base is drawn at its index, so an appended history's base is new.random-only-killwhere the model saidnot-judged(runr73h-disarm-green). The history now pins its generated inputs to ones with no bound (8746768), and the Limits name the model gap for the maintainer. Random seeds keep the bound. This is a model gap, not a fix of the model.49be396,a101585). Its warm daemon is stopped before the pause is armed, so the paused check is served by another daemon. The warm label has a law check: a reset carries the warm label only when the daemon that paused served the warm check. The state "a warm daemon was replaced before the pause" is declared and entered.699c99fon the final head):resetBeforeRead recorded a warm daemon's connection was reset, but its daemon did not serve the warm check(runr73k-warm-red, exit 1).r73k-warm-green, exit 0 (8 s).8746768; none is on the PR branch.recorded a check whose daemon died during it without a daemon pause(r73k-h1-red, exit 1). GREENr73k-disarm-green, exit 0.did not enter a warm daemon's connection was reset(r73k-h2-red, exit 1). GREENr73k-daemon-green, exit 0.did not enter a check whose daemon held it past the CLI's wait, then died(r73k-m1-red, exit 1).did not enter a warm daemon's connection was reset, a check whose connection was reset(r73k-m3-red, exit 1).7c9fa02). Shard 2 now names both halves. On Arclet the shards took 192 s (1), 247 s (2) and 280 s (3) (r73k-s1..s3, all exit 0). The same Arclet pool measured the bug head's shard 3 at 469 s, and CI measured that head at 605 s (ratio 1.29), so the estimate for CI is about 6.5 min. CI's time for this head is the blocking check below. No budget was lowered.0404b5d): win32 reaches "a hang cut off for idleness" but missed it once (run 43683794816, job 132719872940). The sampler's quiet window is the timing dependency; the cause is not confirmed. The witness (a test-only hold at the hang) is owed, and the register's determinism-under-load rule forbids the current reliance on timing, so the maintainer must decide before the Windows job becomes blocking. Not platform-scoped: win32 reaches the state (job 132719872988).ORACLE_GAPS_TEST_SERVED_MARK(49be396).pr-shards.mjsreadsvitest list --json=<file>from a temporary file, so no package manager's stdout line reaches the parse (49be396).checkpasses on the final head.Not done, for the maintainer: the model's work-bound gap (item 1), the Windows idle witness (item 5), and a possible real cut in "a declaration fault: noTypescript": the agent that measured it found two identical cold CLI runs of about 48 s each on Arclet, and did not trace the second. Under P12 that cut comes before any sharding, so it is the first follow-up.
This pull request and its description were written by Isaac.