Skip to content

CI: run the tests on Windows #1693

Open
0 of 7 issues completed
Open

Description

@V3RON

Request: #1070

Problem

CI runs only on Linux. Simlock's Windows code is tested by passing platform: "win32" to a function on Linux, so nothing that only Windows does ever runs in CI: the daemon's named pipe, a batch file run through cmd.exe, a file Windows will not delete while it is open. A change that breaks Windows users merges green.

On a Windows 11 machine at 2df19f881, pnpm test:unit fails 1,664 of 9,544 tests, in 138 of 2,184 files. Most share a few causes: the in-memory fake filesystem reads only / paths while the code builds D:\… paths (about 710 tests); Android tests leave the platform to its default, so the code looks for emulator.exe in a fake SDK that holds emulator (260); tests listen on <folder>/daemon.sock, which Windows refuses (176); the repository's tooling tests call sh and python3 (201); a test deletes its folder while a file in it is still open (96); a test checks a fact Windows does not have, such as mode 0600 (about 20).

Who it is for

Contributors and agents who change Simlock and need to know, before merging, that the change still works for operators who run a worker on Windows. Also a contributor who works on a Windows machine and wants to run the unit tests there.

Outcome

Every push and pull request runs the unit and console DOM tests on a Windows runner, and the required Quality check fails when they fail. When the feature is done, every unit test file runs on Windows and passes, the repository's tooling included, for a contributor who runs pnpm test:unit from Git Bash with Python 3 installed.

How it works for the user

flowchart TD
  P[Contributor pushes a PR] --> L[Linux jobs, as today]
  P --> W[Unit tests on Windows]
  W --> R{All pass?}
  R -- yes --> Q[Quality green]
  R -- no, or cancelled --> X[Quality red: the Windows job's log names the failing test]
  L --> Q
Loading

The PR's checks list before and after:

Before                         After
  Static checks                  Static checks
  Unit tests                     Unit tests
  Timing tests                   Unit tests (Windows)
  E2E (1/3) (2/3) (3/3)          Timing tests
  Quality                        E2E (1/3) (2/3) (3/3)
                                 Quality   <- needs Unit tests (Windows) too

A contributor on Windows runs the same command as on a Mac, from Git Bash:

> pnpm test:unit
 Test Files  2184 passed (2184)        # today: 138 failed | 2046 passed
      Tests  9493 passed | 51 skipped  # today: 1664 failed | 7829 passed | 51 skipped

Examples

How the in-memory fake filesystem reads a path (B3):

Path a caller gives Today After
/home/a/b the entry /home/a/b the entry /home/a/b
D:\home\a\b a different entry from /home/a/b the entry /home/a/b
\home\a\b a different entry from /home/a/b the entry /home/a/b
C:\home\a\b and D:\home\a\b two different entries one entry, /home/a/b
D:/home/a/b (forward slashes) a different entry from /home/a/b the entry /home/a/b
home/a/b (relative) as today as today
realpath("D:\home\a\b") throws ENOENT (no such entry) /home/a/b: what the fake returns is always the / form
a symlink to D:\home\a a link to an entry that does not exist a link to /home/a
an error for D:\home\x names D:\home\x names the path as the caller gave it, or the / form of one the fake derived (a parent, a link target)

Which test files the Windows job runs (B2):

Host File on the list Today After
Windows yes no Windows job not run
Windows no no Windows job run
Linux or macOS yes or no run run

What could go wrong

  • Every PR waits for a Windows runner too; if it is much slower than Linux, merges take longer. Likely a few minutes more.
  • Windows deletes and locks files differently, so a test that is fine on Linux can fail now and then on Windows and turn Quality red. Possible while the folders are fixed.
  • A file still on the list is not run on Windows, so a defect only it would show stays unseen until its task lands.
  • An agent on a Mac cannot run a Windows-only failure itself; it reruns the Windows job instead, which is slower.

Words used

  • Windows job: the CI job Unit tests (Windows) that runs pnpm test:unit on a Windows runner.
  • The list: the files the Windows job leaves out while this feature is delivered, e.g. src/drivers/ios/physical.test.ts until its task fixes it. It only shrinks, and the last task deletes it.
  • In-memory fake filesystem: MemoryFilesystem (src/ports/filesystem.ts), the stand-in for the disk most unit tests use.
  • POSIX-only fact: behaviour Windows does not have, so a test of it is skipped there, e.g. a file created with mode 0600, a umask, a launchd plist.
  • Repository tooling: the agent scripts in .agents/scripts/ and their tests in src/*.test.ts, e.g. tool-guard.mjs, merge-pr.sh.

Non-goals

  • The fast e2e lane on Windows. Its harness finds and stops a daemon through a socket file and lsof (e2e/helpers/env.ts:802); porting it is its own feature.
  • The timing tests, the console lane and the static checks on Windows.
  • iOS on Windows. iOS driver tests run there, with every tool faked, like every unit test.
  • A path flavour injected into production code to suit the tests.

Completion conditions

  • On a pull request, the checks list shows Unit tests (Windows), and the Quality check fails when it fails.
  • On main, pnpm test:unit run from Git Bash on a Windows 11 host with Python 3 passes with no test file left out.
  • No list of files left out on Windows remains in the repository.

Open questions

none

Decisions

  • ADR 0087 — The unit tests run on Windows in CI, and a list of files still failing there only shrinks: the job, the fake filesystem reading both path styles, POSIX-only tests skipped on Windows, the list, product defects fixed where found, and tooling run on Windows (#1703).
  • Decided in the session: the Windows job runs pnpm test:unit, so the console DOM tests too, because that is what the Linux unit job runs and they already pass on Windows.
  • Decided in the session: the runner image is pinned (windows-2025), because every other job pins ubuntu-24.04.
  • Decided in the session: the job runs its steps in Git Bash (shell: bash), because the Linux jobs' steps are written for a POSIX shell and the runner ships Git Bash.
  • Decided in the session: .gitattributes makes every text file check out with LF line endings, because tests read the repository's files and compare text, and a Windows checkout may convert them to CRLF.
  • Decided in the session: a test of a POSIX-only fact is skipped with skipIf(process.platform === "win32"), because the e2e files already skip by platform that way.
  • Decided in the session: the files are fixed in five groups by folder after the first task, in parallel, because each group shares its causes and its fixes do not touch another group's files. The CLI group (The CLI, updater, agent-device, console and remaining folders pass their unit tests on Windows #1709) also owns every folder no other group names.
  • Decided in the session: an Android test's fake SDK is built with the driver's own path functions for the host (emulatorPath, adbPath and withWindowsExtension in sdk-paths.ts, I4, exported by The Android driver passes its unit tests on Windows #1707), so on Windows it holds .exe and .bat names, rather than giving the driver a platform option, because no source file's behaviour is changed to suit a test (ADR 0087 §2) and the Windows search then runs for real.
  • Decided in the session: the list is the union of the files that fail in three runs of the Windows job at The unit tests run on Windows in CI, with a shrinking list of the files still failing there #1705's head, run before the list is filled, because one run misses a file that fails only sometimes. A file a later task finds failing only sometimes is that file's group's to fix.
  • Maintainer's choice: the feature merges as one stack, as every feature does; main gets the Windows job with The Windows job's list of left-out test files is deleted #1712, which also fixes every file other changes broke on Windows meanwhile.
  • Decided in the session: The Windows job's list of left-out test files is deleted #1712's task makes those fixes as commits on the branch of the lowest PR of the stack whose Windows job the break turns red, and the stack sync carries them up, so every PR's CI is green at the gate; one fix check reviews them, as for a slow-lane loop. Because the gate needs every PR of the stack green, a fix only at the top would leave the lower ones red.
  • Decided in the session: the five group tasks each delete their lines from the one sorted list, so two of them that join the stack touch neighbouring lines; the stack sync keeps both sides' removals.
  • Decided in the session: The Android driver passes its unit tests on Windows #1707 exports withWindowsExtension (I4) with its first caller, and Ports, daemon and the server-side modules pass their unit tests on Windows #1706 waits for it, because an export no file imports fails Fallow.
  • Decided in the session: the Windows job's time limit is 30 minutes from the start, because the files that come back off the list are among the slowest.
  • Decided in the session: a contributor on Windows runs the tests from Git Bash with Python 3 installed (maintainer's choice), as the CI job does; toolchain.md says so.
  • Decided in the session: an agent reproduces a failure only the Windows job shows by rerunning that job on the base commit's CI run, because it cannot run Windows itself; toolchain.md says how.
  • Decided in the session: the ui-dom project gets no list, because none of its files fails on Windows.
  • Decided in the session: a test whose folder holds an open file closes it before the folder is removed; when the open file is the product's (a daemon that does not close a database it opened), the product is fixed.

Behaviours

Id Behaviour (invoker, action, what it sees) Seam Cases Task
B1 A contributor pushes or opens a PR: CI runs pnpm run test:unit on the Windows runner, and the required Quality check passes only when that job passed .github/workflows/ci.yml and .gitattributes, read by src/test-lanes.repo.test.ts the job windows-unit named Unit tests (Windows) runs on windows-2025, in Git Bash, and runs pnpm run test:unit with a 30-minute limit; quality's needs lists windows-unit; Quality's loop fails on failure and cancelled and passes on success (it already does for every need, ci.yml:143); .gitattributes holds * text=auto eol=lf #1705
B2 vitest, started on a Windows host, leaves out of the unit project every file on the list; on any other host, and in the ui-dom project, it leaves out nothing more than today windowsSkipped(platform) (I1), called by src/test-lanes.repo.test.ts, and vitest.config.ts's text win32: today's unit excludes plus each listed file; linux and darwin: today's excludes only; the ui-dom project unchanged on every host; every listed path is a tracked test file; the list is sorted and has no duplicate #1705
B3 A test writes, reads, lists, links, renames or removes an entry of the in-memory fake filesystem with a Windows path or a POSIX one: both name the same entry, and what the fake returns is the / form MemoryFilesystem from src/ports/index.ts the rows of the first Examples table: D:\a\b, \a\b, D:/a/b and /a/b are one entry, for writeFileAtomic then readFile, mkdirp then lstat, readdir, rename from one style to the other, and rm; C:\a and D:\a are one entry; realpath("D:\\a") returns /a; a symlink whose target is D:\a resolves to /a; a relative path is read as today; an error names the caller's path or the / form of one the fake derived #1705
B4 The Windows job runs every file under src/ports/, src/daemon/, src/state/, src/history/, src/auth/, src/gateway/, src/mcp/ and src/test-support/, none is on the list, and all pass the Windows job (B1) a test that listens takes its endpoint from resolveDaemonSocketPath, a named pipe on Windows; a test removes its folder only after every file in it is closed (removeUsageRigs, the relay daemons); a test that compares a temporary folder compares it after realpath, so a short RUNNER~1 name matches; a test of a POSIX-only fact (a mode, a umask, a launchd plist, a systemd unit) is skipped on win32; a test that scans the source compares /-joined relative paths; a daemon test whose Android SDK is faked builds it per B5's first case; a product defect a test shows is fixed with it #1706
B5 The Windows job runs every file under src/drivers/android/, none is on the list, and all pass the Windows job (B1) a fake SDK is built with sdk-paths.ts' own path functions for the host (emulatorPath, adbPath, and withWindowsExtension (I4) for the .bat tools), so on Windows it holds emulator.exe, adb.exe, avdmanager.bat and sdkmanager.bat; WorkingDirectoryFilesystem reads a Windows absolute path as absolute; a test that compares a path the code built with node:path builds the expected one with node:path, and one that came from the fake (realpath, readdir) in the / form; a test removes its folder only after every file in it is closed; a product defect a test shows is fixed with it #1707
B6 The Windows job runs every file under src/drivers/ios/ and src/core/, none is on the list, and all pass the Windows job (B1) a test that compares a device-set, root or plist path the code built builds the expected one with node:path, and one that came from the fake in the / form; a scripted process runner expects the command the code builds on the host; a test removes its folder only after state.db and every other file in it is closed (src/core/registry.test.ts); a product defect a test shows is fixed with it #1708
B7 The Windows job runs every file under src/cli/, src/updater/, src/agent-device/, src/instructions/, ui/, and every folder of src/ and packages/ that B4, B5, B6 and B8 do not name, none is on the list, and all pass the Windows job (B1) the updater's expected daemon start command is the host's node and the path joined on the host; a CLI test that starts a real daemon reaches it on its named pipe; a test that runs a tool from node_modules/.bin runs it the way Windows can start it (src/simlock-client/no-core-leak.test.ts runs tsc); a test of a POSIX-only fact is skipped on win32; a product defect a test shows is fixed with it #1709
B8 A contributor on Windows runs the repository's tooling tests (src/*.test.ts) from Git Bash: none is on the list, and all pass the Windows job (B1) a .sh script is run through bash, never by its path alone; a script or test that runs Python uses python3 when python3 --version prints Python 3., else python (on Windows 11 python3 is by default a Store stub that prints "Python was not found"); a fake command a test puts on PATH (a gh, a tsc, a pnpm) is found on Windows, its folder joined with path.delimiter and the command startable there; a script that prints a path prints it with /; a test that compares a script's output reads it with line endings normalized; a test of a POSIX-only fact (os.fork in slow-e2e.test.ts) is skipped on win32; a defect in a script a test shows is fixed with it #1710
B9 Once no file is on the list, the list, windowsSkipped and the list's exception to the testing rule are gone, and every file passes on Windows at the stack's head, those other changes merged to main broke meanwhile included src/test-lanes.repo.test.ts, and the Windows job (B1) a file another feature's merge broke on Windows is fixed by #1712's task, by B4 to B8's cases for its folder, in a commit on the lowest stack PR whose Windows job it turns red; src/test-support/windows-skipped.ts does not exist; vitest.config.ts excludes the same files on every host; docs/internal/agent-rules/testing.md rule 5 keeps the POSIX-only-fact sentence and holds no list exception #1712

Stored state

none

Interfaces

Id What Shape and bounds Errors and limits Where Task Used by
I1 The list and windowsSkipped src/test-support/windows-skipped.ts exports WINDOWS_SKIPPED: readonly string[], repository-relative test file paths joined with /, sorted, each once, and windowsSkipped(platform: NodeJS.Platform): readonly string[], the list on win32 and [] otherwise. vitest.config.ts imports it and gives the unit project exclude: [...today's excludes, ...windowsSkipped(process.platform)], today's excludes left as they are written. src/test-lanes.repo.test.ts imports the module and calls windowsSkipped with win32, linux and darwin. At #1705 it holds every file that fails in any of three runs of the Windows job at #1705's head a listed path that is not a tracked file, or a list out of order, fails src/test-lanes.repo.test.ts src/test-support/windows-skipped.ts, vitest.config.ts:23 #1705 #1706, #1707, #1708, #1709, #1710 (each removes its files), #1712 (deletes it)
I2 The Windows job windows-unit: named Unit tests (Windows), runs-on: windows-2025, permissions: contents: read, timeout-minutes: 30, defaults.run.shell: bash, the unit job's checkout, setup-node 22, Corepack and pnpm install --frozen-lockfile --ignore-scripts steps, then pnpm run test:unit; quality's needs gains windows-unit the job ends at 30 minutes, counted from its start .github/workflows/ci.yml:49, .github/workflows/ci.yml:129 #1705 —
I3 How the fake reads a path and what it returns Every MemoryFilesystem method that takes a path, and every symlink target it stores, reads an absolute one by dropping a leading drive letter ([A-Za-z]:) and reading \ as / before it finds the entry; a relative path is read as today. What it returns (realpath, readdir names, readlink) is the / form. Done in the class itself, not by wrapping it, so subclasses with private fields keep working none src/ports/filesystem.ts:500 #1705 #1706, #1707, #1708, #1709
I4 withWindowsExtension exported export function withWindowsExtension(path: string, platform: NodeJS.Platform, extension: ".bat" | ".exe"): string from src/drivers/android/sdk-paths.ts, unchanged otherwise: the path with the extension on win32, as it is elsewhere none src/drivers/android/sdk-paths.ts:20 #1707 #1706

Existing code

Commit 2df19f8.

Searches: rg -l 'process\.platform' src packages --glob '!*.test.ts' (16 files); rg -l 'from "node:path"' src --glob '!*.test.ts' (67 files, 8 of them import posix or win32); rg -l 'MemoryFilesystem' src --glob '*.test.ts' (122 files); rg -c '"/tmp/|"/home/|"/Users/' src packages ui --glob '*.test.*' (287 hits in 83 files); rg -n 'quality' src/test-lanes.repo.test.ts (3 hits); rg -n 'WorkingDirectoryFilesystem' src (6 hits in 4 files); rg -n 'python3' src/*.test.ts .agents/scripts (7 hits); pnpm exec vitest run --project unit --project ui-dom --reporter=json on Windows 11 23H2, Node 22 (1,664 failed, 138 files); the same five folders with the fake reading both styles (449 fixed, 5 newly failing).

Id Code (path:line) Today With this feature Handled by
X1 .github/workflows/ci.yml:49 the unit job runs pnpm run test:unit on ubuntu-24.04 (ci.yml:69); no job runs on Windows a second unit job runs on windows-2025 B1, I2
X2 .github/workflows/ci.yml:129 quality needs [static, unit, timing, e2e] and fails on any need that did not succeed or skip (ci.yml:143) it needs windows-unit too B1, I2
X3 src/test-lanes.repo.test.ts:143 reads the quality job out of ci.yml and checks it needs timing (:151); reads the unit block of vitest.config.ts and checks it names *.timing.test.ts (:146) also checks the Windows job, .gitattributes and the list; today's excludes stay written in the unit block, so :146 still holds; later, that the list is gone B1, B2, B9
X4 vitest.config.ts:23 the unit project's excludes are the same on every host; the ui-dom project has none (vitest.config.ts:40) the unit excludes gain windowsSkipped(process.platform); ui-dom unchanged B2, I1
X5 src/ports/filesystem.ts:509 the fake's root is the entry /; entries are a Map keyed by the exact path text (:501) each path is read per I3 before the lookup B3, I3
X6 src/ports/filesystem.ts:656 mkdirp splits on / only, so D:\home\x is one segment it sees /home/x B3, I3
X7 src/ports/filesystem.ts:983 parentPath cuts at the last / only; joinMemoryPath joins with / (:979) unchanged; they see only paths and link targets already read per I3 B3, I3
X17 src/ports/filesystem.ts:766 realpath returns the internal key after following links; readlink the target as stored (:841); a link target is resolved as raw text (:957) keys and stored targets are the / form, so all three return it B3, I3
X18 src/drivers/android/test-sdk-fixtures.ts:263 WorkingDirectoryFilesystem.realpath treats a path that does not start with / as relative, so D:\\x becomes <wd>/D:\\x it treats a Windows absolute path as absolute B5
X19 src/drivers/android/index.ts:880 discoverSdk calls locateSdk with no platform, so it searches for the host's names (sdk-paths.ts:69); AndroidDriverOptions has no platform (index.ts:110) unchanged; fake SDKs are built with the host's names B5
X20 src/simlock-client/no-core-leak.test.ts:53 runs node_modules/.bin/tsc with execFileSync and no shell, which Windows cannot start (both tests fail) runs it the way Windows can start it B7
X21 src/slow-e2e.test.ts:60 starts an idle process with Python's os.fork() and os.setsid(), which Windows lacks skipped on win32 B8
X8 src/core/device-root.ts:1 builds the root, staging and marker paths with the host's node:path (resolve, join), so on Windows D:\home\agent\.simlock\devices\ios; 671 tests fail where it assembles the root (device-root.ts:121) unchanged; the fake reads the path it builds B3, B6
X9 src/drivers/android/sdk-paths.ts:96 emulatorPath adds .exe on win32, and the platform defaults to process.platform (:95), so a fake SDK without .exe names is missing on Windows (260 tests fail with SdkMissingError) fake SDKs are built with these functions for the host B4, B5
X10 src/ports/paths.ts:75 resolveDaemonSocketPath gives a named pipe on win32 and <dir>/daemon.sock elsewhere; daemon tests listen on a joined daemon.sock instead (176 tests fail with listen EACCES or connect ENOENT) the tests take their endpoint from it B4, B7
X11 src/history/testing.ts:178 removeUsageRigs removes the rig folders without stopping their history first, so history.db is still open (EBUSY on Windows) the history is stopped before the folder is removed B4
X12 src/ports/windows-command.ts:35 childLaunch runs .bat/.cmd through cmd.exe on win32; its tests pass win32 on Linux runs for real on the Windows job too B4
X13 docs/internal/agent-rules/testing.md:75 rule 5: never skip, disable or quarantine a test to get to green #1705 adds two sentences (ADR 0087): a test of a POSIX-only fact is skipped on Windows, which is not skipping to get to green; and the Windows list is an exception until the feature ends. #1712 removes the second B1, B9
X14 docs/internal/agent-rules/toolchain.md:67 names the CI jobs the Quality check needs names the Windows job too; says a contributor on Windows runs the tests from Git Bash with Python 3; says a failure only the Windows job shows is reproduced by rerunning that job on the base commit's CI run (gh run rerun <run> --job <job>) B1
X15 e2e/helpers/env.ts:802 pidsHoldingSocket finds a daemon by its socket file and lsof; on Windows it finds none, so the e2e lane hangs unchanged: the e2e lane is a non-goal —
X16 .gitattributes:4 holds one merge=union line; no line-ending rule adds * text=auto eol=lf, checked by B1 B1

Tasks

flowchart LR
  A[#1705 Windows job, list, fake] --> C
  C --> B[#1706 ports, daemon]
  A --> C[#1707 Android driver]
  A --> D[#1708 iOS driver, core]
  A --> E[#1709 CLI and the rest]
  A --> F[#1710 tooling]
  B --> G[#1712 delete the list]
  C --> G
  D --> G
  E --> G
  F --> G
Loading

Completion conditions:

Open work in the same area:

Written by an agent.

Activity

  1. added
    feature:specSpec in progress, business and technical together.
    os:windowsRuns on, or affects, Windows hosts
    on Nov 18, 2026
  2. V3RON commented on Nov 18, 2026

    @V3RON
    ContributorAuthor

    Design posted: every business section, Decisions (ADR 0087, PR #1703), Behaviours B1–B9, Interfaces I1–I3 and Existing code X1–X16, from a unit run on Windows 11 at 2df19f8.

    Written by an agent.

  3. V3RON commented on Nov 18, 2026

    @V3RON
    ContributorAuthor

    Spec updated: I4 (T1 exports withWindowsExtension); Task cells and Used by now name #1705 to #1710 and #1712; Tasks written.

    Written by an agent.

  4. added
    feature:plannedSplit into tasks. Never picked up itself.
    and removed
    feature:specSpec in progress, business and technical together.
    on Nov 18, 2026
  5. V3RON commented on Nov 18, 2026

    @V3RON
    ContributorAuthor

    Spec updated: Decisions (one stack, kept by the maintainer; catch-up commits on the lowest red stack PR; I4 exported by #1707; list conflicts keep both removals), B9, I4 and Tasks' open work. All seven tasks are task:ready after the tasks check; #1705 is runnable.

    Written by an agent.

  6. V3RON commented on Nov 18, 2026

    @V3RON
    ContributorAuthor

    Delivery run

    Merged: none (stack #1703 → #1722 open)
    Parked: none
    Waiting: #1705: paused after its build (DesktopWindows offline); next is the audit. #1706–#1710 wait on #1705. #1712 waits on #1706–#1710.
    Decided: #1705: a ui-dom file failing on Windows would go on the same list → none failed, so it was not needed (Assumption line)

    Written by an agent.

  7. V3RON commented on Nov 18, 2026

    @V3RON
    ContributorAuthor

    Delivery run

    Merged: none (stack #1703 → #1722 open)
    Parked: #1705 — review cap reached with one blocking test-logic finding open (R1, src/ports/memory-filesystem-paths.test.ts:50); PR #1722 stays draft
    Waiting: #1706, #1707, #1708, #1709, #1710 — on #1705; #1712 — on #1706–#1710
    Decided: none

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature:plannedSplit into tasks. Never picked up itself.os:windowsRuns on, or affects, Windows hosts

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions