From 27776ba6fdcb9b9a8877de207783dd5532ef6336 Mon Sep 17 00:00:00 2001 From: Mike Arpaia Date: Thu, 24 Sep 2026 16:32:31 -0600 Subject: [PATCH 1/4] Document and execute tutorial commands across backends and shells --- .github/workflows/live-shutdown-windows.yml | 14 +- docs/README.md | 1 + .../tutorial-command-validation.md | 34 ++ docs/tutorials/README.md | 2 + docs/tutorials/analysis.md | 3 + docs/tutorials/biophysics-and-growth.md | 7 + docs/tutorials/commands.md | 179 ++++++++++ docs/tutorials/discrete-state-and-contacts.md | 4 + docs/tutorials/flow-solvers.md | 4 +- docs/tutorials/getting-started.md | 7 +- docs/tutorials/intracellular-dynamics.md | 7 + docs/tutorials/microfluidics.md | 6 +- docs/tutorials/nutrient-validation.md | 2 + docs/tutorials/signaling.md | 5 + docs/tutorials/simbol.md | 5 + environments/cuda/README.md | 2 + environments/metal/README.md | 2 + python/tests/test_tutorial_commands.py | 321 ++++++++++++++++++ viewer/README.md | 6 +- 19 files changed, 601 insertions(+), 10 deletions(-) create mode 100644 docs/development/tutorial-command-validation.md create mode 100644 docs/tutorials/commands.md create mode 100644 python/tests/test_tutorial_commands.py diff --git a/.github/workflows/live-shutdown-windows.yml b/.github/workflows/live-shutdown-windows.yml index 5fee6e0..62f7de5 100644 --- a/.github/workflows/live-shutdown-windows.yml +++ b/.github/workflows/live-shutdown-windows.yml @@ -1,4 +1,4 @@ -name: Windows live-session shutdown +name: Windows CLI and live-session checks on: pull_request: @@ -7,10 +7,14 @@ on: - CMakeLists.txt - cpp/** - python/** + - docs/tutorials/** + - docs/development/tutorial-command-validation.md + - environments/**/README.md + - viewer/README.md - pyproject.toml - uv.lock push: - branches: [master, marpaia/17] + branches: [master, marpaia/17, marpaia/24] workflow_dispatch: permissions: @@ -31,6 +35,7 @@ jobs: CMAKE_BUILD_PARALLEL_LEVEL: "2" # Exercise the host reference engine without requiring a GPU toolkit. CMAKE_ARGS: -DCM_ENABLE_METAL=OFF -DCM_ENABLE_CUDA=OFF -DCM_BUILD_TESTS=OFF + MICROSIMULATOR_COMMAND_REPORT: build/shutdown-evidence/tutorial-commands steps: - name: Check out source uses: actions/checkout@v6 @@ -60,6 +65,11 @@ jobs: python/tests/test_viewer_server.py python/tests/test_viewer_shutdown.py -v --tb=short --junitxml=build/shutdown-evidence/tests.xml + - name: Execute the documented PowerShell CLI commands + run: >- + uv run --no-sync python -m pytest python/tests/test_tutorial_commands.py + -v --tb=short --junitxml=build/shutdown-evidence/tutorial-commands.xml + - name: Upload shutdown evidence if: ${{ always() }} uses: actions/upload-artifact@v7 diff --git a/docs/README.md b/docs/README.md index e26712f..64d0031 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,7 @@ MicroSimulator models microbial populations in microfluidic environments, connec | --- | --- | | Understand how devices, flow, and cells fit together | [Microfluidics modeling guide](microfluidics.md) | | Run a first simulation | [Getting started](tutorials/getting-started.md) | +| Select a backend or use PowerShell | [Copyable tutorial commands](tutorials/commands.md) | | Build a trap or channel with growth and washout | [Microfluidic devices](tutorials/microfluidics.md) | | Choose a flow solver and assess its numerical behavior | [Flow models](microfluidics.md#choosing-a-flow-model) and [flow benchmarks](tutorials/flow-solvers.md#numerical-evidence) | | Measure nutrient penetration and growth | [Controlled nutrient study](tutorials/nutrient-validation.md) | diff --git a/docs/development/tutorial-command-validation.md b/docs/development/tutorial-command-validation.md new file mode 100644 index 0000000..c394902 --- /dev/null +++ b/docs/development/tutorial-command-validation.md @@ -0,0 +1,34 @@ +# Tutorial command verification + +The [command guide](../tutorials/commands.md) separates shell argument handling, backend availability, and scientific backend validation. The executable examples are read directly from its marked Markdown blocks by [`test_tutorial_commands.py`](../../python/tests/test_tutorial_commands.py). + +## Coverage and limits + +The command tests exercise human-readable and JSON device discovery; the identical 100-step trap command on every available backend; explicit unavailable-backend failures; JSON string arguments and line continuation; repository, model, checkpoint, and output paths containing spaces; restored seed/parameters/source digest; and uninterrupted versus resumed controller/native state. They launch the documented trap, growth, and resumed live sessions in sequence on port 8765, save a checkpoint, send authenticated Stop, observe Stopped, and verify each process returns successfully before the next launch. The CLI tests supply minimal static assets and do not claim browser rendering coverage. + +PowerShell tests set `$PSNativeCommandArgumentPassing = 'Standard'` exactly as documented. Windows uses `pwsh` 7.3+; Windows PowerShell 5.1 and `cmd.exe` are not covered. POSIX examples are executed in `sh`, Bash, and Zsh when those shells are installed. A missing shell is explicitly skipped. Reports include the exact script text, exit status, stdout/stderr, platform, Python version, shell version, and enumerated devices. + +The [Windows CLI and live-session workflow](../../.github/workflows/live-shutdown-windows.yml) builds a native CPU extension on `windows-2025`, executes these command tests and the existing live-session tests, and uploads reports. Its GPU backends are deliberately disabled. The shutdown tests additionally generate an actual Windows `CTRL_C_EVENT` in an isolated console; they do not simulate a human keyboard press in every terminal application. The viewer guide retains a [manual terminal procedure](../../viewer/README.md#stop-one-model-and-start-another). + +## Execution record + +The local command run on 2026-09-24 passed all nine tests with Python 3.12.8, Bash 3.2.57, Zsh 5.9, and PowerShell 7.6.4. The associated pull request links the exact source SHA and hosted workflow run; reports are retained as CI artifacts. + +| Platform and shell | Backend execution | Command coverage | +| --- | --- | --- | +| macOS, POSIX `sh`, Bash, Zsh | Native CPU and Apple M4 Max Metal | Passed: discovery, 100-step CPU/Metal trap, JSON/space paths, headless CPU resume, live CPU checkpoint/Stop/restart | +| macOS, PowerShell 7.6.4 with Standard arguments | Native CPU and Apple M4 Max Metal | Passed: same commands, PowerShell backtick continuation and string quoting | +| Windows Server 2025, PowerShell 7 with Standard arguments | Native CPU | Hosted command verification pending; see the pull request workflow result | +| NVIDIA CUDA hardware | Not available in the command-verification campaign | Unavailable-backend error checked; no CUDA runtime or numerical claim | + +The test runner reports actual Metal availability for each macOS run. This run executed Metal headless trap commands; its JSON scenario, resume, and live-session examples selected CPU. A skipped/unavailable Metal result does not count as GPU execution. Backend support still requires the independent [native and application conformance gates](validation.md), even when a tutorial smoke command succeeds. + +## Reproduce + +After installing the development environment, run: + +```console +uv run --no-sync python -m pytest python/tests/test_tutorial_commands.py -v +``` + +Set `MICROSIMULATOR_COMMAND_REPORT` to a new output directory to retain per-shell JSON reports; the Windows workflow supplies this environment variable. Run from an environment with loopback networking enabled and port 8765 free. The tests use their own temporary working directories and never overwrite tutorial outputs in the source checkout. diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md index e55816a..4d363c2 100644 --- a/docs/tutorials/README.md +++ b/docs/tutorials/README.md @@ -2,6 +2,8 @@ These tutorials explore cells in microfluidic devices through runnable models: geometry and flow supply the environment, while growth, mechanics, and circuits determine how populations respond. The [modeling guide](../microfluidics.md) introduces the full workflow and its assumptions. Each example can also be used independently. +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + ## Start here Follow [getting started](getting-started.md) to run a nutrient-fed trap, configure the viewer, and resume a checkpoint. Then choose a path below. diff --git a/docs/tutorials/analysis.md b/docs/tutorials/analysis.md index 120bdc5..4fc6564 100644 --- a/docs/tutorials/analysis.md +++ b/docs/tutorials/analysis.md @@ -2,6 +2,8 @@ MicroSimulator separates simulation output into three artifacts: +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + - a checkpoint is an exact, integrity-checked restart artifact; - a scene is an immutable presentation snapshot; and - an analysis dataset is an immutable Parquet/Zarr projection with schemas and provenance. @@ -13,6 +15,7 @@ Use checkpoints for resuming, scenes for viewing, and datasets for statistics. ```console uv run microsimulator run \ --model examples/tutorials/biophysics.py \ + --backend cpu \ --parameter scenario='"basics"' \ --seed 42 \ --steps 200 \ diff --git a/docs/tutorials/biophysics-and-growth.md b/docs/tutorials/biophysics-and-growth.md index 0b72a64..02ef7e3 100644 --- a/docs/tutorials/biophysics-and-growth.md +++ b/docs/tutorials/biophysics-and-growth.md @@ -2,6 +2,8 @@ This tutorial introduces cell geometry, growth, division, lineage, cell types, mechanical constraints, and competition. Its five runnable scenarios are defined in `examples/tutorials/biophysics.py`. +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + ## 1. A founder that grows and divides Run the basic model: @@ -9,6 +11,7 @@ Run the basic model: ```console uv run microsimulator view \ --model examples/tutorials/biophysics.py \ + --backend cpu \ --parameter scenario='"basics"' \ --seed 42 \ --dt 0.05 \ @@ -49,6 +52,7 @@ During new tutorial construction, each founder target is sampled exactly once fr ```console uv run microsimulator view \ --model examples/tutorials/biophysics.py \ + --backend cpu \ --parameter scenario='"two_types"' \ --seed 42 \ --dt 0.02 \ @@ -62,6 +66,7 @@ The model places type 0 at `x = -10` and type 1 at `x = 10`. Both use the same g ```console uv run microsimulator view \ --model examples/tutorials/biophysics.py \ + --backend cpu \ --parameter scenario='"short_cells"' \ --seed 42 \ --dt 0.01 \ @@ -75,6 +80,7 @@ This scenario lowers the post-founder division length to produce short spherocyl ```console uv run microsimulator view \ --model examples/tutorials/biophysics.py \ + --backend cpu \ --parameter scenario='"competition"' \ --seed 7 \ --dt 0.01 \ @@ -98,6 +104,7 @@ Use `Growth rate` coloring to see the active zone and `Cell type` coloring to se ```console uv run microsimulator view \ --model examples/tutorials/biophysics.py \ + --backend cpu \ --parameter scenario='"box"' \ --seed 42 \ --dt 0.01 \ diff --git a/docs/tutorials/commands.md b/docs/tutorials/commands.md new file mode 100644 index 0000000..72e3497 --- /dev/null +++ b/docs/tutorials/commands.md @@ -0,0 +1,179 @@ +# Tutorial commands by backend and shell + +Run these commands from the repository root. They use the same Python models on every backend. `--backend` selects `cpu`, `metal`, or `cuda`; `--device-index` selects the zero-based device within that backend. Model paths, JSON parameters, seeds, and timesteps do not acquire backend-specific syntax. + +## Prepare and discover devices + +Install Python 3.12, `uv`, CMake, Ninja, and a C++23 compiler. Windows CPU builds need the Visual Studio C++ build tools and Windows SDK available to the build process; the tested Windows route uses PowerShell 7 on `windows-2025`. Metal needs macOS and an accessible Apple GPU. CUDA additionally needs a CUDA-enabled build, an NVIDIA GPU, and a compatible toolkit/driver; see the [Metal](../../environments/metal/README.md) and [CUDA](../../environments/cuda/README.md) environment guides. + +```console +uv sync --locked --group dev +``` + +The commands below use `uv run --no-sync` after this installation so running a tutorial does not change the environment. Run `uv sync --locked --group dev` again after changing dependencies or checking out another version. No virtual-environment activation is required. + + +```console +uv run --no-sync microsimulator devices +``` + + +```console +uv run --no-sync microsimulator devices --json +``` + +An entry such as `metal:0 ...` identifies backend `metal`, device index `0`. CPU also uses index `0`. For a second enumerated GPU, change only `--device-index 0` to `--device-index 1`. `devices` reports what this installation can construct; it does not certify scientific conformance. CPU is the reference backend, Metal is the supported Apple backend, and CUDA remains under development pending the required NVIDIA runtime and application gates. A CUDA compile check alone does not establish runtime support. See the [validation policy](../development/validation.md). + +Requesting an unavailable backend/device fails with an error such as `backend cuda device 0 is unavailable (0 device(s) found)` and a nonzero exit status. There is no automatic CPU fallback. Choose an available backend explicitly or install/configure the requested backend. Do not change a backend flag merely to label a CPU result as a GPU result. + +## Choose a shell + +Single-line `console` commands on this page work in POSIX `sh`, Bash, Zsh, and PowerShell 7.3 or newer configured as below. Multiline `sh` blocks use a trailing backslash; multiline `powershell` blocks use a trailing backtick. The continuation character must be the last character on the line, with no trailing spaces or comment. Do not paste the backslashes from a `sh` block into PowerShell; use the PowerShell block or join the command onto one line. + +For PowerShell, open `pwsh` and set its native argument-passing mode once in that session: + + +```powershell +$PSNativeCommandArgumentPassing = 'Standard' +``` + +The tested Windows route is PowerShell 7.3+ (`pwsh`), not Windows PowerShell 5.1 (`powershell.exe`) or Command Prompt (`cmd.exe`). Their native JSON quoting differs; install/use `pwsh` for these examples. A shell passing the arguments correctly does not establish GPU availability on that operating system. CUDA's current hardware conformance scripts target Linux. + +## Run the same trap on CPU, Metal, or CUDA + +Choose one available backend. These commands all run [`examples/microfluidic_trap.py`](../../examples/microfluidic_trap.py), with seed `42`, no model parameter overrides, 100 steps, `dt=0.02`, device index `0`, and checkpoints every 20 steps. Only the backend and output filename differ. Keeping separate output names prevents one backend's result from replacing another's. + +CPU: + + +```console +uv run --no-sync microsimulator run --model examples/microfluidic_trap.py --backend cpu --device-index 0 --seed 42 --steps 100 --dt 0.02 --checkpoint-every 20 --output "results/tutorial runs/trap-cpu.json" +``` + +Metal: + + +```console +uv run --no-sync microsimulator run --model examples/microfluidic_trap.py --backend metal --device-index 0 --seed 42 --steps 100 --dt 0.02 --checkpoint-every 20 --output "results/tutorial runs/trap-metal.json" +``` + +CUDA: + + +```console +uv run --no-sync microsimulator run --model examples/microfluidic_trap.py --backend cuda --device-index 0 --seed 42 --steps 100 --dt 0.02 --checkpoint-every 20 --output "results/tutorial runs/trap-cuda.json" +``` + +The quoted output paths contain spaces. Parent directories are created automatically. Final and periodic outputs must be new: choose another output name on a second run. `--overwrite` is an explicit replacement option, not a prerequisite for running a tutorial. Identical seeds define the same experiment; floating-point results across backends must be compared using the project's numerical tolerances, not an assumption of byte-identical checkpoints. + +## JSON parameters and model paths with spaces + +`--parameter` takes one `NAME=JSON` argument. For a JSON string, the shell must preserve the inner double quotes: `'scenario="basics"'` reaches Python as `scenario="basics"`. A bare `scenario=basics` is invalid JSON. Numeric and Boolean examples are `--parameter copies_per_cell=6` and `--parameter enabled=true`, when the selected model defines those parameters. Repeat `--parameter` for additional names. + +This example copies the self-contained growth model to a path containing spaces, then runs its `basics` scenario. Choose the block for your shell. Both blocks describe the same run and output, so run only one. + +POSIX `sh`, Bash, or Zsh: + + +```sh +mkdir -p "results/tutorial models" +cp examples/tutorials/biophysics.py "results/tutorial models/biophysics.py" +``` + + +```sh +uv run --no-sync microsimulator run \ + --model "results/tutorial models/biophysics.py" \ + --parameter 'scenario="basics"' \ + --backend cpu --device-index 0 --seed 42 \ + --steps 10 --dt 0.02 \ + --output "results/tutorial runs/basics.json" +``` + +PowerShell 7.3+ with `Standard` argument passing: + + +```powershell +New-Item -ItemType Directory -Force "results/tutorial models" | Out-Null +Copy-Item examples/tutorials/biophysics.py "results/tutorial models/biophysics.py" +``` + + +```powershell +uv run --no-sync microsimulator run ` + --model "results/tutorial models/biophysics.py" ` + --parameter 'scenario="basics"' ` + --backend cpu --device-index 0 --seed 42 ` + --steps 10 --dt 0.02 ` + --output "results/tutorial runs/basics.json" +``` + +To run this scenario on a GPU, replace `--backend cpu` with an enumerated `metal` or `cuda` backend and choose a new output filename. Keep the model, `scenario`, seed, step count, and timestep unchanged for a comparison. Quotes also protect an absolute repository or model path containing spaces; on Windows, forward slashes in these Python CLI paths are accepted. + +## Resume saved parameters and state + +Resume the preceding scenario for ten additional steps: + + +```console +uv run --no-sync microsimulator run --model "results/tutorial models/biophysics.py" --resume "results/tutorial runs/basics.json" --backend cpu --device-index 0 --steps 10 --dt 0.02 --output "results/tutorial runs/basics-resumed.json" +``` + +For native `--model ... --resume ...`, the CLI obtains the seed and parameters from the checkpoint, restores controller/random/native state, and checks the model file's SHA-256 before executing it. Do not pass `--parameter` on resume: the CLI rejects it. `--seed` is a construction option and does not override the saved resume seed; omit it here. The model file may move, but its bytes must match the saved digest. Keep the original model source when updating a checkout. `--steps` is an additional step count, and `--dt` remains an explicit choice; retain the original timestep when continuing the same experiment. An available backend/device can be selected explicitly for resume; that does not guarantee bitwise equality across backends. + +To continue the CPU trap from above: + + +```console +uv run --no-sync microsimulator run --model examples/microfluidic_trap.py --resume "results/tutorial runs/trap-cpu.json" --backend cpu --device-index 0 --steps 100 --dt 0.02 --output "results/tutorial runs/trap-cpu-resumed.json" +``` + +For a Metal or CUDA trap checkpoint, change both `--resume` and `--output` to that run's filenames and select the intended available backend. Controller-backed checkpoints need their original `--model`; a checkpoint is data, not a substitute for model behavior. + +## Live view, Stop, and restart + +Build the browser assets once from the repository root: + +```console +pnpm --dir viewer install +pnpm --dir viewer build +``` + +Start a CPU trap session. Open the tokenized loopback URL printed in the terminal, or append `--open` to open it automatically. The explicit `--viewer-dist` resolves from the current directory. + + +```console +uv run --no-sync microsimulator view --model examples/microfluidic_trap.py --backend cpu --device-index 0 --seed 42 --dt 0.02 --viewer-dist viewer/dist --checkpoint-output "results/tutorial runs/live-trap.json" +``` + +To use a GPU, the only simulation selection changes are `--backend metal` or `--backend cuda` and, if needed, `--device-index`. `--port` defaults to `8765`; choose another free port explicitly if it is occupied. Use the new tokenized URL after each launch. + +| Control or action | Effect | +| --- | --- | +| Pause | Stops continuous playback; keeps the model process and current state available. | +| Reset | Rebuilds this session's original model. For a resumed session, reloads its starting checkpoint. Does not select a different model. | +| Close the browser | Disconnects and pauses the session; the server remains available for reconnection. | +| Checkpoint | Saves to `--checkpoint-output`; use this before Stop when restartable state is needed. | +| Stop session or one terminal Ctrl+C | Finishes the current individual step or checkpoint write, drains the worker, releases the port, and returns to the prompt. No automatic checkpoint is written. | + +Wait for **Stopped** and the terminal prompt before starting another command. A long `--frame-steps` batch is interrupted between individual steps; Stop does not forcibly interrupt a single model callback or solver. After stopping the trap, launch another model on the same default port: + + +```console +uv run --no-sync microsimulator view --model examples/tutorials/biophysics.py --parameter 'scenario="basics"' --backend cpu --device-index 0 --seed 42 --dt 0.02 --viewer-dist viewer/dist +``` + +After stopping that session, the saved headless scenario can also be opened live: + + +```console +uv run --no-sync microsimulator view --model "results/tutorial models/biophysics.py" --resume "results/tutorial runs/basics.json" --backend cpu --device-index 0 --dt 0.02 --viewer-dist viewer/dist +``` + +The [viewer guide](../../viewer/README.md#stop-one-model-and-start-another) describes shutdown checks, including the distinction between a generated Windows console Ctrl+C event and a human keyboard press in a particular terminal application. + +## Executed-command coverage + +[`test_tutorial_commands.py`](../../python/tests/test_tutorial_commands.py) executes the marked command blocks above through the actual shell, in a temporary repository path containing spaces. It checks device discovery, each available backend's trap checkpoint, unavailable-backend errors, JSON string parameters, quoted model/output/resume paths, saved provenance, continuation syntax, resume equivalence, and real live-session Stop/restart on the same port. Live CLI tests use minimal static assets; browser control behavior is covered separately by the [viewer shutdown tests](../../python/tests/test_viewer_shutdown.py). + +The [Windows CLI and live-session workflow](../../.github/workflows/live-shutdown-windows.yml) runs PowerShell commands against a freshly built native CPU extension and uploads shell/platform/backend/command evidence. The [command verification record](../development/tutorial-command-validation.md) states which platforms, shells, and hardware were actually exercised. Missing GPU hardware is recorded as unavailable, never counted as a passing GPU execution. These smoke checks establish command behavior, not complete backend conformance. diff --git a/docs/tutorials/discrete-state-and-contacts.md b/docs/tutorials/discrete-state-and-contacts.md index c7b8e5e..0f16879 100644 --- a/docs/tutorials/discrete-state-and-contacts.md +++ b/docs/tutorials/discrete-state-and-contacts.md @@ -2,6 +2,8 @@ This tutorial uses plasmid segregation and conjugation to show how discrete biological state, stochastic events, and contact-dependent behavior fit into a MicroSimulator model. +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + New founders preserve their requested length unless it exceeds the single sampled division target. This also handles the rare short Gaussian target in the conjugation model without rejection sampling. Checkpoint restoration keeps stored lengths and targets. See [founder initialization](biophysics-and-growth.md#length-and-volume). ## 1. Incompatible plasmid segregation @@ -9,6 +11,7 @@ New founders preserve their requested length unless it exceeds the single sample ```console uv run microsimulator view \ --model examples/tutorials/plasmid_segregation.py \ + --backend cpu \ --parameter copies_per_cell=10 \ --seed 42 \ --dt 0.02 \ @@ -71,6 +74,7 @@ Contact graphs are derived on demand and have no fixed scientific contact cap; a ```console uv run microsimulator view \ --model examples/tutorials/conjugation.py \ + --backend cpu \ --parameter transfer_probability=0.1 \ --seed 42 \ --dt 0.02 \ diff --git a/docs/tutorials/flow-solvers.md b/docs/tutorials/flow-solvers.md index 27ec792..d41091a 100644 --- a/docs/tutorials/flow-solvers.md +++ b/docs/tutorials/flow-solvers.md @@ -2,8 +2,10 @@ The [pillar-channel model](../../examples/tutorials/pillar_channel.py) combines cylindrical walls, a depth-integrated flow calculation, attached founder lineages, and released daughters: +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + ```console -uv run microsimulator view --model examples/tutorials/pillar_channel.py --seed 7 --dt 0.01 --backend metal --open +uv run microsimulator view --model examples/tutorials/pillar_channel.py --seed 7 --dt 0.01 --backend cpu --open ``` ## Geometry and flow diff --git a/docs/tutorials/getting-started.md b/docs/tutorials/getting-started.md index b475f7f..f057599 100644 --- a/docs/tutorials/getting-started.md +++ b/docs/tutorials/getting-started.md @@ -2,6 +2,8 @@ Start with a cell trap supplied by a flowing nutrient channel. This model combines device walls, a steady flow solve, solute transport, nutrient-dependent growth, and cell motion. The [microfluidics tutorial](microfluidics.md) explains the model, and the [modeling guide](../microfluidics.md) introduces the broader workflow. +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + ## Prepare the workspace MicroSimulator requires Python 3.12, CMake, Ninja, a C++23 compiler, and `uv`. From the repository root: @@ -66,20 +68,19 @@ uv run microsimulator view \ --open ``` -The viewer can play, pause, step, reset, and request a checkpoint. For the trap, enable a nutrient signal slice and choose `Growth rate` coloring to inspect the population alongside its environment. Other models can use `Species` coloring for intracellular channels or `Cell type` for strain or discrete-state categories. Selecting a cell shows its stable ID, lineage parent, geometry, type, growth rate, and ordered species values. +The viewer can play, pause, step, reset, request a checkpoint, and stop the session. Pause retains the current process/state; Reset rebuilds this session's starting model; closing the browser pauses for reconnection. Stop session or terminal Ctrl+C releases the server after current work completes, without automatically saving. See [Stop and restart](commands.md#live-view-stop-and-restart) before launching another model. For the trap, enable a nutrient signal slice and choose `Growth rate` coloring to inspect the population alongside its environment. Other models can use `Species` coloring for intracellular channels or `Cell type` for strain or discrete-state categories. Selecting a cell shows its stable ID, lineage parent, geometry, type, growth rate, and ordered species values. The browser owns only presentation state. Python owns the clock, model, backend, checkpoint path, and random state. ## Resume exactly -Controller-backed checkpoints must be resumed with the same model source, seed, and parameters. MicroSimulator verifies the source digest before running the file: +Controller-backed checkpoints require the same model source bytes. The CLI restores the saved seed and parameters automatically and verifies the source digest before running the file. Do not pass new `--parameter` values; they are rejected. Omit `--seed` because it does not override the saved seed during resume. The step count below is additional, and the timestep is retained explicitly: ```console uv run microsimulator run \ --model examples/microfluidic_trap.py \ --resume results/tutorial-trap.json \ --backend cpu \ - --seed 42 \ --steps 100 \ --dt 0.02 \ --output results/trap-resumed.json diff --git a/docs/tutorials/intracellular-dynamics.md b/docs/tutorials/intracellular-dynamics.md index 1bcb155..d803b11 100644 --- a/docs/tutorials/intracellular-dynamics.md +++ b/docs/tutorials/intracellular-dynamics.md @@ -2,6 +2,8 @@ This tutorial introduces intracellular concentrations, growth dilution, typed rate equations, gene-circuit feedback, and quantitative time-course analysis. The runnable scenarios are collected in `examples/tutorials/gene_expression.py`. +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + All five gene-expression scenarios request a founder centerline length of 3.5 and cap it at the one sampled division target. Initial concentrations are unchanged; the smaller biomass can change total initial amount. See [founder initialization and volume conventions](biophysics-and-growth.md#length-and-volume). ## The native species contract @@ -24,6 +26,7 @@ Equal division copies concentrations to both daughters. Since their effective vo ```console uv run microsimulator view \ --model examples/tutorials/gene_expression.py \ + --backend cpu \ --parameter scenario='"constitutive"' \ --seed 42 \ --dt 0.01 \ @@ -54,6 +57,7 @@ The `legacy_constitutive` scenario provides an alternative parameterization with ```console uv run microsimulator view \ --model examples/tutorials/gene_expression.py \ + --backend cpu \ --parameter scenario='"dilution"' \ --seed 42 \ --dt 0.01 \ @@ -67,6 +71,7 @@ The founder starts at `x = 10` and the explicit chemical rate is zero. Any decli ```console uv run microsimulator view \ --model examples/tutorials/gene_expression.py \ + --backend cpu \ --parameter scenario='"derepression"' \ --seed 42 \ --dt 0.01 \ @@ -87,6 +92,7 @@ As growth dilutes `x0`, reporter production approaches one. Inspect both channel ```console uv run microsimulator view \ --model examples/tutorials/gene_expression.py \ + --backend cpu \ --parameter scenario='"oscillator"' \ --seed 42 \ --dt 0.005 \ @@ -111,6 +117,7 @@ Create periodic checkpoints, export them, and plot or inspect one stable cell li ```console uv run microsimulator run \ --model examples/tutorials/gene_expression.py \ + --backend cpu \ --parameter scenario='"oscillator"' \ --seed 42 \ --steps 400 \ diff --git a/docs/tutorials/microfluidics.md b/docs/tutorials/microfluidics.md index 36d68b9..4662549 100644 --- a/docs/tutorials/microfluidics.md +++ b/docs/tutorials/microfluidics.md @@ -2,6 +2,8 @@ This tutorial connects device geometry, flowing media, and cell biology in runnable MicroSimulator models. The [modeling guide](../microfluidics.md) introduces the workflow and the choice of flow solver. Four examples cover the range: +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + The microfluidic-trap, Danino, biopixel, and pillar tutorial founders request centerline length 3.5, capped at their single sampled target in [3.2, 3.8]. Attachment, position, radius, and concentrations are preserved. This affects new construction only; saved geometry is restored unchanged. See [founder initialization and volume conventions](biophysics-and-growth.md#length-and-volume). | Model | Device | Demonstrates | @@ -14,7 +16,7 @@ The microfluidic-trap, Danino, biopixel, and pillar tutorial founders request ce Run any of them live: ```console -uv run microsimulator view --model examples/microfluidic_trap.py --seed 42 --dt 0.02 --backend metal --open +uv run microsimulator view --model examples/microfluidic_trap.py --seed 42 --dt 0.02 --backend cpu --open ``` ## Walls that cells and chemistry both respect @@ -143,7 +145,7 @@ With `include_blocks=True`, the reader also exposes geometry in unplaced block d The executable example loads and checks this layout, then simulates one cavity using the independently published dimensions. That single-trap reduction assumes one selected local inlet condition; it does not assert uniform flow across the array, reproduce the array manifold, or include inter-trap coupling. Run it live: ```console -uv run microsimulator view --model examples/tutorials/biopixel_trap.py --seed 5 --dt 0.02 --backend metal --open +uv run microsimulator view --model examples/tutorials/biopixel_trap.py --seed 5 --dt 0.02 --backend cpu --open ``` ## Units and timescales diff --git a/docs/tutorials/nutrient-validation.md b/docs/tutorials/nutrient-validation.md index bc0b197..838bcc5 100644 --- a/docs/tutorials/nutrient-validation.md +++ b/docs/tutorials/nutrient-validation.md @@ -2,6 +2,8 @@ Run the controlled numerical study with: +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + ```console uv run python scripts/run_nutrient_benchmarks.py --backend cpu --output build/nutrient-cpu.json uv run python scripts/run_nutrient_benchmarks.py --backend metal --output build/nutrient-metal.json diff --git a/docs/tutorials/signaling.md b/docs/tutorials/signaling.md index e617be3..e9ec3d5 100644 --- a/docs/tutorials/signaling.md +++ b/docs/tutorials/signaling.md @@ -2,6 +2,8 @@ This tutorial introduces extracellular grids, diffusion, cell-grid exchange, sender-receiver communication, and two-strain mutualism. Run the scenarios in `examples/tutorials/signaling.py` with a small time step such as `0.01`. +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + Each new founder requests centerline length 3.5 and is capped at its sampled division threshold. Its radius, position, cell type, and initial concentrations are preserved. See [founder initialization](biophysics-and-growth.md#length-and-volume). ## Grid geometry and units @@ -30,6 +32,7 @@ Both coefficient arrays must be finite and non-negative. A relaxation toward a n ```console uv run microsimulator view \ --model examples/tutorials/signaling.py \ + --backend cpu \ --parameter scenario='"single_gene"' \ --seed 42 \ --dt 0.01 \ @@ -58,6 +61,7 @@ Two inward-facing planes at `y = -16` and `y = 16` confine cells. The signal gri ```console uv run microsimulator view \ --model examples/tutorials/signaling.py \ + --backend cpu \ --parameter scenario='"communication"' \ --seed 42 \ --dt 0.01 \ @@ -86,6 +90,7 @@ Use cell-type coloring to identify sender and receiver lineages, species channel ```console uv run microsimulator view \ --model examples/tutorials/signaling.py \ + --backend cpu \ --parameter scenario='"mutualism"' \ --seed 42 \ --dt 0.01 \ diff --git a/docs/tutorials/simbol.md b/docs/tutorials/simbol.md index 9b3908d..503842b 100644 --- a/docs/tutorials/simbol.md +++ b/docs/tutorials/simbol.md @@ -2,6 +2,8 @@ SimBOL connects an SBOL 3 design to simulator-specific code through a summarized JSON representation. This tutorial presents typed MicroSimulator versions of six BioBrick circuit examples and a spatial quorum-sensing clock. +For backend selection, PowerShell syntax, quoted JSON parameters, and paths with spaces, see [tutorial commands by backend and shell](commands.md#choose-a-shell). Multiline commands on this page use POSIX shell backslashes; the guide provides the PowerShell equivalents and [explicit CPU, Metal, and CUDA trap launches](commands.md#run-the-same-trap-on-cpu-metal-or-cuda). + These are explicit example models, not a general SBOL-to-rate-plan import path. The [source reference](../compatibility/tutorial-source-provenance.md#simbol-source-workflow) describes how they relate to the SimBOL notebook, generated Python, and JSON fixtures. ## Run the six circuits @@ -11,6 +13,7 @@ Use one model and select a circuit: ```console uv run microsimulator view \ --model examples/tutorials/simbol_circuits.py \ + --backend cpu \ --parameter circuit='"bba_0001"' \ --seed 42 \ --dt 0.01 \ @@ -33,6 +36,7 @@ Parameters are JSON numbers: ```console uv run microsimulator run \ --model examples/tutorials/simbol_circuits.py \ + --backend cpu \ --parameter circuit='"bba_0004"' \ --parameter inducer_concentration=4.0 \ --seed 42 \ @@ -115,6 +119,7 @@ These choices change trajectories relative to the generated callback scripts. A ```console uv run microsimulator view \ --model examples/tutorials/danino_clock.py \ + --backend cpu \ --seed 42 \ --dt 0.005 \ --open diff --git a/environments/cuda/README.md b/environments/cuda/README.md index cb16efe..603dbb5 100644 --- a/environments/cuda/README.md +++ b/environments/cuda/README.md @@ -2,6 +2,8 @@ CUDA is the NVIDIA backend under active development. It is implemented directly in CUDA C++ with the CUDA Runtime API: no portability layer, translated Metal source, or CPU computational fallback is used. +For application commands, use the [shared CPU, Metal, and CUDA tutorial examples](../../docs/tutorials/commands.md#run-the-same-trap-on-cpu-metal-or-cuda), [device discovery](../../docs/tutorials/commands.md#prepare-and-discover-devices), and [shell quoting guide](../../docs/tutorials/commands.md#choose-a-shell). Select this backend with `--backend cuda` and an enumerated `--device-index`; the model syntax is unchanged. + `CM_ENABLE_CUDA` is off by default so ordinary CPU builds do not acquire a CUDA toolchain dependency. ## Compile check diff --git a/environments/metal/README.md b/environments/metal/README.md index 53500ea..d5214f7 100644 --- a/environments/metal/README.md +++ b/environments/metal/README.md @@ -2,6 +2,8 @@ Metal is the feature-complete Apple GPU backend. It is implemented directly with the Metal API and independent Metal Shading Language kernels, and it is validated against both the CPU reference and recorded behavior from the original CellModeller OpenCL runtime. +For application commands, use the [shared CPU, Metal, and CUDA tutorial examples](../../docs/tutorials/commands.md#run-the-same-trap-on-cpu-metal-or-cuda), [device discovery](../../docs/tutorials/commands.md#prepare-and-discover-devices), and [shell quoting guide](../../docs/tutorials/commands.md#choose-a-shell). Select this backend with `--backend metal` and an enumerated `--device-index`; the model syntax is unchanged. + Metal is enabled by default on Apple platforms. It compiles embedded MSL source at runtime through `MTLDevice`, making kernel compilation part of device construction and validation. ## Native conformance diff --git a/python/tests/test_tutorial_commands.py b/python/tests/test_tutorial_commands.py new file mode 100644 index 0000000..0d669c9 --- /dev/null +++ b/python/tests/test_tutorial_commands.py @@ -0,0 +1,321 @@ +"""Execute the published CLI examples through real shells, without rewriting them.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import math +import os +import platform +import re +import shlex +import shutil +import subprocess +import sys +from collections.abc import Iterator +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, cast +from urllib.parse import urlsplit + +import pytest +from aiohttp import ClientSession, WSMsgType +from microsimulator import load_checkpoint_bundle +from microsimulator.cli import _parser # pyright: ignore[reportPrivateUsage] + +ROOT = Path(__file__).resolve().parents[2] +GUIDE = ROOT / "docs/tutorials/commands.md" +BLOCKS = dict( + re.findall( + r"\s*```\w+\n(.*?)\n```", + GUIDE.read_text(encoding="utf-8"), + re.DOTALL, + ) +) +SHELLS = ("pwsh",) if sys.platform == "win32" else ("sh", "bash", "zsh", "pwsh") + + +@dataclass +class CommandShell: + name: str + executable: str + cwd: Path + records: list[dict[str, Any]] = field(default_factory=lambda: list[dict[str, Any]]()) + + def prepare(self, command: str) -> tuple[list[str], dict[str, str]]: + environment = dict(os.environ) + environment.pop("VIRTUAL_ENV", None) + environment["UV_PROJECT_ENVIRONMENT"] = sys.prefix + environment["PYTHONUNBUFFERED"] = "1" + environment["UV_OFFLINE"] = "1" + if self.name == "pwsh": + script = self.cwd / "command.ps1" + script.write_text( + "$ErrorActionPreference = 'Stop'\n" + "$global:LASTEXITCODE = 0\n" + + BLOCKS["powershell-mode"] + + "\n" + + command + + "\nexit $LASTEXITCODE\n", + encoding="utf-8", + ) + arguments = [self.executable, "-NoLogo", "-NoProfile", "-File", str(script)] + else: + script = self.cwd / "command.sh" + script.write_text("set -e\n" + command + "\n", encoding="utf-8") + arguments = [self.executable, str(script)] + return arguments, environment + + def run(self, identifier: str, *, success: bool = True) -> subprocess.CompletedProcess[str]: + command = BLOCKS[identifier] + arguments, environment = self.prepare(command) + result = subprocess.run( + arguments, + cwd=self.cwd, + env=environment, + text=True, + capture_output=True, + timeout=120, + check=False, + ) + self.records.append( + { + "id": identifier, + "command": command, + "exit_code": result.returncode, + "stdout": result.stdout, + "stderr": result.stderr, + } + ) + assert (result.returncode == 0) is success, result.stdout + result.stderr + return result + + +@pytest.fixture(params=SHELLS) +def shell(request: pytest.FixtureRequest, tmp_path: Path) -> Iterator[CommandShell]: + name = cast(str, request.param) + executable = shutil.which(name) + if executable is None: + pytest.skip(f"{name} is not installed; no {name} coverage claimed") + cwd = tmp_path / "repository with spaces" + cwd.mkdir() + shutil.copytree(ROOT / "examples", cwd / "examples") + (cwd / "pyproject.toml").write_text( + '[project]\nname="tutorial-command-test"\nversion="0.0.0"\n' + 'requires-python=">=3.12,<3.13"\n', + encoding="utf-8", + ) + # CLI lifecycle checks deliberately do not claim to test the browser bundle. + (cwd / "viewer/dist/assets").mkdir(parents=True) + (cwd / "viewer/dist/index.html").write_text("CLI test") + instance = CommandShell(name, executable, cwd) + version_args = ( + ["-NoLogo", "-NoProfile", "-Command", "$PSVersionTable.PSVersion.ToString()"] + if name == "pwsh" + else ["--version"] if name != "sh" else ["-c", "echo POSIX-sh"] + ) + version = subprocess.run( + [executable, *version_args], capture_output=True, text=True, timeout=60, check=True + ).stdout.strip() + if name == "pwsh": + assert tuple(int(part) for part in version.split(".")[:2]) >= (7, 3) + try: + yield instance + finally: + if destination := os.environ.get("MICROSIMULATOR_COMMAND_REPORT"): + directory = Path(destination) + directory.mkdir(parents=True, exist_ok=True) + node = cast(Any, request).node # pytest's request.node is untyped. + (directory / f"{node.name}.json").write_text( + json.dumps( + { + "platform": platform.platform(), + "python": sys.version, + "shell": name, + "shell_version": version, + "cwd_contains_spaces": True, + "commands": instance.records, + }, + indent=2, + ), + encoding="utf-8", + ) + + +def _document(path: Path) -> dict[str, Any]: + # Authenticate checkpoints as well as inspecting their human-readable fields. + load_checkpoint_bundle(path) + return cast(dict[str, Any], json.loads(path.read_text(encoding="utf-8"))) + + +def test_documented_headless_commands(shell: CommandShell) -> None: + shell.run("devices") + devices = cast(list[dict[str, Any]], json.loads(shell.run("devices-json").stdout)) + assert {record["backend"] for record in devices} == {"cpu", "metal", "cuda"} + for record in devices: + backend = record["backend"] + result = shell.run(f"trap-{backend}", success=record["available"]) + output = shell.cwd / f"results/tutorial runs/trap-{backend}.json" + if not record["available"]: + assert f"backend {backend} device 0 is unavailable" in result.stderr + assert not output.exists() + continue + document = _document(output) + assert document["source_backend"]["kind"] == backend + assert document["provenance"]["model"]["seed"] == 42 + assert document["provenance"]["model"]["parameters"] == {} + assert document["provenance"]["run"]["completed_steps"] == 100 + assert math.isclose(document["simulation"]["time"], 2.0) + assert len(list(output.parent.glob(f"trap-{backend}.step-*.json"))) == 5 + shell.run("resume-trap") + assert math.isclose( + _document(shell.cwd / "results/tutorial runs/trap-cpu-resumed.json")["simulation"]["time"], + 4.0, + ) + + suffix = "powershell" if shell.name == "pwsh" else "posix" + shell.run(f"copy-{suffix}") + shell.run(f"basics-{suffix}") + shell.run("resume-basics") + initial = _document(shell.cwd / "results/tutorial runs/basics.json") + resumed = _document(shell.cwd / "results/tutorial runs/basics-resumed.json") + for document in (initial, resumed): + provenance = document["provenance"]["model"] + assert provenance["seed"] == 42 + assert provenance["parameters"] == {"scenario": "basics"} + assert provenance["sha256"] == hashlib.sha256( + (shell.cwd / "results/tutorial models/biophysics.py").read_bytes() + ).hexdigest() + assert math.isclose(initial["simulation"]["time"], 0.2) + assert math.isclose(resumed["simulation"]["time"], 0.4) + assert resumed["provenance"]["resume"]["sha256"] == hashlib.sha256( + (shell.cwd / "results/tutorial runs/basics.json").read_bytes() + ).hexdigest() + + # Independently establish the claimed continuation semantics, using the same + # installed CLI for a 20-step uninterrupted run. + subprocess.run( + [ + sys.executable, "-m", "microsimulator", "run", "--model", + str(shell.cwd / "results/tutorial models/biophysics.py"), + "--parameter", 'scenario="basics"', "--seed", "42", "--steps", "20", + "--dt", "0.02", "--output", str(shell.cwd / "uninterrupted.json"), + ], + check=True, capture_output=True, text=True, timeout=60, + ) + uninterrupted = _document(shell.cwd / "uninterrupted.json") + assert resumed["simulation"] == uninterrupted["simulation"] + assert resumed["controller"] == uninterrupted["controller"] + + # The documented guard rejects overrides and edited source before executing + # it; neither failed resume should create an output or run injected code. + arguments, environment = shell.prepare(BLOCKS["resume-basics"] + " --parameter 'scenario=1'") + rejected = subprocess.run( + arguments, env=environment, cwd=shell.cwd, capture_output=True, text=True, + timeout=60, check=False, + ) + assert rejected.returncode != 0 + assert "do not pass --parameter" in rejected.stderr + model = shell.cwd / "results/tutorial models/biophysics.py" + model.write_text("raise RuntimeError('model executed before digest check')\n", encoding="utf-8") + rejected = shell.run("resume-basics", success=False) + assert "model digest does not match checkpoint" in rejected.stderr + assert "model executed before digest check" not in rejected.stderr + + +def test_documented_live_commands(shell: CommandShell) -> None: + suffix = "powershell" if shell.name == "pwsh" else "posix" + shell.run(f"copy-{suffix}") + shell.run(f"basics-{suffix}") + + async def exercise() -> None: + for identifier in ("live-trap", "live-basics", "live-resume"): + arguments, environment = shell.prepare(BLOCKS[identifier]) + process = await asyncio.create_subprocess_exec( + *arguments, cwd=shell.cwd, env=environment, + stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, + ) + stopped = False + try: + assert process.stdout is not None + line = (await asyncio.wait_for(process.stdout.readline(), 60)).decode().strip() + assert line.startswith("MicroSimulator live viewer: "), line + url = urlsplit(line.split(": ", 1)[1]) + assert url.port == 8765 + async with ClientSession() as client, client.ws_connect( + f"http://{url.netloc}/api/v1/session?{url.query}", + headers={"Origin": f"http://{url.netloc}"}, + ) as ws: + frame = await ws.receive_json(timeout=10) + assert frame["playing"] is False + assert math.isclose( + frame["scene"]["frame"]["time"], + 0.2 if identifier == "live-resume" else 0.0, + ) + if identifier == "live-trap": + await ws.send_json({"type": "checkpoint"}) + saved = await ws.receive_json(timeout=10) + assert saved["type"] == "checkpoint" + _document(shell.cwd / "results/tutorial runs/live-trap.json") + await ws.send_json({"type": "stop"}) + while True: + message = await ws.receive(timeout=15) + if message.type == WSMsgType.CLOSE: + break + if message.type == WSMsgType.TEXT: + stopped |= message.json() == {"type": "session", "state": "stopped"} + stdout, stderr = await asyncio.wait_for(process.communicate(), 15) + shell.records.append( + {"id": identifier, "command": BLOCKS[identifier], + "exit_code": process.returncode, "stopped": stopped, + "stdout": stdout.decode(), "stderr": stderr.decode()} + ) + assert stopped + assert process.returncode == 0, stderr.decode() + finally: + if process.returncode is None: + process.kill() + await process.wait() + + asyncio.run(exercise()) + + +def test_tutorial_flags_paths_and_links() -> None: + documents = [ + *sorted((ROOT / "docs/tutorials").glob("*.md")), + ROOT / "environments/metal/README.md", ROOT / "environments/cuda/README.md", + ROOT / "viewer/README.md", ROOT / "docs/development/tutorial-command-validation.md", + ] + parser = _parser() + checked = 0 + for path in documents: + text = path.read_text(encoding="utf-8") + for target in re.findall(r"\[[^\]]+\]\(([^)]+)\)", text): + if re.match(r"[a-z]+://", target): + continue + filename, _, anchor = target.partition("#") + destination = (path.parent / filename).resolve() if filename else path + assert destination.exists(), (path, target) + if anchor and destination.suffix == ".md": + headings = re.findall(r"^#+\s+(.+)$", destination.read_text(), re.MULTILINE) + anchors = {re.sub(r"[^\w -]", "", h.lower()).replace(" ", "-") for h in headings} + assert anchor in anchors, (path, target) + if path.parent == ROOT / "docs/tutorials" and path != GUIDE: + assert "commands.md#" in text, path + for language, block in re.findall(r"```(console|sh)\n(.*?)\n```", text, re.DOTALL): + del language + for line in block.replace("\\\n", " ").splitlines(): + parts = shlex.split(line) + if "microsimulator" not in parts or "--help" in parts: + continue + # Exclude prose or output; only parse literal CLI invocations. + if parts[:2] != ["uv", "run"]: + continue + arguments = parser.parse_args(parts[parts.index("microsimulator") + 1:]) + if (model := getattr(arguments, "model", None)) is not None: + model = cast(Path, model) + if model != Path("results/tutorial models/biophysics.py"): + assert (ROOT / model).is_file(), (path, model) + checked += 1 + assert checked >= 35 diff --git a/viewer/README.md b/viewer/README.md index 2b540ac..41ff93a 100644 --- a/viewer/README.md +++ b/viewer/README.md @@ -2,6 +2,8 @@ The MicroSimulator viewer displays cells, device walls, and signal fields so you can inspect a population in its microfluidic environment. Use it to explore saved scenes or follow a live simulation with growth-rate coloring, nutrient slices, and individual-cell inspection. +See [tutorial commands by backend and shell](../docs/tutorials/commands.md#live-view-stop-and-restart) for copyable CPU/Metal/CUDA selection, PowerShell quoting, paths with spaces, checkpoint resume, and Stop/restart. Multiline commands below use POSIX shell backslashes. + The viewer is a TypeScript and Three.js client for `microsimulator-scene` documents. Standalone mode reads scene files; live mode sends typed controls to a Python-owned engine session and verifies every returned scene document. Python owns the model, simulation clock, backend, and checkpoint writer. ## Run locally @@ -45,7 +47,7 @@ For example, after stopping the trap model above, launch a different model on th uv run microsimulator view --model examples/tutorials/biophysics.py --backend cpu --seed 42 --dt 0.02 --port 8765 --open ``` -The command is a single line and also works in PowerShell where the Python/native build is available. To distinguish Windows console behavior from browser behavior, use this manual verification procedure in an attached PowerShell or Command Prompt console: +The command is a single line and also works in PowerShell 7.3+ where the Python/native build is available; configure [Standard argument passing](../docs/tutorials/commands.md#choose-a-shell) for JSON-valued parameters. To distinguish Windows console behavior from browser behavior, use this manual verification procedure in an attached PowerShell or Command Prompt console: 1. Record the Windows version, terminal application/version, Python version, and exact launch command. Start the command above and click Stop while paused. Confirm the prompt returns, then start the second model on port 8765. 2. Repeat with Play active and `--frame-steps 10000`. Confirm Stopping transitions to Stopped without finishing the entire batch. @@ -54,7 +56,7 @@ The command is a single line and also works in PowerShell where the Python/nativ The automated `python/tests/test_viewer_shutdown.py` suite covers same-socket Stop, checkpoint completion, worker cleanup, and repeated real subprocess restarts. It sends SIGINT on POSIX. On Windows it starts each viewer in an isolated console and uses a separate attached sender to deliver a real [Windows CTRL_C_EVENT](https://learn.microsoft.com/en-us/windows/console/generateconsolectrlevent), leaving the test runner unaffected. Both paths verify orderly browser notifications, clean process exit, and three different models reusing the same port. This exercises the operating-system interruption path; use the manual procedure above to check a particular interactive terminal application and keyboard configuration. -The `Windows live-session shutdown` GitHub Actions job builds the CPU extension on `windows-2025` and runs the server and shutdown tests, including isolated-console Ctrl+C, with dependencies from `uv.lock`. Its uploaded report records Windows, PowerShell, Python, backend availability, and individual test results. +The `Windows CLI and live-session checks` GitHub Actions job builds the CPU extension on `windows-2025` and runs the server and shutdown tests, including isolated-console Ctrl+C, with dependencies from `uv.lock`. Its uploaded report records Windows, PowerShell, Python, backend availability, and individual test results. ## Capabilities From 215e87dbf58c9964aef5a7821cdb3cb494c91490 Mon Sep 17 00:00:00 2001 From: Mike Arpaia Date: Thu, 24 Sep 2026 16:41:26 -0600 Subject: [PATCH 2/4] Keep command checks float32-aware and bound failed process cleanup --- .../tutorial-command-validation.md | 4 +- python/tests/test_tutorial_commands.py | 185 ++++++++++++++---- 2 files changed, 146 insertions(+), 43 deletions(-) diff --git a/docs/development/tutorial-command-validation.md b/docs/development/tutorial-command-validation.md index c394902..21fa597 100644 --- a/docs/development/tutorial-command-validation.md +++ b/docs/development/tutorial-command-validation.md @@ -4,7 +4,7 @@ The [command guide](../tutorials/commands.md) separates shell argument handling, ## Coverage and limits -The command tests exercise human-readable and JSON device discovery; the identical 100-step trap command on every available backend; explicit unavailable-backend failures; JSON string arguments and line continuation; repository, model, checkpoint, and output paths containing spaces; restored seed/parameters/source digest; and uninterrupted versus resumed controller/native state. They launch the documented trap, growth, and resumed live sessions in sequence on port 8765, save a checkpoint, send authenticated Stop, observe Stopped, and verify each process returns successfully before the next launch. The CLI tests supply minimal static assets and do not claim browser rendering coverage. +The command tests exercise human-readable and JSON device discovery; the identical 100-step trap command on every available backend; explicit unavailable-backend failures; JSON string arguments and line continuation; repository, model, checkpoint, and output paths containing spaces; restored seed/parameters/source digest; and uninterrupted versus resumed controller/native state. They launch the documented trap, growth, and resumed live sessions in sequence on port 8765, save a checkpoint, send authenticated Stop, observe Stopped, and verify each process returns successfully before the next launch. A cleanup regression also verifies that a failed test terminates its own subprocess tree and drains inherited pipes. The CLI tests supply minimal static assets and do not claim browser rendering coverage. PowerShell tests set `$PSNativeCommandArgumentPassing = 'Standard'` exactly as documented. Windows uses `pwsh` 7.3+; Windows PowerShell 5.1 and `cmd.exe` are not covered. POSIX examples are executed in `sh`, Bash, and Zsh when those shells are installed. A missing shell is explicitly skipped. Reports include the exact script text, exit status, stdout/stderr, platform, Python version, shell version, and enumerated devices. @@ -12,7 +12,7 @@ The [Windows CLI and live-session workflow](../../.github/workflows/live-shutdow ## Execution record -The local command run on 2026-09-24 passed all nine tests with Python 3.12.8, Bash 3.2.57, Zsh 5.9, and PowerShell 7.6.4. The associated pull request links the exact source SHA and hosted workflow run; reports are retained as CI artifacts. +The local command run on 2026-09-24 passed all ten tests with Python 3.12.8, Bash 3.2.57, Zsh 5.9, and PowerShell 7.6.4. The associated pull request links the exact source SHA and hosted workflow run; reports are retained as CI artifacts. | Platform and shell | Backend execution | Command coverage | | --- | --- | --- | diff --git a/python/tests/test_tutorial_commands.py b/python/tests/test_tutorial_commands.py index 0d669c9..7430093 100644 --- a/python/tests/test_tutorial_commands.py +++ b/python/tests/test_tutorial_commands.py @@ -11,9 +11,11 @@ import re import shlex import shutil +import signal import subprocess import sys from collections.abc import Iterator +from contextlib import suppress from dataclasses import dataclass, field from pathlib import Path from typing import Any, cast @@ -36,6 +38,49 @@ SHELLS = ("pwsh",) if sys.platform == "win32" else ("sh", "bash", "zsh", "pwsh") +def _same_time(actual: float, expected: float) -> bool: + # dt is stored in native float32; retain pytest.approx's intended tolerance. + return math.isclose(actual, expected, rel_tol=1e-6, abs_tol=1e-8) + + +def _kill_tree(pid: int) -> None: + # Only called for a subprocess created by this test. Killing just its shell + # leaves uv/the live server holding the inherited stdout pipe and port. + if sys.platform == "win32": + subprocess.run( + ["taskkill", "/PID", str(pid), "/T", "/F"], + capture_output=True, + check=False, + timeout=10, + ) + else: + with suppress(ProcessLookupError): + os.killpg(pid, signal.SIGKILL) + + +def _run_script( + arguments: list[str], + environment: dict[str, str], + cwd: Path, +) -> subprocess.CompletedProcess[str]: + with subprocess.Popen( + arguments, + cwd=cwd, + env=environment, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + start_new_session=sys.platform != "win32", + ) as process: + try: + stdout, stderr = process.communicate(timeout=120) + except subprocess.TimeoutExpired as error: + _kill_tree(process.pid) + stdout, stderr = process.communicate(timeout=10) + raise AssertionError(f"command timed out: {stdout} {stderr}") from error + return subprocess.CompletedProcess(arguments, process.returncode, stdout, stderr) + + @dataclass class CommandShell: name: str @@ -70,15 +115,8 @@ def prepare(self, command: str) -> tuple[list[str], dict[str, str]]: def run(self, identifier: str, *, success: bool = True) -> subprocess.CompletedProcess[str]: command = BLOCKS[identifier] arguments, environment = self.prepare(command) - result = subprocess.run( - arguments, - cwd=self.cwd, - env=environment, - text=True, - capture_output=True, - timeout=120, - check=False, - ) + print(f"Executing {self.name}: {identifier}", flush=True) + result = _run_script(arguments, environment, self.cwd) self.records.append( { "id": identifier, @@ -113,7 +151,9 @@ def shell(request: pytest.FixtureRequest, tmp_path: Path) -> Iterator[CommandShe version_args = ( ["-NoLogo", "-NoProfile", "-Command", "$PSVersionTable.PSVersion.ToString()"] if name == "pwsh" - else ["--version"] if name != "sh" else ["-c", "echo POSIX-sh"] + else ["--version"] + if name != "sh" + else ["-c", "echo POSIX-sh"] ) version = subprocess.run( [executable, *version_args], capture_output=True, text=True, timeout=60, check=True @@ -166,10 +206,10 @@ def test_documented_headless_commands(shell: CommandShell) -> None: assert document["provenance"]["model"]["seed"] == 42 assert document["provenance"]["model"]["parameters"] == {} assert document["provenance"]["run"]["completed_steps"] == 100 - assert math.isclose(document["simulation"]["time"], 2.0) + assert _same_time(document["simulation"]["time"], 2.0) assert len(list(output.parent.glob(f"trap-{backend}.step-*.json"))) == 5 shell.run("resume-trap") - assert math.isclose( + assert _same_time( _document(shell.cwd / "results/tutorial runs/trap-cpu-resumed.json")["simulation"]["time"], 4.0, ) @@ -184,25 +224,46 @@ def test_documented_headless_commands(shell: CommandShell) -> None: provenance = document["provenance"]["model"] assert provenance["seed"] == 42 assert provenance["parameters"] == {"scenario": "basics"} - assert provenance["sha256"] == hashlib.sha256( - (shell.cwd / "results/tutorial models/biophysics.py").read_bytes() + assert ( + provenance["sha256"] + == hashlib.sha256( + (shell.cwd / "results/tutorial models/biophysics.py").read_bytes() + ).hexdigest() + ) + assert _same_time(initial["simulation"]["time"], 0.2) + assert _same_time(resumed["simulation"]["time"], 0.4) + assert ( + resumed["provenance"]["resume"]["sha256"] + == hashlib.sha256( + (shell.cwd / "results/tutorial runs/basics.json").read_bytes() ).hexdigest() - assert math.isclose(initial["simulation"]["time"], 0.2) - assert math.isclose(resumed["simulation"]["time"], 0.4) - assert resumed["provenance"]["resume"]["sha256"] == hashlib.sha256( - (shell.cwd / "results/tutorial runs/basics.json").read_bytes() - ).hexdigest() + ) # Independently establish the claimed continuation semantics, using the same # installed CLI for a 20-step uninterrupted run. subprocess.run( [ - sys.executable, "-m", "microsimulator", "run", "--model", + sys.executable, + "-m", + "microsimulator", + "run", + "--model", str(shell.cwd / "results/tutorial models/biophysics.py"), - "--parameter", 'scenario="basics"', "--seed", "42", "--steps", "20", - "--dt", "0.02", "--output", str(shell.cwd / "uninterrupted.json"), + "--parameter", + 'scenario="basics"', + "--seed", + "42", + "--steps", + "20", + "--dt", + "0.02", + "--output", + str(shell.cwd / "uninterrupted.json"), ], - check=True, capture_output=True, text=True, timeout=60, + check=True, + capture_output=True, + text=True, + timeout=60, ) uninterrupted = _document(shell.cwd / "uninterrupted.json") assert resumed["simulation"] == uninterrupted["simulation"] @@ -212,8 +273,13 @@ def test_documented_headless_commands(shell: CommandShell) -> None: # it; neither failed resume should create an output or run injected code. arguments, environment = shell.prepare(BLOCKS["resume-basics"] + " --parameter 'scenario=1'") rejected = subprocess.run( - arguments, env=environment, cwd=shell.cwd, capture_output=True, text=True, - timeout=60, check=False, + arguments, + env=environment, + cwd=shell.cwd, + capture_output=True, + text=True, + timeout=60, + check=False, ) assert rejected.returncode != 0 assert "do not pass --parameter" in rejected.stderr @@ -233,8 +299,12 @@ async def exercise() -> None: for identifier in ("live-trap", "live-basics", "live-resume"): arguments, environment = shell.prepare(BLOCKS[identifier]) process = await asyncio.create_subprocess_exec( - *arguments, cwd=shell.cwd, env=environment, - stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, + *arguments, + cwd=shell.cwd, + env=environment, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + start_new_session=sys.platform != "win32", ) stopped = False try: @@ -243,13 +313,16 @@ async def exercise() -> None: assert line.startswith("MicroSimulator live viewer: "), line url = urlsplit(line.split(": ", 1)[1]) assert url.port == 8765 - async with ClientSession() as client, client.ws_connect( - f"http://{url.netloc}/api/v1/session?{url.query}", - headers={"Origin": f"http://{url.netloc}"}, - ) as ws: + async with ( + ClientSession() as client, + client.ws_connect( + f"http://{url.netloc}/api/v1/session?{url.query}", + headers={"Origin": f"http://{url.netloc}"}, + ) as ws, + ): frame = await ws.receive_json(timeout=10) assert frame["playing"] is False - assert math.isclose( + assert _same_time( frame["scene"]["frame"]["time"], 0.2 if identifier == "live-resume" else 0.0, ) @@ -261,22 +334,27 @@ async def exercise() -> None: await ws.send_json({"type": "stop"}) while True: message = await ws.receive(timeout=15) - if message.type == WSMsgType.CLOSE: + if message.type in {WSMsgType.CLOSE, WSMsgType.CLOSED, WSMsgType.ERROR}: break if message.type == WSMsgType.TEXT: stopped |= message.json() == {"type": "session", "state": "stopped"} stdout, stderr = await asyncio.wait_for(process.communicate(), 15) shell.records.append( - {"id": identifier, "command": BLOCKS[identifier], - "exit_code": process.returncode, "stopped": stopped, - "stdout": stdout.decode(), "stderr": stderr.decode()} + { + "id": identifier, + "command": BLOCKS[identifier], + "exit_code": process.returncode, + "stopped": stopped, + "stdout": stdout.decode(), + "stderr": stderr.decode(), + } ) assert stopped assert process.returncode == 0, stderr.decode() finally: if process.returncode is None: - process.kill() - await process.wait() + await asyncio.to_thread(_kill_tree, process.pid) + await asyncio.wait_for(process.communicate(), 10) asyncio.run(exercise()) @@ -284,8 +362,10 @@ async def exercise() -> None: def test_tutorial_flags_paths_and_links() -> None: documents = [ *sorted((ROOT / "docs/tutorials").glob("*.md")), - ROOT / "environments/metal/README.md", ROOT / "environments/cuda/README.md", - ROOT / "viewer/README.md", ROOT / "docs/development/tutorial-command-validation.md", + ROOT / "environments/metal/README.md", + ROOT / "environments/cuda/README.md", + ROOT / "viewer/README.md", + ROOT / "docs/development/tutorial-command-validation.md", ] parser = _parser() checked = 0 @@ -312,10 +392,33 @@ def test_tutorial_flags_paths_and_links() -> None: # Exclude prose or output; only parse literal CLI invocations. if parts[:2] != ["uv", "run"]: continue - arguments = parser.parse_args(parts[parts.index("microsimulator") + 1:]) + arguments = parser.parse_args(parts[parts.index("microsimulator") + 1 :]) if (model := getattr(arguments, "model", None)) is not None: model = cast(Path, model) if model != Path("results/tutorial models/biophysics.py"): assert (ROOT / model).is_file(), (path, model) checked += 1 assert checked >= 35 + + +def test_failed_command_cleanup_drains_descendant_pipes() -> None: + # Regression for a failed live assertion hanging after only its shell died. + code = ( + "import subprocess, sys, time; " + "subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)']); " + "print('ready', flush=True); time.sleep(60)" + ) + with subprocess.Popen( + [sys.executable, "-c", code], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=sys.platform != "win32", + ) as process: + try: + assert process.stdout is not None + assert process.stdout.readline().strip() == "ready" + finally: + _kill_tree(process.pid) + process.communicate(timeout=10) + assert process.returncode is not None From 5e1fdecb404feb89f82e75cea698f870fb70e37d Mon Sep 17 00:00:00 2001 From: Mike Arpaia Date: Thu, 24 Sep 2026 16:44:58 -0600 Subject: [PATCH 3/4] Record executed Windows PowerShell tutorial verification --- docs/development/tutorial-command-validation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development/tutorial-command-validation.md b/docs/development/tutorial-command-validation.md index 21fa597..737de9e 100644 --- a/docs/development/tutorial-command-validation.md +++ b/docs/development/tutorial-command-validation.md @@ -12,13 +12,13 @@ The [Windows CLI and live-session workflow](../../.github/workflows/live-shutdow ## Execution record -The local command run on 2026-09-24 passed all ten tests with Python 3.12.8, Bash 3.2.57, Zsh 5.9, and PowerShell 7.6.4. The associated pull request links the exact source SHA and hosted workflow run; reports are retained as CI artifacts. +The local command run on 2026-09-24 passed all ten tests with Python 3.12.8, Bash 3.2.57, Zsh 5.9, and PowerShell 7.6.4. Windows verification also passed on 2026-09-24 at commit `215e87dbf58c9964aef5a7821cdb3cb494c91490`: [workflow run 36068841367](https://github.com/DRAGGON-Lab/MicroSimulator/actions/runs/36068841367) recorded four command/cleanup tests passing in 17.934 seconds and all 28 live-session shutdown tests passing. Reports are retained as CI artifacts. | Platform and shell | Backend execution | Command coverage | | --- | --- | --- | | macOS, POSIX `sh`, Bash, Zsh | Native CPU and Apple M4 Max Metal | Passed: discovery, 100-step CPU/Metal trap, JSON/space paths, headless CPU resume, live CPU checkpoint/Stop/restart | | macOS, PowerShell 7.6.4 with Standard arguments | Native CPU and Apple M4 Max Metal | Passed: same commands, PowerShell backtick continuation and string quoting | -| Windows Server 2025, PowerShell 7 with Standard arguments | Native CPU | Hosted command verification pending; see the pull request workflow result | +| Windows Server 2025 build 26100, PowerShell 7.6.6, Python 3.12.10 | Native CPU | Passed: discovery, 100-step trap, PowerShell JSON/space paths, resume, three live Stop/restarts, cleanup regression; Metal/CUDA unavailable as expected | | NVIDIA CUDA hardware | Not available in the command-verification campaign | Unavailable-backend error checked; no CUDA runtime or numerical claim | The test runner reports actual Metal availability for each macOS run. This run executed Metal headless trap commands; its JSON scenario, resume, and live-session examples selected CPU. A skipped/unavailable Metal result does not count as GPU execution. Backend support still requires the independent [native and application conformance gates](validation.md), even when a tutorial smoke command succeeds. From 09663dfb7b1d1024bfcbe01a8932c9ee08a12886 Mon Sep 17 00:00:00 2001 From: Mike Arpaia Date: Thu, 24 Sep 2026 21:02:43 -0600 Subject: [PATCH 4/4] Decode linked Markdown as UTF-8 in Windows command validation --- python/tests/test_tutorial_commands.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/python/tests/test_tutorial_commands.py b/python/tests/test_tutorial_commands.py index 7430093..738df00 100644 --- a/python/tests/test_tutorial_commands.py +++ b/python/tests/test_tutorial_commands.py @@ -378,7 +378,9 @@ def test_tutorial_flags_paths_and_links() -> None: destination = (path.parent / filename).resolve() if filename else path assert destination.exists(), (path, target) if anchor and destination.suffix == ".md": - headings = re.findall(r"^#+\s+(.+)$", destination.read_text(), re.MULTILINE) + headings = re.findall( + r"^#+\s+(.+)$", destination.read_text(encoding="utf-8"), re.MULTILINE + ) anchors = {re.sub(r"[^\w -]", "", h.lower()).replace(" ", "-") for h in headings} assert anchor in anchors, (path, target) if path.parent == ROOT / "docs/tutorials" and path != GUIDE: