bench(cli): measure cold executable startup and verify diagnostics

.agents/skills/polyxml-codegen-workflow/SKILL.md

@@ -332,6 +332,12 @@ derived types inherit their base's patterns at parse time):
 - `benchmarks/cli/benchmark.sh` uses hyperfine with `--shell=none` to avoid shell
   calibration error for sub-5ms startup measurements. Results are fresh processes
   with warm OS caches, not machine-reboot or cold-disk startup measurements.
+- For issue #62, `benchmarks/cli/startup.py` alternates warm `posix_spawn` runs
+  with runs after `POSIX_FADV_DONTNEED` on the CLI executable. Run a release
+  build under `scripts/memcap.sh` first, then check the recorded major-fault
+  counts before calling the second group executable-cache cold. The script
+  does not evict shared libraries or reproduce a post-reboot host; see
+  `benchmarks/cli/README.md` for results and limits.
 
 - Color diagnostics respect `NO_COLOR`. When checking terminal color in a PTY,
   unset `NO_COLOR` in the test subprocess and use a color-capable `TERM`; test

benchmarks/cli/README.md

@@ -25,8 +25,54 @@ A local Linux x86-64 release build measured on 2026-09-22 with hyperfine 1.19.0:
 | `generate-help` | 1.5 ± 0.2 ms |
 | `parse-and-validate` | 1.6 ± 0.2 ms |
 
-These measurements meet issue #50's 5ms target on this host. They use warm OS
-caches: they do **not** establish cold-disk or post-reboot startup latency.
+These measurements use warm OS caches: they do **not** establish cold-disk or
+post-reboot startup latency.
 The parsing case includes tiny-schema I/O and parsing, so it is not an isolated
 argument-parser microbenchmark. Results depend on hardware and system load;
 rerun the script to compare changes on the same host.
+
+## Controlled executable-cache measurement
+
+On Linux, build the release CLI and run:
+
+```bash
+./scripts/memcap.sh cargo build --release -p polyxml-cli
+python3 benchmarks/cli/startup.py --runs 100
+```
+
+`startup.py` times each fresh process from `posix_spawn` through exit. It
+alternates warm runs with runs after `POSIX_FADV_DONTNEED` requests eviction of
+the CLI executable's file pages. It measures top-level help and an invalid
+backend that fails during argument validation, before schema I/O. It records
+individual times and major page faults in `benchmarks/cli/target/startup.json`.
+This is a **cold executable-cache proxy**, not a first run after reboot: shared
+libraries, the loader, Python benchmark process, and filesystem caches outside
+the CLI executable may remain warm. Verify that evicted runs have major faults;
+otherwise the eviction request did not establish the intended condition.
+
+On a Linux 6.18 WSL2 x86-64 host with an ext4 filesystem and a 2,296,776-byte
+release CLI, 100 runs per case and mode on 2026-09-22 measured:
+
+| Case | Cache state | Mean | Median | p95 | Runs with major faults |
+| --- | --- | ---: | ---: | ---: | ---: |
+| `--help` | Warm | 1.57 ms | 1.52 ms | 1.94 ms | 0/100 |
+| `--help` | Executable evicted | 7.64 ms | 7.61 ms | 8.15 ms | 100/100 |
+| Invalid backend | Warm | 1.66 ms | 1.64 ms | 1.99 ms | 0/100 |
+| Invalid backend | Executable evicted | 7.75 ms | 7.71 ms | 8.43 ms | 100/100 |
+
+The original #50 proposal mentioned a <5 ms cold-start target without a
+supported-host requirement or user report behind it. These measurements do not
+identify a startup problem: even the executable-evicted median is under 8 ms
+on this host. The major faults and roughly 6 ms gap point to file loading as
+the main extra cost here; the measurement does not establish which part is
+inherent or what another storage system would do. No fixed cold-start target
+is currently required.
+
+## Error display
+
+Misspelled and incompatible backends and invalid features produce an actionable
+message listing supported values. On a terminal with `TERM=xterm-256color` and
+no `NO_COLOR`, clap colors the error label and usage text. Redirected output and
+terminals with `NO_COLOR=1` contain no ANSI escape codes. These three cases
+were checked with both a pseudo-terminal and captured stderr on the same host;
+the CLI needed no diagnostic code change.

benchmarks/cli/startup.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""Measure fresh CLI processes with warm or evicted executable file pages.
+
+Run on Linux after building the release CLI. Eviction uses POSIX_FADV_DONTNEED
+for the CLI executable only; shared libraries and the filesystem may stay warm.
+"""
+
+import argparse
+import json
+import os
+import platform
+import resource
+import statistics
+import tempfile
+import time
+from pathlib import Path
+
+
+def run_once(
+    binary: Path, args: tuple[str, ...], expected_status: int
+) -> dict[str, float | int]:
+    before = resource.getrusage(resource.RUSAGE_CHILDREN)
+    started = time.perf_counter_ns()
+    pid = os.posix_spawn(
+        binary,
+        (str(binary), *args),
+        os.environ,
+        file_actions=(
+            (os.POSIX_SPAWN_OPEN, 1, os.devnull, os.O_WRONLY, 0),
+            (os.POSIX_SPAWN_OPEN, 2, os.devnull, os.O_WRONLY, 0),
+        ),
+    )
+    _, status = os.waitpid(pid, 0)
+    elapsed_ms = (time.perf_counter_ns() - started) / 1_000_000
+    actual_status = os.waitstatus_to_exitcode(status)
+    if actual_status != expected_status:
+        raise RuntimeError(
+            f"{' '.join(args)} exited {actual_status}, expected {expected_status}"
+        )
+    after = resource.getrusage(resource.RUSAGE_CHILDREN)
+    return {
+        "elapsed_ms": elapsed_ms,
+        "major_faults": after.ru_majflt - before.ru_majflt,
+    }
+
+
+def evict_executable(binary: Path) -> None:
+    with binary.open("rb") as executable:
+        os.posix_fadvise(executable.fileno(), 0, 0, os.POSIX_FADV_DONTNEED)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--binary", type=Path, default=Path("target/release/polyxml"))
+    parser.add_argument("--runs", type=int, default=30)
+    parser.add_argument(
+        "--output", type=Path, default=Path("benchmarks/cli/target/startup.json")
+    )
+    args = parser.parse_args()
+    if args.runs < 1:
+        parser.error("--runs must be positive")
+    if not hasattr(os, "posix_fadvise"):
+        parser.error("POSIX_FADV_DONTNEED requires Linux/POSIX")
+    binary = args.binary.resolve(strict=True)
+
+    with tempfile.TemporaryDirectory() as temp_dir:
+        missing_schema = str(Path(temp_dir) / "missing.xsd")
+        cases = {
+            "help": (("--help",), 0),
+            "argument-validation": (
+                ("generate", missing_schema, "--lang", "ts", "--backend", "zodd"),
+                2,
+            ),
+        }
+        results = {}
+        for name, (command, expected_status) in cases.items():
+            for _ in range(5):
+                run_once(binary, command, expected_status)
+            samples = {"warm": [], "executable-evicted": []}
+            for _ in range(args.runs):
+                samples["warm"].append(run_once(binary, command, expected_status))
+                evict_executable(binary)
+                samples["executable-evicted"].append(
+                    run_once(binary, command, expected_status)
+                )
+            results[name] = {}
+            for mode, observations in samples.items():
+                times = [observation["elapsed_ms"] for observation in observations]
+                summary = {
+                    "mean_ms": statistics.mean(times),
+                    "median_ms": statistics.median(times),
+                    "p95_ms": sorted(times)[max(0, (95 * len(times) + 99) // 100 - 1)],
+                    "min_ms": min(times),
+                    "max_ms": max(times),
+                    "runs_with_major_faults": sum(
+                        observation["major_faults"] > 0 for observation in observations
+                    ),
+                    "samples": observations,
+                }
+                results[name][mode] = summary
+                print(
+                    f"{name} {mode}: mean {summary['mean_ms']:.2f} ms, "
+                    f"median {summary['median_ms']:.2f} ms, "
+                    f"major faults in {summary['runs_with_major_faults']}/{args.runs} runs"
+                )
+
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(
+        json.dumps(
+            {
+                "method": "fresh process via posix_spawn; executable file pages requested evicted via POSIX_FADV_DONTNEED; shared libraries and schema cache untouched",
+                "binary": str(binary),
+                "binary_size_bytes": binary.stat().st_size,
+                "host": platform.platform(),
+                "runs_per_case_and_mode": args.runs,
+                "results": results,
+            },
+            indent=2,
+        )
+        + "\n"
+    )
+
+
+if __name__ == "__main__":
+    main()