Experiment: persistent evaluator for inline CUDA and Triton

scripts/persistent_eval/README.md

@@ -0,0 +1,109 @@
+# Persistent evaluator experiment
+
+KernelBot's warm Modal function still starts fresh Python processes for submission
+import and evaluation. This experiment measures how much wall time can be saved
+by retaining one Python interpreter and CUDA context across trusted submissions.
+It does not modify or enable the production runner.
+
+## Run
+
+From the repository root, with Modal authentication configured:
+
+```sh
+uv run --no-project --with modal==1.5.5 --with pyyaml python \
+  scripts/persistent_eval/launch.py --workload inline \
+  --output-dir /tmp/kernelbot-inline-results
+```
+
+Use `--workload triton` for the other existing vector-add example. The output
+directory must not exist. The default is four submissions per batch, two rounds
+in fresh/persistent then persistent/fresh order, on one ephemeral T4 with eight
+CPU cores and 32 GiB RAM. `--count 1 --rounds 1` makes a smaller smoke test.
+Every run also checks correct → incorrect → correct code in one persistent
+worker. For CUDA this keeps the same `load_inline(name="add_cuda")` name and
+changes addition to subtraction; the middle request must fail correctness.
+
+The launcher saves complete results, phase output, source hashes, compiled-library
+names/hashes, machine metadata, and logs before stopping the sandbox. Result
+export is compressed, chunked, and checksummed to avoid provider log-line limits.
+Each completed batch is checkpointed separately so its full details can be
+recovered from retained provider logs if the local connection drops.
+GPU cost estimates exclude CPU/RAM and are approximate, using $0.000164/T4-second.
+
+## What changes
+
+Both modes call the repository's `run_config`. The experiment coordinator caches
+`SystemInfo` once for both modes. Each submission has a separate working directory
+and cold Torch extension, Triton, and CUDA disk caches. For inline CUDA, the normal
+preliminary import compiles the extension, and the benchmark request reloads it.
+The compiler is limited to two jobs per submission.
+
+```mermaid
+flowchart LR
+    C[run_config] --> I[Submission import]
+    I --> B[Benchmark request]
+    B --> F[Fresh: eval.py plus spawned GPU worker]
+    B --> P[Persistent: socket to the same Python and CUDA process]
+    F --> E[Existing correctness and CUDA-event timing bodies]
+    P --> E
+```
+
+Fresh mode uses `run_program` unchanged. Persistent mode replaces only that
+transport with a Unix socket request, reloads task/reference/submission modules,
+and calls the existing `run_benchmarking` body with an in-process `Pool.apply`
+adapter. This removes both interpreter startup and the evaluator's spawned GPU
+worker. The worker synchronizes CUDA and clears unused allocator cache after
+successful requests. GPU measurements are sequential in both modes; submitting
+an entire batch at once is not required. The persistent worker owns the sandbox's
+assigned GPU; this experiment does not need a GPU admission lock.
+
+Batch time includes worker startup/teardown, compilation, validation, and all
+submission requests, but excludes Modal image/container startup and the once-per-
+experiment machine probe. Each request records compile/import and benchmark wall
+time separately. Kernel times remain a distinct metric from evaluation latency.
+
+## Scope and review points
+
+This is a benchmark prototype for the bundled trusted examples, not isolation
+for arbitrary uploaded code. It covers single-GPU benchmark mode only. The
+persistent transport does not implement production per-request timeout handling,
+CUDA-fault recovery, process-state isolation, or unloading native extensions.
+Modules and allocator cleanup do not reset all process state. Python output is
+captured per request, while compiler subprocess output remains in the sandbox
+log; the existing stdout-based `CompileResult` classification can therefore
+differ between modes. Phase timings and generated `.so` files are recorded
+separately to verify that compilation happened. A production design
+needs bounded worker recycling and scoring consistency checks. The sandbox has
+an overall 30-minute timeout and is terminated by the launcher on exit.
+
+The minimal pinned CUDA 12.8.1 / Torch 2.7.1 image is deliberately consistent with
+the earlier experiment; it is not KernelBot's production image. Results on this
+small vector-add workload should not be extrapolated to compilation-heavy or
+long-running competition tasks without measuring them.
+
+[Inline CUDA results](RESULTS.md): **19–28% less evaluation wall time**, with
+compilation still dominating. Reported kernel medians shifted 12–15% in the
+complete repeat, so score equivalence remains unresolved. Earlier separate Triton measurements were 51.3 seconds fresh versus
+12.5 seconds persistent per four-submission batch (two rounds). Those results
+motivated this experiment; they are not an inline CUDA claim. That experiment
+also saw an approximately 8.5% shift in the smallest shape's reported kernel
+time, which needs investigation before using persistence for ranked scores.
+
+## Local checks
+
+```sh
+uv run --no-project --with pytest --with modal==1.5.5 --with pyyaml \
+  pytest scripts/persistent_eval/test_experiment.py -q
+uv run --no-project --with ruff ruff check . --exclude examples/ --line-length 120
+```
+
+To audit a completed run and emit compact shareable JSON:
+
+```sh
+python scripts/persistent_eval/summarize.py /tmp/kernelbot-inline-results/results.json \
+  > /tmp/kernelbot-inline-summary.json
+```
+
+The GPU run supplies the actual correctness/reload and performance evidence.
+The CPU tests cover source/config preservation and evidence transport, including
+large outputs and checksum rejection.

scripts/persistent_eval/RESULTS.md

@@ -0,0 +1,96 @@
+# Inline CUDA: persistence saves 19–28% of evaluation wall time
+
+Measured on 2026-09-04 using KernelBot's bundled inline-CUDA FP16 vector-add
+example, three benchmark shapes (1024², 2048², 4096²), and cold private build
+caches for every submission. GPU measurements remain sequential.
+
+## Turnaround
+
+Each batch contains four submissions and includes worker startup/teardown.
+Compilation and correctness checks are included; Modal image/container startup
+and the once-per-experiment machine probe are excluded.
+
+| Experiment | Fresh processes | Persistent worker | Less wall time |
+|---|---:|---:|---:|
+| Initial allocation, median of two counterbalanced rounds | 173.8 s | 125.4 s | 27.8% |
+| Complete repeat on another allocation, one fresh/persistent round | 151.1 s | 122.2 s | 19.1% |
+
+That is approximately **29–48 seconds saved per four-submission batch** on this
+workload. Each comparison uses the same assigned GPU and CPU resources for both
+modes. Absolute times vary across the two allocations; do not pool those times
+into a single baseline. This small experiment is not a production workload
+estimate or a confidence interval.
+
+## What still takes time
+
+Per-request phase medians from the complete repeat:
+
+| Phase | Fresh processes | Persistent worker |
+|---|---:|---:|
+| Compile and import submission | 31.74 s | 29.85 s |
+| Benchmark request, including validation and setup | 5.89 s | 0.41 s |
+
+Keeping Python/CUDA alive removes most benchmark-request startup. Rebuilding
+C++/CUDA dominates the remaining time. Every inline request produced its own
+native library; persistent requests used `add_cuda.so`, then `add_cuda_v1.so`,
+`add_cuda_v2.so`, and `add_cuda_v3.so` in separate cold build directories.
+The phase medians are individual-request statistics, not additive batch totals.
+
+## Kernel scores are not yet equivalent
+
+The same correctness and CUDA-event timing function bodies produced different
+reported kernel times. Below are medians of each submission's reported mean in
+the complete repeat. These are GPU microseconds, distinct from evaluation wall
+seconds above.
+
+| Shape | Fresh, µs | Persistent, µs | Persistent change |
+|---|---:|---:|---:|
+| 1024² | 36.02 | 40.28 | +11.8% |
+| 2048² | 108.37 | 123.65 | +14.1% |
+| 4096² | 407.96 | 468.47 | +14.8% |
+
+These 12–15% shifts require investigation before using the prototype for ranked
+scores. This one ordered repeat does not separate process-state effects from
+clock/thermal drift or measurement variability; GPU clocks were not pinned.
+There is no claim that the kernel itself became faster or that the two execution
+modes are score-equivalent.
+
+## Correctness and code replacement
+
+All 24 timed submissions passed across the two experiments. The complete repeat
+also ran **correct addition → incorrect subtraction → correct addition** in one
+persistent worker, retaining the requested extension name `add_cuda`. Observed
+results were **pass → fail → pass**. The incorrect variant compiled successfully
+and failed with numerical mismatches on all three shapes; this was not a compiler
+or import failure. Restored code produced correct results again.
+
+The worker still does not isolate arbitrary user code, unload native modules,
+implement production request timeouts, or recover from a poisoned CUDA context.
+This PR remains an experiment with no production runner changes.
+
+## Evidence and reproduction
+
+- [Initial timing observations](results/inline-initial-timings.json): two complete
+  rounds, recovered from provider logs after a local DNS/connection failure. The
+  final reload check was interrupted. Full phase/kernel records were not exported,
+  so this file explicitly marks its limited detail.
+- [Complete repeat](results/inline-repeat.json): all per-submission/phase timings,
+  correctness output, kernel statistics, native-library names/hashes, and a hash
+  of the full local result. Compiler logs are omitted from this compact artifact.
+- [Repeat manifest](results/inline-repeat-manifest.json): runtime/source hashes,
+  pinned image, resources, and stopped sandbox metadata. The uploaded source
+  hashes match the PR runtime files at commit `4de31428`.
+- [Run instructions](README.md): reproducible fresh/persistent comparison and
+  correct/incorrect/correct check. The default runs two counterbalanced rounds;
+  this complete repeat used `--rounds 1` after the original two-round matrix.
+
+The complete repeat used one Tesla T4, eight reserved CPU cores, 32 GiB RAM,
+CUDA 12.8.1 compiler image, Torch 2.7.1, and `MAX_JOBS=2`. It took 406 seconds
+including local launch/admission overhead, with an estimated $0.067 GPU-only
+cost at $0.000164/T4-second; CPU/RAM are excluded. Both experiment sandboxes are
+stopped. No production deployment, submissions, or database writes were made.
+
+Validation: repository-wide Ruff and three CPU regression tests pass. The
+summarizer additionally audits the actual GPU result, requiring valid timing
+statistics, successful compilation, native-library artifacts, and numerical
+mismatch evidence for the negative reload test.

scripts/persistent_eval/experiment.py

@@ -0,0 +1,169 @@
+"""Compare fresh and persistent processes with cold per-submission caches."""
+
+import argparse
+import contextlib
+import hashlib
+import json
+import os
+import statistics
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+from export_result import emit_result
+from persistent import running_server
+
+
+def run_job(root, index, workload, mode):
+    """Use a new coordinator and private build directory for each submission."""
+    directory = root / f"job-{index}-{workload}"
+    directory.mkdir(parents=True)
+    env = dict(
+        os.environ,
+        MAX_JOBS="2",
+        OMP_NUM_THREADS="1",
+        MKL_NUM_THREADS="1",
+        OPENBLAS_NUM_THREADS="1",
+        TORCH_CUDA_ARCH_LIST="7.5",
+        TORCH_EXTENSIONS_DIR=str(directory / "torch-cache"),
+        TRITON_CACHE_DIR=str(directory / "triton-cache"),
+        CUDA_CACHE_PATH=str(directory / "cuda-cache"),
+    )
+    started = time.monotonic()
+    child = subprocess.run(
+        [sys.executable, "/experiment/worker.py", f"/experiment/{workload}.json", mode],
+        cwd=directory,
+        env=env,
+        capture_output=True,
+        text=True,
+        timeout=420,
+    )
+    wall_s = time.monotonic() - started
+    result_file = directory / "result.json"
+    row = json.loads(result_file.read_text()) if result_file.exists() else {"passed": False}
+    artifacts = [
+        {
+            "name": str(p.relative_to(directory)),
+            "bytes": p.stat().st_size,
+            "sha256": hashlib.sha256(p.read_bytes()).hexdigest(),
+        }
+        for p in sorted(directory.rglob("*.so"))
+    ]
+    row.update(
+        index=index,
+        workload=workload,
+        mode=mode,
+        wall_s=wall_s,
+        returncode=child.returncode,
+        stdout=child.stdout,
+        stderr=child.stderr,
+        compiled_extensions=artifacts,
+    )
+    print("JOB", json.dumps({k: row[k] for k in ("index", "workload", "mode", "wall_s", "passed")}), flush=True)
+    return row
+
+
+def run_batch(root, index, mode, workload, count):
+    """Include worker startup and teardown; GPU measurements stay sequential."""
+    started = time.monotonic()
+    directory = root / f"batch-{index}-{mode}"
+    server = running_server() if mode == "persistent" else contextlib.nullcontext()
+    with server:
+        jobs = [run_job(directory, i, workload, mode) for i in range(count)]
+    result = {
+        "index": index,
+        "mode": mode,
+        "wall_s": time.monotonic() - started,
+        "passed": all(j["passed"] for j in jobs),
+        "jobs": jobs,
+    }
+    print("BATCH", json.dumps({k: v for k, v in result.items() if k != "jobs"}), flush=True)
+    return result
+
+
+def check_reload(root, workload):
+    """Require correct/wrong/correct with the same inline extension name."""
+    wrong = json.loads(Path(f"/experiment/{workload}.json").read_text())
+    before, after = (
+        ("C[idx] = A[idx] + B[idx]", "C[idx] = A[idx] - B[idx]") if workload == "inline" else ("C = A + B", "C = A - B")
+    )
+    assert wrong["sources"]["submission.py"].count(before) == 1
+    wrong["sources"]["submission.py"] = wrong["sources"]["submission.py"].replace(before, after)
+    Path("/experiment/incorrect.json").write_text(json.dumps(wrong))
+    with running_server():
+        jobs = [
+            run_job(root / "reload", i, name, "persistent") for i, name in enumerate([workload, "incorrect", workload])
+        ]
+    observed = [j["passed"] for j in jobs]
+    # A compiler/import failure is not evidence of detecting incorrect math.
+    wrong_runs = jobs[1].get("result", {}).get("runs", {})
+    wrong_results = [r.get("run", {}) for r in wrong_runs.values()]
+    correctness_rejected = any(r and r.get("result", {}).get("check") == "fail" for r in wrong_results)
+    result = {
+        "expected": [True, False, True],
+        "observed": observed,
+        "jobs": jobs,
+        "correctness_rejected": correctness_rejected,
+        "passed": observed == [True, False, True] and correctness_rejected,
+    }
+    print("RELOAD", json.dumps({k: v for k, v in result.items() if k != "jobs"}), flush=True)
+    return result
+
+
+def main():
+    """Run counterbalanced rounds and export checksummed complete evidence."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--workload", choices=["inline", "triton"], default="inline")
+    parser.add_argument("--count", type=int, default=4)
+    parser.add_argument("--rounds", type=int, default=2)
+    args = parser.parse_args()
+    assert args.count > 0 and args.rounds > 0
+    root = Path("/experiment/output")
+    root.mkdir()
+    info = subprocess.check_output(
+        [
+            sys.executable,
+            "-c",
+            "import json,dataclasses; from libkernelbot.run_eval import make_system_info; "
+            "print(json.dumps(dataclasses.asdict(make_system_info())))",
+        ],
+        text=True,
+    )
+    Path("/experiment/system.json").write_text(info)
+    assert json.loads(info)["device_count"] == 1
+    hardware = subprocess.check_output(
+        ["nvidia-smi", "--query-gpu=name,uuid,driver_version,memory.total", "--format=csv"], text=True
+    )
+    print("HARDWARE", hardware.strip(), flush=True)
+    batches = []
+    for round_index in range(args.rounds):
+        order = ["fresh", "persistent"] if round_index % 2 == 0 else ["persistent", "fresh"]
+        for mode in order:
+            batches.append(run_batch(root, len(batches), mode, args.workload, args.count))
+            emit_result({"checkpoint": "batch", "batch": batches[-1]})
+            if not batches[-1]["passed"]:
+                break
+        if not batches[-1]["passed"]:
+            break
+    reload_check = check_reload(root, args.workload) if all(b["passed"] for b in batches) else None
+    summary = {
+        mode: statistics.median(b["wall_s"] for b in batches if b["mode"] == mode)
+        for mode in {b["mode"] for b in batches}
+    }
+    result = {
+        "system": json.loads(info),
+        "hardware": hardware,
+        "arguments": vars(args),
+        "batches": batches,
+        "summary": summary,
+        "reload": reload_check,
+        "passed": all(b["passed"] for b in batches) and bool(reload_check and reload_check["passed"]),
+    }
+    print("SUMMARY", json.dumps(summary), flush=True)
+    emit_result(result)
+    return 0 if result["passed"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/persistent_eval/export_result.py

@@ -0,0 +1,15 @@
+"""Export complete evidence without exceeding the provider's log line limit."""
+
+import base64
+import gzip
+import hashlib
+import json
+
+
+def emit_result(result):
+    """Emit compressed chunks and a checksum for reliable local reconstruction."""
+    raw = json.dumps(result).encode()
+    encoded = base64.b64encode(gzip.compress(raw)).decode()
+    for offset in range(0, len(encoded), 24000):
+        print("EXPERIMENT_CHUNK=" + encoded[offset : offset + 24000], flush=True)
+    print("EXPERIMENT_END=" + hashlib.sha256(raw).hexdigest(), flush=True)