Red-team campaign: trust boundary, script injection, serialization fuzzing, cost safety #126

Part of #108 · Phase 6 · label: red-team · Do this after Phases 0-5, not before.

Red-teaming a codebase whose test suite cannot run and whose cloud path has never executed would just rediscover known defects. This issue is for adversarial review of the hardened system.

Scope

1. The remote -> local trust boundary

The deepest structural risk (see the Phase 3 security issue). Once payload authentication lands, attack it:

• Tampered result.pkl with a valid-looking structure -> must be rejected
• Replayed payload from a previous job -> must be rejected
• Malicious error.pkl (the error path is easy to forget to authenticate)
• A job that writes an oversized/zip-bomb result -> must fail gracefully, not OOM the client
• Confirm the restricted unpickler cannot be escaped via __reduce__

2. Injection into generated remote scripts

Script generation concatenates strings (utils.py:1149-1186, executor_kubernetes.py:152-243, executor_cloud.py:376-414), and utils.py:1123 emits unquoted export {var}={value}.

• Hostile values in every interpolated config field: memory, time, partition, queue, env var names/values, remote_work_dir, cluster_name
• Shell metacharacters, newlines, $(...), backticks, null bytes, unicode lookalikes
• Same for the K8s payload's nested quoting around its base64 blob

3. Serialization / function-capture fuzzing

• Functions with closures over mutable state, generators, coroutines, lambdas, nested decorators, C extensions, unpicklable members
• Deeply recursive and cyclic argument structures
• Functions defined in a REPL (the known limitation — assert a clear error, per the docs issue)
• Confirm utils.py:81's eval(range_part) replacement cannot be driven to execute arbitrary code
• Confirm executor_kubernetes.py:217's exec(cleaned_source, ...) reconstruction is safe or gone

4. Credential handling

• Confirm no credential reaches argv, /proc/<pid>/environ in a way that surprises, a log, a traceback, or a temp file that outlives the process
• Verify file modes under a hostile umask
• Confirm the GCP temp-file fix actually unlinks on crash, not just on clean exit

5. Concurrency and resource exhaustion

• Many simultaneous submissions -> file-descriptor and thread growth (ties to the resource-leak issue)
• Job-ID collisions; the busy-poll dict at executor_cloud.py:438 under concurrent mutation
• Cancellation and timeout paths — currently several waits have no timeout at all

6. Cost safety

• Attempt to make the test suite spend money; it must be impossible without explicit opt-in (regression test for tests/integration/ is unmarked, so the documented unit-test command provisions billable AWS EKS clusters #109)
• Verify provisioning failures always tear down; no orphaned billable resources

Method

• Property-based testing (Hypothesis) for serialization and script generation
• An independent reviewer who did not write the fixes — the audit that produced this plan found that self-review consistently missed the seams
• bandit / pip-audit / CodeQL wired into CI
• Every finding gets a regression test before it is closed

Acceptance criteria

• Documented red-team report with reproduction steps for each finding
• Every finding either fixed with a regression test, or accepted with written rationale
• Security tooling runs in CI and blocks on new high-severity findings