Master Plan: Get Clustrix Production-Ready

Status: Planning · Opened: 2026-08-17 · Target: v0.2.0 (honest beta) → v1.0.0 (production)

This is the umbrella issue for taking Clustrix from its current state to a package that does what it advertises, is tested against real infrastructure, and has been adversarially reviewed. It is grounded in a five-track audit performed 2026-08-17 covering the full source tree, the entire test suite (actually executed), all 29 open issues, all documentation, and a security review of the working tree and git history.

Every claim below is cited to file:line or to verbatim command output. Where something could not be verified it is marked UNVERIFIED rather than assumed.

1. Executive summary: what is actually true today

The single most important structural finding

Roughly every "completed" item in this project is half done in the same specific way: the scaffolding landed, the seam did not.

• Closure variables are detected and injected as parameters (function_flattening.py:623,654-695) — but never passed at the call site (:749-751).
• The REPL limitation is documented (README.md:554) — but the code still fails opaquely (utils.py:118-119).
• Kubernetes provisioners exist for all six targets — but GCP's _assign_iam_role() assigns nothing (kubernetes/gcp_provisioner.py:356) and Azure's _create_service_principal() creates nothing (kubernetes/azure_provisioner.py:331).
• AWS cleanup scripts exist — but have no dry-run, confirmation, or scoping (grep -rn "dry_run\|confirm\|input(" scripts/aws/*.py → 0 hits).
• The cloud executor serializes {"function": ...} (utils.py:151) and deserializes func_data['func'] (executor_cloud.py:390) — a guaranteed KeyError on the first real invocation.

That last one is diagnostic. It is not a regression; it is unconditional. Its survival proves the path was never executed. This is what heavy mocking produces: coverage rises while every integration seam stays untested. The fix is not "more tests" — it is tests that actually execute.

2. Root causes (fix these, not the symptoms)

RC1 — Tests mock the thing under test. 60 of 240 test files, 2,513 occurrences of unittest.mock|MagicMock|@patch|monkeypatch. Representative: tests/test_executor_schedulers.py:73-131 stubs execute_remote_command to return "Submitted batch job 12345", then asserts job_id == "12345" — it tests a regex against a string the test itself supplied. No SLURM is involved. This directly violates the project's own rule: "Do not use mock services for anything ever."

RC2 — Production code is mock-aware. clustrix/executor_scheduler_status.py:89 imports unittest.mock and branches on isinstance(self.connection_manager.ssh_client, Mock), with the comment "Use robust checking only if we have a real SSH connection (not unit tests)". The shipped code path _check_slurm_job_status_robust is structurally unreachable from any unit test. Separately, clustrix/notebook_magic_mocks.py ships fake widgets (_MockDropdown, observe(): pass) imported by six production modules.

RC3 — CI does not run the tests. tests.yml and fast_ci.yml both run only tests/unit/ = 5 files, 391 LOC, 15 tests. real-world-tests.yml has all three jobs gated if: false. real_world_tests.yml sets continue-on-error: true, as does the mypy step in tests.yml. ~99.3% of the suite never executes in CI, which is why all 127 failures are invisible.

RC4 — Planning ran against stale snapshots. #102 targets "notebook_magic.py, 1,236 statements"; that file is now 90 lines split across nine modules. #104 targets "executor.py, 511 lines"; that is now a 39-line re-export shim (executor.py:19-26). The work items were decomposed from one snapshot and the code was refactored underneath them.

RC5 — Failure is silent by default. 40+ except Exception: → pass/return None sites. executor_kubernetes.py:304-308 returns "completed" on any API exception. config.py:206 swallows all cloud-dependency install errors during __init__. Errors surface as wrong results rather than as errors.

RC6 — Local work never reached the remote. master is 30 commits ahead of origin/master (git rev-list --left-right --count origin/master...master → 0 30). All 30 implement issues #99–#106. quality_gates.yml is committed locally but returns HTTP 404 on GitHub — it has never run.

3. Backend reality check

Additionally, three providers return fabricated hostnames from get_cluster_config() error branches — placeholder.gcp.com (cloud_providers/gcp.py:544), placeholder.azure.com (azure.py:669), placeholder.lambdalabs.com (lambda_cloud.py:327,338) — which executor_cloud.py:260 then attempts to SSH into.

4. New capability: HuggingFace Jobs as the integration-test substrate

Verified working today against the contextlab org (academia plan). A real container ran and returned:

CLUSTRIX_HF_OK 3.12.14 x86_64

Available flavors: cpu-basic, cpu-upgrade, cpu-xl, t4-small, t4-medium, l4x1, l4x4, l40sx1, l40sx4, l40sx8, a10g-small, a10g-large, a10g-largex2, a10g-largex4, a100-large, h100, h100x8.

Why this matters. The reason the cloud path stayed broken for months is that verifying it meant provisioning, IAM, billing, and teardown — so nobody did, and a first-line KeyError went unnoticed. HF Jobs collapses that loop to one CLI call and a few cents, with no cluster reservation, no VPN, no institutional SSH credentials. That makes it a far more reliable CI substrate than the Dartmouth hosts the current real-world tests depend on (ndoli.dartmouth.edu, tensor01.dartmouth.edu), whose scheduled workflow has failed on all of its last 60 runs.

Note also that clustrix currently targets the wrong HF primitive. cloud_providers/huggingface_spaces.py targets Spaces — long-lived web apps. Jobs is exactly clustrix's model: hand over a container, run a function, collect a result, exit. An HF Jobs backend is both easier to implement correctly and immediately useful as test infrastructure.

Cost guardrail: anything CI-facing must pin --flavor cpu-basic with an explicit --timeout. GPU flavors stay manual and opt-in. h100x8 on the academia plan is real money.

5. Plan of record

Phases are ordered by dependency. Phase 0 must complete first — until then, running the test suite is unsafe and measuring anything is meaningless.

Phase 0 — Stop the bleeding (safety + ability to measure)

Nothing else can be trusted until these land.

• Contain the billable-test landmine
• Make the dev environment installable and the suite runnable
• Security hardening + token rotation + secret scanning
• Reconcile the 30 unpushed commits

Phase 1 — Establish ground truth

• Make CI actually run the suite
• Fix the 127 failures and 8 collection errors
• Set an honest, measured coverage baseline and a gate that means something

Phase 2 — De-mock

• Remove mock-awareness from shipped code
• Replace assertion-free mock tests with tests that execute
• Land the HF Jobs backend and adopt it as the default integration substrate

Phase 3 — Fix the core

• Cloud execution path and the provider interface
• PBS backend
• Local auto-parallelization
• The pickle trust model and SSH host-key verification

Phase 4 — Pay down architecture debt

• Delete ~5,100 lines of orphaned modules
• Replace silent-failure handling; fix resource leaks

Phase 5 — Make the docs true

• Fix every broken README example; unify versions
• Rewrite CLAUDE.md and MIGRATION.md against the real module layout

Phase 6 — Red team

• Adversarial review of serialization, credentials, injection, and the remote trust boundary

Phase 7 — Release

• Cut an honest v0.2.0; define v1.0 exit criteria

6. Exit criteria for v1.0

A release is production-ready when all of the following hold:

1. pip install -e ".[dev]" on a clean machine, followed by the documented test command, runs the suite to completion — with zero network calls to billable services.
2. CI executes >95% of the test suite on every PR, with continue-on-error removed everywhere.
3. Zero failing tests. Zero collection errors.
4. unittest.mock appears zero times in clustrix/ (shipped code).
5. Every backend claimed in README.md has a test that actually executed it end to end, with a dated CI run link as evidence.
6. Coverage is measured, published, and gated at a threshold the project actually meets.
7. Every code example in README.md is executed by a doc test in CI.
8. A single source of truth for the version; all four current locations agree.
9. Secret scanning + push protection enabled; no plaintext credential ever written world-readable.
10. The remote→local trust boundary is documented and either authenticated or explicitly opt-in.

7. Audit provenance

Five parallel audits, 2026-08-17, all read-only:

Known limits of this audit. The full-suite coverage figure (56.14%) is from combined worker data at 98% completion — the run reproducibly hangs in a real AWS retry loop (kubernetes/aws_provisioner.py:763,771), so it is a slight underestimate. PyPI publication status was not checked. Issue #87 (AWS EKS Service Control Policy) could not be verified from code as it is an AWS Organizations-level condition.

Sub-issues are linked below. Each carries its own evidence, acceptance criteria, and verification command.

8. Sub-issue roadmap

Phase 0 — Stop the bleeding (do these first; nothing else is trustworthy until they land)

• tests/integration/ is unmarked, so the documented unit-test command provisions billable AWS EKS clusters #109 — tests/integration/ is unmarked, so the documented unit-test command provisions billable AWS EKS clusters ⚠️
• pip install -e ".[dev]" produces an environment where pytest cannot start #110 — pip install -e ".[dev]" produces an environment where pytest cannot start
• Security hardening: rotate local-only HF tokens, enable secret scanning, fix credential file permissions #111 — Security hardening: rotate local-only HF tokens, enable secret scanning, fix credential file permissions
• Reconcile 30 unpushed local commits; quality_gates.yml has never run #112 — Reconcile the 30 unpushed local commits; quality_gates.yml has never run

Phase 1 — Establish ground truth

• Make CI actually run the test suite (currently 15 of 2,280 tests) #113 — Make CI actually run the test suite (currently 15 of 2,280)
• Fix the 127 test failures and 8 collection errors that CI never sees #114 — Fix the 127 test failures and 8 collection errors that CI never sees
• Establish a reproducible coverage baseline (4 conflicting numbers in circulation) #115 — Establish a reproducible coverage baseline (4 conflicting numbers in circulation)

Phase 2 — De-mock

• Remove mock-awareness from shipped code (production branches on isinstance(..., Mock)) #116 — Remove mock-awareness from shipped code (production branches on isinstance(..., Mock))
• Replace assertion-free mock tests with tests that actually execute (2,513 mock occurrences) #117 — Replace assertion-free mock tests with tests that execute (2,513 mock occurrences)
• Add HuggingFace Jobs backend and adopt it as the integration-test substrate #118 — Add HuggingFace Jobs backend and adopt it as the integration-test substrate

Phase 3 — Fix the core

• Fix the cloud execution path: KeyError, provider interface mismatch, placeholder hostnames #119 — Fix the cloud execution path: KeyError, provider interface mismatch, placeholder hostnames
• Fix broken backends: PBS runs a nonexistent file, local auto-parallelization silently no-ops #120 — Fix broken backends: PBS runs a nonexistent file; local auto-parallelization silently no-ops
• Security: unauthenticated pickle of remote results (RCE) and disabled SSH host key verification #121 — Security: unauthenticated pickle of remote results (RCE) and disabled SSH host-key verification ⚠️

Phase 4 — Architecture debt

• Delete ~5,100 lines of orphaned modules (13% of the package has zero importers) #122 — Delete ~5,100 lines of orphaned modules (13% of the package has zero importers)
• Replace silent-failure handling, fix resource leaks and import-time side effects #123 — Replace silent-failure handling; fix resource leaks and import-time side effects

Phase 5 — Make the docs true

• Fix all broken README/docs examples and unify the version across 4 locations #124 — Fix all broken README/docs examples; unify the version across 4 locations
• Rewrite CLAUDE.md against the real architecture; resolve the 3-way mocking-policy contradiction #125 — Rewrite CLAUDE.md against the real architecture; resolve the 3-way mocking-policy contradiction

Phase 6 — Red team

• Red-team campaign: trust boundary, script injection, serialization fuzzing, cost safety #126 — Trust boundary, script injection, serialization fuzzing, cost safety

Phase 7 — Release

• Release v0.2.0 (honest beta) and define v1.0 exit criteria #127 — Release v0.2.0 (honest beta) and define v1.0 exit criteria

9. Suggested order of attack

#109 first, alone. It is the only issue where not acting has an ongoing cost — anyone running the documented test command with AWS credentials is billing EKS right now.

Then #110 → #114 → #113: make the suite runnable, make it green, then turn CI on. Turning CI on before #114 makes master permanently red, and the temptation will be to re-add continue-on-error, which is how this situation arose.

#116 before #117. While production code still branches on isinstance(..., Mock), you can delete a thousand mock assertions and still be exercising the wrong code path.

#122 before #115. Deleting dead code moves the coverage number without writing a test; re-baselining first means measuring a denominator you are about to change.

#118 early — it unblocks honest testing for #119, #120, and #126.

10. Issue triage performed alongside this plan

Of the 29 previously-open issues: 9 closed, 8 updated with corrected bodies.