Restore Kubernetes execution and provisioning: removed from v0.2.0 as never verified against a real cluster

[jeremymanning]

Kubernetes execution and auto-provisioning are implemented and removed from v0.2.0 as unverified. No job has ever been run against a real cluster.

What exists, and where

Code	Lines	Introduced
clustrix/executor_kubernetes.py	594	6061247 (2025-08-23)
clustrix/kubernetes/cluster_provisioner.py	423	d6b4b7b (2025-08-16)
clustrix/kubernetes/local_provisioner.py	455	d6b4b7b
clustrix/kubernetes/{aws,azure,gcp,lambda,huggingface}_provisioner.py	3,611	d6b4b7b
clustrix/kubernetes/__init__.py	60	d6b4b7b

Plus the widget's Kubernetes section, the k8s_* configuration fields, and docs/source/notebooks/kubernetes_tutorial.ipynb.

Defects found while auditing

Two were real and are worth carrying forward whenever this is restored:

1. check_k8s_job_status reported failed jobs as successful. Its exception paths returned "completed", so a job that died came back as a normal completion.
2. Results were decoded with ast.literal_eval on the pod log, falling back to returning the raw log text as if it were the function's return value.

Both were fixed during the v0.2.0 sweep, and neither fix has been run against a real cluster.

Note on scope

The five cloud provisioners under clustrix/kubernetes/ are removed with this issue rather than with the per-provider issues, because they are Kubernetes cluster provisioners, not the VM-based cloud execution backends. See the AWS/GCP/Azure/Lambda issues for those.

Why it is being removed rather than fixed

Not because the code is known to be wrong. Because it has never been run against the real thing, and shipping it in the cluster-type dropdown states otherwise. A user who selects it gets a code path no one has ever seen succeed.

v0.2.0 keeps exactly the four backends that have been demonstrated end to end -- local, ssh, slurm, huggingface -- and the documentation now says the rest are planned for a future release rather than currently supported.

Restoring it

Nothing is lost: every line cited above stays reachable in git history at the commits named. Reinstating it means reverting the removal commit and then doing the part that was never done -- running it against real hardware and recording the evidence in this issue.

Definition of done

• Backend restored from history
• A real job submitted, executed and its result returned, with the transcript pasted into this issue
• Failure paths exercised (job rejected, job killed, node lost)
• Re-added to SUPPORTED_CLUSTER_TYPES, the widget dropdown and the CLI
• Documentation moved from "planned" to "supported"

[jeremymanning]

Removed in PR #149

Kubernetes is gone from the code as of PR #149. This issue stays open as the record of what was removed and what bringing it back requires.

Evidence that a user is told what happened

>>> from clustrix.config import ClusterConfig
>>> ClusterConfig(cluster_type="kubernetes")
ValueError: cluster_type='kubernetes' is no longer implemented. It was removed in v0.2.0 because it had never been verified against real hardware. Its return is tracked in issue #142. Supported types are: local, ssh, slurm, huggingface.

The same check runs from configure(), from load_config() on an existing clustrix.yml, and in ClusterExecutor before it connects. That last one matters: the type used to be checked after self.connect(), so asking for kubernetes cost an SSH round trip to a host that was never going to be used, and then said only Unsupported cluster type: kubernetes.

Evidence it is actually gone

$ grep -rn "\bkubernetes\b" clustrix/ | grep -v <the removed-backend tables in config.py>
59 matches

The only remaining mentions in clustrix/ are the REMOVED_CLUSTER_TYPES and _REMOVED_SETTINGS tables in config.py, which exist precisely so this issue number reaches the user.

Supported types are now exactly local, ssh, slurm, huggingface — read from SUPPORTED_CLUSTER_TYPES by the CLI's click.Choice and by all three notebook widgets, so the list cannot drift again. (enhanced_notebook_widget.py was carrying a third hardcoded copy that still offered five backends the executor could not dispatch.)

Verification on the final tree

pytest tests/ -m "not real_world" ...   1240 passed, 18 skipped, 0 failed
black --check (pinned 26.3.1)           223 files unchanged
flake8                                  clean
mypy clustrix/                          no issues in 34 source files
scripts/check_docs_examples.py          143/143 passed
sphinx -b html -W                        build succeeded, zero warnings

Tests covering this specific behaviour live in tests/test_config.py::TestRemovedBackendsAreExplained — real YAML files written to disk and read back through load_config, asserting the message names the backend, says it was never verified against real hardware, cites this issue number, and lists the supported set.

What is still required to close this

Unchanged from the original description: every line cited above stays reachable in git history at the commits named there. Reinstating Kubernetes means reverting the removal and then doing the part that was never done — running it against real hardware and recording the evidence here. Restoring the code alone does not close this issue.

[jeremymanning]

Tracked as a sub-issue of #160 (deferred). Consolidated after a three-agent audit that verified every open issue against the code; see #160 for the plan and the ordering.