Restore PBS support: removed from v0.2.0 as never verified against a real scheduler

[jeremymanning]

PBS support is implemented and removed from v0.2.0 as unverified. It has never been run against a real PBS/Torque scheduler.

What exists, and where

Introduced in 4b2aba5 (2025-06-24). At 299109f:

Code	Location
Job submission	clustrix/executor_schedulers.py:218 submit_pbs_job
Status polling	clustrix/executor_scheduler_status.py:489 _check_pbs_status
Dispatch	clustrix/executor_core.py:111,547,571
Script generation	clustrix/utils.py:2772 _create_pbs_script; dispatch at :2509,2546
Tutorial	docs/source/notebooks/pbs_tutorial.ipynb

Known defect found while auditing, never fixed

PBS was the one scheduler that never set up its remote environment -- SLURM and SGE staged and built a venv, PBS did not, so a PBS job ran against whatever interpreter happened to be on the node. That was corrected during the v0.2.0 sweep (all four schedulers now share one staging path), but the fix was itself never exercised against a real PBS queue. So the correction is as unverified as the original.

Why it is being removed rather than fixed

Not because the code is known to be wrong. Because it has never been run against the real thing, and shipping it in the cluster-type dropdown states otherwise. A user who selects it gets a code path no one has ever seen succeed.

v0.2.0 keeps exactly the four backends that have been demonstrated end to end -- local, ssh, slurm, huggingface -- and the documentation now says the rest are planned for a future release rather than currently supported.

Restoring it

Nothing is lost: every line cited above stays reachable in git history at the commits named. Reinstating it means reverting the removal commit and then doing the part that was never done -- running it against real hardware and recording the evidence in this issue.

Definition of done

• Backend restored from history
• A real job submitted, executed and its result returned, with the transcript pasted into this issue
• Failure paths exercised (job rejected, job killed, node lost)
• Re-added to SUPPORTED_CLUSTER_TYPES, the widget dropdown and the CLI
• Documentation moved from "planned" to "supported"

[jeremymanning]

Removed in PR #149

PBS is gone from the code as of PR #149. This issue stays open as the record of what was removed and what bringing it back requires.

Evidence that a user is told what happened

>>> from clustrix.config import ClusterConfig
>>> ClusterConfig(cluster_type="pbs")
ValueError: cluster_type='pbs' is no longer implemented. It was removed in v0.2.0 because it had never been verified against real hardware. Its return is tracked in issue #140. Supported types are: local, ssh, slurm, huggingface.

The same check runs from configure(), from load_config() on an existing clustrix.yml, and in ClusterExecutor before it connects. That last one matters: the type used to be checked after self.connect(), so asking for pbs cost an SSH round trip to a host that was never going to be used, and then said only Unsupported cluster type: pbs.

Evidence it is actually gone

$ grep -rn "\bpbs\b" clustrix/ | grep -v <the removed-backend tables in config.py>
25 matches

The only remaining mentions in clustrix/ are the REMOVED_CLUSTER_TYPES and _REMOVED_SETTINGS tables in config.py, which exist precisely so this issue number reaches the user.

Supported types are now exactly local, ssh, slurm, huggingface — read from SUPPORTED_CLUSTER_TYPES by the CLI's click.Choice and by all three notebook widgets, so the list cannot drift again. (enhanced_notebook_widget.py was carrying a third hardcoded copy that still offered five backends the executor could not dispatch.)

Verification on the final tree

pytest tests/ -m "not real_world" ...   1240 passed, 18 skipped, 0 failed
black --check (pinned 26.3.1)           223 files unchanged
flake8                                  clean
mypy clustrix/                          no issues in 34 source files
scripts/check_docs_examples.py          143/143 passed
sphinx -b html -W                        build succeeded, zero warnings

Tests covering this specific behaviour live in tests/test_config.py::TestRemovedBackendsAreExplained — real YAML files written to disk and read back through load_config, asserting the message names the backend, says it was never verified against real hardware, cites this issue number, and lists the supported set.

What is still required to close this

Unchanged from the original description: every line cited above stays reachable in git history at the commits named there. Reinstating PBS means reverting the removal and then doing the part that was never done — running it against real hardware and recording the evidence here. Restoring the code alone does not close this issue.

[jeremymanning]

Tracked as a sub-issue of #160 (deferred). Consolidated after a three-agent audit that verified every open issue against the code; see #160 for the plan and the ordering.