Restore SGE support: removed from v0.2.0 as never verified against a real scheduler

[jeremymanning]

SGE (Sun Grid Engine / Grid Engine) support is implemented and removed from v0.2.0 as unverified. It has never been run against a real Grid Engine installation.

What exists, and where

Introduced in 4b2aba5 (2025-06-24). At 299109f:

Code	Location
Job submission	clustrix/executor_schedulers.py:251 submit_sge_job
Status polling	clustrix/executor_scheduler_status.py:517 _check_sge_status
Dispatch	clustrix/executor_core.py:115,553
Script generation	clustrix/utils.py:2803 _create_sge_script; dispatch at :2511,2548
Tutorial	docs/source/notebooks/sge_tutorial.ipynb

Specific risk to check on restoration

SGE shares the SLURM/PBS staging and environment-setup path, but its directive syntax (#$ -l h_rt=, #$ -pe) and its qstat output format are distinct from both. The status parser and the resource-directive generator are the two places where an untested assumption would show up first: a misparsed qstat line means clustrix reports a running job as finished, which is a silent wrong answer rather than an error.

Why it is being removed rather than fixed

Not because the code is known to be wrong. Because it has never been run against the real thing, and shipping it in the cluster-type dropdown states otherwise. A user who selects it gets a code path no one has ever seen succeed.

v0.2.0 keeps exactly the four backends that have been demonstrated end to end -- local, ssh, slurm, huggingface -- and the documentation now says the rest are planned for a future release rather than currently supported.

Restoring it

Nothing is lost: every line cited above stays reachable in git history at the commits named. Reinstating it means reverting the removal commit and then doing the part that was never done -- running it against real hardware and recording the evidence in this issue.

Definition of done

• Backend restored from history
• A real job submitted, executed and its result returned, with the transcript pasted into this issue
• Failure paths exercised (job rejected, job killed, node lost)
• Re-added to SUPPORTED_CLUSTER_TYPES, the widget dropdown and the CLI
• Documentation moved from "planned" to "supported"

[jeremymanning]

Removed in PR #149

SGE is gone from the code as of PR #149. This issue stays open as the record of what was removed and what bringing it back requires.

Evidence that a user is told what happened

>>> from clustrix.config import ClusterConfig
>>> ClusterConfig(cluster_type="sge")
ValueError: cluster_type='sge' is no longer implemented. It was removed in v0.2.0 because it had never been verified against real hardware. Its return is tracked in issue #141. Supported types are: local, ssh, slurm, huggingface.

The same check runs from configure(), from load_config() on an existing clustrix.yml, and in ClusterExecutor before it connects. That last one matters: the type used to be checked after self.connect(), so asking for sge cost an SSH round trip to a host that was never going to be used, and then said only Unsupported cluster type: sge.

Evidence it is actually gone

$ grep -rn "\bsge\b" clustrix/ | grep -v <the removed-backend tables in config.py>
21 matches

The only remaining mentions in clustrix/ are the REMOVED_CLUSTER_TYPES and _REMOVED_SETTINGS tables in config.py, which exist precisely so this issue number reaches the user.

Supported types are now exactly local, ssh, slurm, huggingface — read from SUPPORTED_CLUSTER_TYPES by the CLI's click.Choice and by all three notebook widgets, so the list cannot drift again. (enhanced_notebook_widget.py was carrying a third hardcoded copy that still offered five backends the executor could not dispatch.)

Verification on the final tree

pytest tests/ -m "not real_world" ...   1240 passed, 18 skipped, 0 failed
black --check (pinned 26.3.1)           223 files unchanged
flake8                                  clean
mypy clustrix/                          no issues in 34 source files
scripts/check_docs_examples.py          143/143 passed
sphinx -b html -W                        build succeeded, zero warnings

Tests covering this specific behaviour live in tests/test_config.py::TestRemovedBackendsAreExplained — real YAML files written to disk and read back through load_config, asserting the message names the backend, says it was never verified against real hardware, cites this issue number, and lists the supported set.

What is still required to close this

Unchanged from the original description: every line cited above stays reachable in git history at the commits named there. Reinstating SGE means reverting the removal and then doing the part that was never done — running it against real hardware and recording the evidence here. Restoring the code alone does not close this issue.

[jeremymanning]

Tracked as a sub-issue of #160 (deferred). Consolidated after a three-agent audit that verified every open issue against the code; see #160 for the plan and the ordering.