Unreviewed behavior change: comprehensions become @cluster auto-parallelization candidates #132

Part of #108 · Discovered while reviewing the closed PR #128.

Problem

The closed epic branch epic/test-coverage-90-percent contains an unreviewed behavior change to the core @cluster feature, buried inside a PR labelled "test coverage".

clustrix/loop_analysis.py goes 787 → 1,712 lines. Critically, LoopDetector gains:

visit_ListComp, visit_SetComp, visit_DictComp, visit_GeneratorExp

LoopDetector is reached via find_parallelizable_loops, which clustrix/decorator.py:8 imports. So comprehensions become auto-parallelization candidates for @cluster — a real semantic change to the package's headline feature.

It also adds public API: try_parse_ast_safely, integrate_with_decorator, validate_analysis_results, parallelization_suggestions.

Why this needs its own review

1. Zero CI coverage. The tests written for it live at tests/ root; CI runs only tests/unit/.
2. Auto-parallelization is already broken. Per Fix broken backends: PBS runs a nonexistent file, local auto-parallelization silently no-ops #120, decorator.py:778 injects _parallel_{variable} into kwargs that user functions cannot accept, the resulting TypeError is swallowed at :708-716, and the feature silently falls back to sequential. Widening what gets detected while the execution path is broken increases the number of silently-degraded cases rather than improving anything.
3. Comprehensions have different semantics from loops. A comprehension is an expression producing a value; splitting one across workers and recombining is not obviously equivalent, particularly for dict/set comprehensions where ordering and key collisions matter.

Acceptance criteria

• Decide whether comprehension parallelization is wanted at all, and record the reasoning
• If yes: land it as an explicit feature change with its own PR, after Fix broken backends: PBS runs a nonexistent file, local auto-parallelization silently no-ops #120 makes auto-parallelization actually work
• A test that runs a @cluster function containing a comprehension end to end and asserts the combined result equals the sequential result
• Explicit handling for dict/set comprehension ordering and key-collision semantics
• The mock-free tests tests/test_loop_analysis_ast.py and tests/test_loop_analysis_advanced.py come along, moved into tests/unit/

Where the code is

Branch epic/test-coverage-90-percent (retained; PR #128 closed).
git diff master...origin/epic/test-coverage-90-percent -- clustrix/loop_analysis.py