fix: cap ThreadPoolExecutor max_workers to prevent resource exhaustion

[hobostay]

Summary

• Cap ThreadPoolExecutor(max_workers=len(items)) to min(len(items), MAX_WORKERS) in workflow/executor/parallel_executor.py
• Prevents creating an unbounded number of threads when processing large batches of items

Bug Details

In workflow/executor/parallel_executor.py (line 106), the parallel executor creates a thread pool with max_workers=len(items):

with concurrent.futures.ThreadPoolExecutor(max_workers=len(items)) as executor:

If a workflow contains many parallel nodes, this could create thousands of threads, causing:

• Excessive memory usage
• Thread scheduling overhead that degrades performance
• Potential OOM crashes on resource-constrained systems

Python's own ThreadPoolExecutor default is min(32, os.cpu_count() + 4), which is a well-tested heuristic.

Fix

Added a MAX_WORKERS constant following the standard library pattern and capped the pool size:

MAX_WORKERS = min(32, (os.cpu_count() or 1) + 4)
...
with concurrent.futures.ThreadPoolExecutor(max_workers=min(len(items), MAX_WORKERS)) as executor:

Test plan

• Run workflows with parallel nodes and verify they still execute correctly
• Test with a large number of parallel items (>32) to verify they are properly throttled

🤖 Generated with Claude Code