Skip to content

fix(evaluator): stop timed-out evaluations from hanging the worker - #485

Open
Ghosts6 wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
Ghosts6:fix/399-cascade-eval-timeout-hang
Open

fix(evaluator): stop timed-out evaluations from hanging the worker#485
Ghosts6 wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
Ghosts6:fix/399-cascade-eval-timeout-hang

Conversation

@Ghosts6

@Ghosts6 Ghosts6 commented Aug 26, 2026

Copy link
Copy Markdown

Fixes a hang where, once a program evaluation times out, the worker process never comes back matching #399 ("Hang after evaluation timeout").

Root Cause

Each cascade/direct evaluation stage runs via asyncio.wait_for(loop.run_in_executor(None, ...), timeout=...). On timeout, wait_for only stops waiting the blocking evaluate/evaluate_stageN call keeps executing in the loop's default executor thread, since Python cannot force-kill a running thread.

process_parallel.py calls asyncio.run(...) once per iteration, and asyncio.run()'s cleanup blocks inside shutdown_default_executor() until every thread in that default executor finishes. So a single timed-out evaluation (e.g. an evolved program with an infinite loop) hangs the entire worker process, even though the timeout itself already returned a result.

Fix

openevolve/evaluator.py:

  • Evaluator now creates its own dedicated ThreadPoolExecutor and passes it explicitly to all four run_in_executor calls (direct evaluate + stage1/2/3), instead of None.
  • Since it's never the loop's default executor, asyncio.run() no longer waits on it, so a timed-out call's orphaned thread can't block cleanup.

Testing

  • python -m unittest discover tests — 431 passed (was 430; total runtime also dropped 65.5s → 31.5s, since other timeout tests were silently blocked on the same issue).
  • Added test_timeout_does_not_hang_process_on_cleanup in tests/test_evaluator_timeout.py, which measures wall-clock time around asyncio.run() itself (mirroring how process_parallel.py invokes the evaluator). Confirmed it fails (~8s) on the unpatched code and passes (~3s) with the fix.
  • black --check clean on both changed files.

asyncio.wait_for(loop.run_in_executor(None, ...)) only stops waiting on
timeout; the blocking evaluate/evaluate_stageN call keeps running in the
asyncio loop's default executor thread, since Python cannot force-kill a
running thread. process_parallel.py calls asyncio.run() once per
iteration, and asyncio.run()'s cleanup blocks in
shutdown_default_executor() until every thread in that default executor
finishes - so one timed-out evaluation (e.g. an evolved program with an
infinite loop) hangs the entire worker process, matching issue algorithmicsuperintelligence#399.

Give Evaluator its own dedicated ThreadPoolExecutor and pass it
explicitly to every run_in_executor call instead of None. It is never
the loop's default executor, so asyncio.run() no longer waits on it and
a timed-out call's orphaned thread can't block cleanup.

Resolves algorithmicsuperintelligence#399
@CLAassistant

CLAassistant commented Aug 26, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants