fix(evaluator): stop timed-out evaluations from hanging the worker - #485
Open
Ghosts6 wants to merge 1 commit into
Open
fix(evaluator): stop timed-out evaluations from hanging the worker#485Ghosts6 wants to merge 1 commit into
Ghosts6 wants to merge 1 commit into
Conversation
asyncio.wait_for(loop.run_in_executor(None, ...)) only stops waiting on timeout; the blocking evaluate/evaluate_stageN call keeps running in the asyncio loop's default executor thread, since Python cannot force-kill a running thread. process_parallel.py calls asyncio.run() once per iteration, and asyncio.run()'s cleanup blocks in shutdown_default_executor() until every thread in that default executor finishes - so one timed-out evaluation (e.g. an evolved program with an infinite loop) hangs the entire worker process, matching issue algorithmicsuperintelligence#399. Give Evaluator its own dedicated ThreadPoolExecutor and pass it explicitly to every run_in_executor call instead of None. It is never the loop's default executor, so asyncio.run() no longer waits on it and a timed-out call's orphaned thread can't block cleanup. Resolves algorithmicsuperintelligence#399
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a hang where, once a program evaluation times out, the worker process never comes back matching #399 ("Hang after evaluation timeout").
Root Cause
Each cascade/direct evaluation stage runs via
asyncio.wait_for(loop.run_in_executor(None, ...), timeout=...). On timeout,wait_foronly stops waiting the blockingevaluate/evaluate_stageNcall keeps executing in the loop's default executor thread, since Python cannot force-kill a running thread.process_parallel.pycallsasyncio.run(...)once per iteration, andasyncio.run()'s cleanup blocks insideshutdown_default_executor()until every thread in that default executor finishes. So a single timed-out evaluation (e.g. an evolved program with an infinite loop) hangs the entire worker process, even though the timeout itself already returned a result.Fix
openevolve/evaluator.py:Evaluatornow creates its own dedicatedThreadPoolExecutorand passes it explicitly to all fourrun_in_executorcalls (direct evaluate + stage1/2/3), instead ofNone.asyncio.run()no longer waits on it, so a timed-out call's orphaned thread can't block cleanup.Testing
python -m unittest discover tests— 431 passed (was 430; total runtime also dropped 65.5s → 31.5s, since other timeout tests were silently blocked on the same issue).test_timeout_does_not_hang_process_on_cleanupintests/test_evaluator_timeout.py, which measures wall-clock time aroundasyncio.run()itself (mirroring howprocess_parallel.pyinvokes the evaluator). Confirmed it fails (~8s) on the unpatched code and passes (~3s) with the fix.black --checkclean on both changed files.