Under the default raise_on_exception: true, gepa 0.1.4 does not stop when the reflection LM fails. This is the version DSPy 3.3.1 pins. It retries once per task, logs "did not propose a new candidate", proposes nothing that iteration, and continues. Imp's pinned profiles (:gepa_v0_1_4_merge, the default, and :beam_native) instead return {:error, {:optimizer_failed, ...}} for the same failure. Found while building #238, which leaves the default path as it was.
This is a parity divergence in the default configuration. A single transient reflection-model failure ends an Imp GEPA run that DSPy would have continued.
Proposed stance, to discuss: the pinned gepa profile does what gepa 0.1.4 does. A reflection-LM failure is logged and counted, using #238's report.errors and failed_proposals, and the iteration proposes nothing. raise_on_exception keeps gepa's meaning, which covers evaluation exceptions, not reflection failures. :beam_native should either match or record the difference and the reason for it.
Done when:
- A scripted reflection LM fails on one iteration and succeeds on the others. Imp's default profile then returns the same result as gepa 0.1.4 run on the same schedule, checked with the existing GEPA differential harness.
- The failure appears in the report.
- A reflection LM that always fails returns the seed with
status: :with_errors, as gepa does.
- The test fails on
main.
Under the default
raise_on_exception: true, gepa 0.1.4 does not stop when the reflection LM fails. This is the version DSPy 3.3.1 pins. It retries once per task, logs "did not propose a new candidate", proposes nothing that iteration, and continues. Imp's pinned profiles (:gepa_v0_1_4_merge, the default, and:beam_native) instead return{:error, {:optimizer_failed, ...}}for the same failure. Found while building #238, which leaves the default path as it was.This is a parity divergence in the default configuration. A single transient reflection-model failure ends an Imp GEPA run that DSPy would have continued.
Proposed stance, to discuss: the pinned gepa profile does what gepa 0.1.4 does. A reflection-LM failure is logged and counted, using #238's
report.errorsandfailed_proposals, and the iteration proposes nothing.raise_on_exceptionkeeps gepa's meaning, which covers evaluation exceptions, not reflection failures.:beam_nativeshould either match or record the difference and the reason for it.Done when:
status: :with_errors, as gepa does.main.