agent-eval 0.144.11 with agent-eval-rpc 0.144.11 can discard a valid generic trace answer.
Reproduction:
- Run the DSPy RLM bridge with operation=analyze, controlAdapter=two-step, and generic analyst instructions.
- Ask for one cited fact from a successful trace.
- Use glm-5.2 through an OpenAI-compatible endpoint.
- The controller returns a concrete answer with exact trace and span identifiers.
- The bridge returns findings=[] and createTraceAnalyst fails requireStructuredFindings with returned no valid structured findings.
The generic fallback in agent_eval_rpc/dspy_rlm_bridge.py uses _FINDINGS_REPAIR_PROMPT. That prompt says Return [] if the answer reports no incorrect steps. This is a codetrace-specific rule, but the generic question path also uses it. A successful factual answer naturally reports no incorrect steps, so the repair call erases its cited finding.
Expected:
The generic repair prompt must request the strict findings_json array for the answer and return [] only when the answer has no citable finding. The incorrect-steps rule must stay limited to the codetrace contract.
Observed in a clean @tangle-network/braid 0.1.1 packed install. The model answer cited the exact braid.run.finished and braid.run.requested spans and included the requested output marker, but the final findings array was empty.
agent-eval 0.144.11 with agent-eval-rpc 0.144.11 can discard a valid generic trace answer.
Reproduction:
The generic fallback in agent_eval_rpc/dspy_rlm_bridge.py uses _FINDINGS_REPAIR_PROMPT. That prompt says Return [] if the answer reports no incorrect steps. This is a codetrace-specific rule, but the generic question path also uses it. A successful factual answer naturally reports no incorrect steps, so the repair call erases its cited finding.
Expected:
The generic repair prompt must request the strict findings_json array for the answer and return [] only when the answer has no citable finding. The incorrect-steps rule must stay limited to the codetrace contract.
Observed in a clean @tangle-network/braid 0.1.1 packed install. The model answer cited the exact braid.run.finished and braid.run.requested spans and included the requested output marker, but the final findings array was empty.