Skip to content

fix(dspy-rlm): generic findings repair uses codetrace empty-result rule #579

Description

@drewstone

agent-eval 0.144.11 with agent-eval-rpc 0.144.11 can discard a valid generic trace answer.

Reproduction:

  1. Run the DSPy RLM bridge with operation=analyze, controlAdapter=two-step, and generic analyst instructions.
  2. Ask for one cited fact from a successful trace.
  3. Use glm-5.2 through an OpenAI-compatible endpoint.
  4. The controller returns a concrete answer with exact trace and span identifiers.
  5. The bridge returns findings=[] and createTraceAnalyst fails requireStructuredFindings with returned no valid structured findings.

The generic fallback in agent_eval_rpc/dspy_rlm_bridge.py uses _FINDINGS_REPAIR_PROMPT. That prompt says Return [] if the answer reports no incorrect steps. This is a codetrace-specific rule, but the generic question path also uses it. A successful factual answer naturally reports no incorrect steps, so the repair call erases its cited finding.

Expected:

The generic repair prompt must request the strict findings_json array for the answer and return [] only when the answer has no citable finding. The incorrect-steps rule must stay limited to the codetrace contract.

Observed in a clean @tangle-network/braid 0.1.1 packed install. The model answer cited the exact braid.run.finished and braid.run.requested spans and included the requested output marker, but the final findings array was empty.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions