Skip to content

fix(analyst): keep Python and TypeScript finding validation identical #606

Description

@drewstone

Defect

A paid DSPy RLM run can return findings that the Python bridge accepts and the TypeScript boundary silently drops.
The Python success prevents the bounded format-repair turn from running.
A consumer with requireStructuredFindings: true then fails after useful work completed.

Observed with @tangle-network/agent-eval@0.145.11 through Braid 0.1.2 and a retained Claude Code CLI Bridge run.
The model submitted four structured rows.
All four used free-form subjects such as What changed.
Python reported them as findings, while RawAnalystFindingSchema rejected every row.

Minimal reproduction

Python accepts any bounded non-empty subject in clients/python/src/agent_eval_rpc/dspy_rlm_bridge.py::_validate_finding:

row = {
    "severity": "info",
    "claim": "x",
    "subject": "What changed",
    "confidence": 0.9,
    "evidence": [{"uri": "trace://run/span/span", "excerpt": "exact quote"}],
}
assert dspy_rlm_bridge._parse_findings_json(json.dumps([row])) == [row]

The published TypeScript parser rejects the same row:

import { parseRawFinding } from '@tangle-network/agent-eval/analyst'

parseRawFinding({
  severity: 'info',
  claim: 'x',
  subject: 'What changed',
  confidence: 0.9,
  evidence: [{ uri: 'trace://run/span/span', excerpt: 'exact quote' }],
}) // null

parseFindingSubject('What changed') also returns null.

Production evidence

  • Source trace: run-721d5d04-e603-4a3d-b05b-2ab66e7e2682
  • Analyst model calls: 3
  • Model output: four structured findings plus a prose answer
  • Consumer result: zero findings and trace analyst 'question' returned no valid structured findings
  • Provider usage: 6 input tokens, 2,921 output tokens, $0.6197 reported cost, 60,163 ms model latency

Required behavior

  • Define one canonical raw-finding acceptance contract across the Python and TypeScript boundaries.
  • Reject or repair every Python row that TypeScript would reject before Python reports success.
  • Preserve per-row rejection reasons in the runtime result.
  • Add a parity test with valid subjects, invalid subjects, unknown fields, bad evidence shapes, and non-finite confidence.
  • Prove one malformed sibling does not erase valid siblings.
  • Prove an all-invalid final submission triggers the one declared repair turn or fails with the exact row defects.

Store-dependent evidence resolution can remain in TypeScript.
The cross-language shape contract must not disagree before that stage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions