Defect
A paid DSPy RLM run can return findings that the Python bridge accepts and the TypeScript boundary silently drops.
The Python success prevents the bounded format-repair turn from running.
A consumer with requireStructuredFindings: true then fails after useful work completed.
Observed with @tangle-network/agent-eval@0.145.11 through Braid 0.1.2 and a retained Claude Code CLI Bridge run.
The model submitted four structured rows.
All four used free-form subjects such as What changed.
Python reported them as findings, while RawAnalystFindingSchema rejected every row.
Minimal reproduction
Python accepts any bounded non-empty subject in clients/python/src/agent_eval_rpc/dspy_rlm_bridge.py::_validate_finding:
row = {
"severity": "info",
"claim": "x",
"subject": "What changed",
"confidence": 0.9,
"evidence": [{"uri": "trace://run/span/span", "excerpt": "exact quote"}],
}
assert dspy_rlm_bridge._parse_findings_json(json.dumps([row])) == [row]
The published TypeScript parser rejects the same row:
import { parseRawFinding } from '@tangle-network/agent-eval/analyst'
parseRawFinding({
severity: 'info',
claim: 'x',
subject: 'What changed',
confidence: 0.9,
evidence: [{ uri: 'trace://run/span/span', excerpt: 'exact quote' }],
}) // null
parseFindingSubject('What changed') also returns null.
Production evidence
- Source trace:
run-721d5d04-e603-4a3d-b05b-2ab66e7e2682
- Analyst model calls: 3
- Model output: four structured findings plus a prose answer
- Consumer result: zero findings and
trace analyst 'question' returned no valid structured findings
- Provider usage: 6 input tokens, 2,921 output tokens, $0.6197 reported cost, 60,163 ms model latency
Required behavior
- Define one canonical raw-finding acceptance contract across the Python and TypeScript boundaries.
- Reject or repair every Python row that TypeScript would reject before Python reports success.
- Preserve per-row rejection reasons in the runtime result.
- Add a parity test with valid subjects, invalid subjects, unknown fields, bad evidence shapes, and non-finite confidence.
- Prove one malformed sibling does not erase valid siblings.
- Prove an all-invalid final submission triggers the one declared repair turn or fails with the exact row defects.
Store-dependent evidence resolution can remain in TypeScript.
The cross-language shape contract must not disagree before that stage.
Defect
A paid DSPy RLM run can return findings that the Python bridge accepts and the TypeScript boundary silently drops.
The Python success prevents the bounded format-repair turn from running.
A consumer with
requireStructuredFindings: truethen fails after useful work completed.Observed with
@tangle-network/agent-eval@0.145.11through Braid 0.1.2 and a retained Claude Code CLI Bridge run.The model submitted four structured rows.
All four used free-form subjects such as
What changed.Python reported them as findings, while
RawAnalystFindingSchemarejected every row.Minimal reproduction
Python accepts any bounded non-empty subject in
clients/python/src/agent_eval_rpc/dspy_rlm_bridge.py::_validate_finding:The published TypeScript parser rejects the same row:
parseFindingSubject('What changed')also returnsnull.Production evidence
run-721d5d04-e603-4a3d-b05b-2ab66e7e2682trace analyst 'question' returned no valid structured findingsRequired behavior
Store-dependent evidence resolution can remain in TypeScript.
The cross-language shape contract must not disagree before that stage.