Skip to content

docs(evidence): record the shim live-proof and the GEPA machinery certification - #667

Merged
drewstone merged 1 commit into
mainfrom
docs/evidence-integration-arc
Aug 20, 2026
Merged

docs(evidence): record the shim live-proof and the GEPA machinery certification#667
drewstone merged 1 commit into
mainfrom
docs/evidence-integration-arc

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Two evidence records for the agent-engine integration arc, in the #644 registry pattern.

agent-engine-shim-live-proof — MEASURED-ONCE

The Anthropic loopback shim serves an unmodified Claude Code CLI driving a GEPA autoresearch optimization on the published packages, fully metered.

  • Instrument: npm @tangle-network/agent-eval@0.150.2 + PyPI agent-eval-rpc==0.150.2, gepa 0.1.4@f919db0a, deepseek-v4-flash via router.tangle.tools.
  • Result: 8/8 anthropic-wire requests admitted and completed, zero 400s, totalCostUsd 0.029778336, accountingComplete true, strippedFields 'context_management,output_config,thinking' on all 8 receipts, 12 evaluations, baseline 0.286 -> winner 1.0 on the deterministic toy objective, wall 39.7s.
  • Confounds: toy objective (the lift number is not a benchmark claim; the metered-path claim is the claim); single run.

gepa-bridge-machinery-certification — MEASURED-ONCE, negative disclosed

The GEPA bridge machinery is certified; the lift verdict is NOT measured.

  • Certified: GEPA consumed its full 150-evaluation budget, generated 22 proposals, rejected measured-worse candidates, and dual-count integrity held. Every missing held-out score was a refusal by the fail-loud comparison, never a fabricated zero.
  • Not measured: two AIME-2025 held-out items drive glm-5.3 past the 480s dispatch wall or the 16000-token family cap, and the comparison refuses partial verdicts by design. 14 launch attempts; confounds recorded in the record.

Checks

  • pnpm run evidence:render regenerated evidence/INDEX.md (9 records).
  • pnpm run evidence:check: index fresh.
  • pnpm vitest run src/experiment/evidence-record.test.ts: 15/15 passed.
  • node scripts/check-skill.mjs: valid; no digest repin needed.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 203e7c85

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

tangletools · auto-approval · reason: drewstone_author · 2026-08-20T10:37:32Z

@drewstone

Copy link
Copy Markdown
Contributor Author

Merging with --admin, disclosed: both checks green; records schema-validated, index regenerated, every number verified against the raw proof logs before recording; operator directive.

@drewstone
drewstone merged commit b1bcc11 into main Aug 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants