Skip to content

council: eval harness (E0) and JSON-escaped tag spellings - #44

Merged
tekn0ir merged 2 commits into
mainfrom
council-eval-harness
Nov 14, 2026
Merged

tekn0ir merged 2 commits into
mainfrom
council-eval-harness

Conversation

@tekn0ir

@tekn0ir tekn0ir commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Two commits, reviewed separately.

council/tools: escape the block's tags in their JSON-escaped spellings
Most tool results reach the model as JSON text. untrustedTag now also takes < as an opener, the JSON escapes of C0 controls and of U+2028/U+2029 in the gap and inside the name (inside the name at any depth), and U+2800 in the gap. The escape set stays narrow on purpose: a general \uXXXX in the gap would swallow an escaped opener (<</untrusted> must escape both). The limit is documented and pinned in TestUntrustedLookAlikes: escapes a device writes itself one level down (\\u003c\\u002f, \\uff1c) are not caught; the prompts' rule is the boundary, and the hardening is a later-stage plan item.

council: an eval harness that runs whole turns against a fixture platform (E0)
TestEval drives council turns through the real tools.Set with a scripted model and a fixture MCP platform. Scenarios: cut results (header counts/offset, read_result page), injection (an ASCII corpus in testdata/eval/injection.json, delivered as JSON, as json.Marshal JSON and as plain logs; one block per result; opener-only escape; hidden runes kept; level refusal; read never offered inspect_*), linear escaping, and the round and token budgets. A new test.yml workflow runs the toolchain floor (1.27.4+), gofmt, vet, a cross-build per release OS plus linux/arm, the race tests and the templates module.

Verified on the dev host

  • go test -race -count=3 -run TestEval ./pkg/council/, go test -race ./pkg/council/..., ./pkg/resultcut/..., ./cmd/..., the templates module (vet + race tests): all ok.
  • go vet ./..., gofmt -l (clean), golangci-lint --new-from-rev (0 issues), actionlint, the cross-build loop (windows/amd64, windows/arm64, darwin/arm64, linux/arm).
  • 27 mutations (escape branches, the narrow escape set, the cut header, the level check, the tool-set leaks, the budgets, the quadratic escape, and the new depth/\b/\f cases) each fail the test that targets them.
  • An Opus xhigh adversarial review of the tools change, with two skeptics per finding; its two test gaps and two comment fixes are in.

🤖 Generated with Claude Code

tekn0ir and others added 2 commits November 13, 2026 23:48
Most tool results reach the model as JSON text, where a "<" can be \u003c
(Go's json.Marshal) and the controls and separators a tag hides behind are
escapes (\b \f \n \r \t, \u0000-\u001f, \u2028, \u2029). untrustedTag now
takes \u003c as an opener, those escapes in the gap and inside the name
(inside the name at any depth, so a device's own JSON one level down is
covered for them), and the braille blank U+2800 in the gap. The escape set
stays narrow: a general \uXXXX in the gap would swallow an escaped opener.

The comments record the limit: escapes a device writes itself one level
down (\\u003c\\u002f, \\uff1c) are not caught; the prompts' rule is the
boundary, and the plan lists the hardening as a later-stage item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…form (E0)

TestEval drives council turns through the real tools.Set with a scripted
model and a fixture MCP platform that answers with the production tool
names and indentation. It pins Stage 0's guarantees:
- cut results: the RESULT PARTIAL header's counts and offset, and the
  read_result page that follows;
- injection: one <untrusted> block per result, every tag of the corpus
  (testdata/eval/injection.json) escaped at its opener only, as plain JSON,
  as json.Marshal JSON with \u003c, and as plain log text; hidden runes kept
  as the tool sent them; a level refusal; read never offered inspect_*;
- the escape stays linear on large input;
- the round budget (a forced answer ends Fleet's loop) and the token budget
  (the phase line, and the answer call past the cap).

A test workflow runs on push and pull requests: the Go toolchain floor,
gofmt, vet, a cross-build of one target per release OS plus linux/arm, the
race tests, and the templates module. cmd/domain.go gets its imports
sorted for the gofmt step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir
tekn0ir merged commit 8c20f02 into main Nov 14, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant