Repository navigation
council: eval harness (E0) and JSON-escaped tag spellings - #44
Merged
Merged
Conversation
Most tool results reach the model as JSON text, where a "<" can be \u003c (Go's json.Marshal) and the controls and separators a tag hides behind are escapes (\b \f \n \r \t, \u0000-\u001f, \u2028, \u2029). untrustedTag now takes \u003c as an opener, those escapes in the gap and inside the name (inside the name at any depth, so a device's own JSON one level down is covered for them), and the braille blank U+2800 in the gap. The escape set stays narrow: a general \uXXXX in the gap would swallow an escaped opener. The comments record the limit: escapes a device writes itself one level down (\\u003c\\u002f, \\uff1c) are not caught; the prompts' rule is the boundary, and the plan lists the hardening as a later-stage item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…form (E0) TestEval drives council turns through the real tools.Set with a scripted model and a fixture MCP platform that answers with the production tool names and indentation. It pins Stage 0's guarantees: - cut results: the RESULT PARTIAL header's counts and offset, and the read_result page that follows; - injection: one <untrusted> block per result, every tag of the corpus (testdata/eval/injection.json) escaped at its opener only, as plain JSON, as json.Marshal JSON with \u003c, and as plain log text; hidden runes kept as the tool sent them; a level refusal; read never offered inspect_*; - the escape stays linear on large input; - the round budget (a forced answer ends Fleet's loop) and the token budget (the phase line, and the answer call past the cap). A test workflow runs on push and pull requests: the Go toolchain floor, gofmt, vet, a cross-build of one target per release OS plus linux/arm, the race tests, and the templates module. cmd/domain.go gets its imports sorted for the gofmt step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits, reviewed separately.
council/tools: escape the block's tags in their JSON-escaped spellings
Most tool results reach the model as JSON text.
untrustedTagnow also takes<as an opener, the JSON escapes of C0 controls and of U+2028/U+2029 in the gap and inside the name (inside the name at any depth), and U+2800 in the gap. The escape set stays narrow on purpose: a general\uXXXXin the gap would swallow an escaped opener (<</untrusted>must escape both). The limit is documented and pinned inTestUntrustedLookAlikes: escapes a device writes itself one level down (\\u003c\\u002f,\\uff1c) are not caught; the prompts' rule is the boundary, and the hardening is a later-stage plan item.council: an eval harness that runs whole turns against a fixture platform (E0)
TestEvaldrives council turns through the realtools.Setwith a scripted model and a fixture MCP platform. Scenarios: cut results (header counts/offset,read_resultpage), injection (an ASCII corpus intestdata/eval/injection.json, delivered as JSON, asjson.MarshalJSON and as plain logs; one block per result; opener-only escape; hidden runes kept; level refusal; read never offeredinspect_*), linear escaping, and the round and token budgets. A newtest.ymlworkflow runs the toolchain floor (1.27.4+), gofmt, vet, a cross-build per release OS plus linux/arm, the race tests and the templates module.Verified on the dev host
go test -race -count=3 -run TestEval ./pkg/council/,go test -race ./pkg/council/...,./pkg/resultcut/...,./cmd/..., the templates module (vet + race tests): all ok.go vet ./...,gofmt -l(clean),golangci-lint --new-from-rev(0 issues),actionlint, the cross-build loop (windows/amd64, windows/arm64, darwin/arm64, linux/arm).\b/\fcases) each fail the test that targets them.🤖 Generated with Claude Code