test(dynamic-workflow): mechanical claims table and reflection formation example - #45
Conversation
|
Related: #47 (zh/en term-consistency audit) — same spirit as the claims table here, extending mechanical checks from behavior to documentation. No action needed on this PR. |
a950f42 to
8d4a56e
Compare
hetaoBackend
left a comment
There was a problem hiding this comment.
Request changes for exact current head 8d4a56e.
Blocking issues:
- The PR changes the maintained
plugins/hetaoBackend/mcode-dynamic-workflows/tree, while the PR author/commits aremodacker/moc <moc@sgmov.com>and the manifests still identifyhetaoBackendas owner (plugin.json:6,.claude-plugin/plugin.json:5). UnderCONTRIBUTING.md:28,80, provide explicit owner authorization or move the change to the contributor's own plugin scope before merge. scripts/verify-claims.mjs:10-17extracts Markdown cells and passes each cell directly to/bin/sh -c. This turns a modified verification document into arbitrary local command execution, with no allowlist for commands/argv or shell metacharacters. Replace it with structured, fixed commands/argv (or an explicit documented trusted-only policy plus strict validation); do not use an unrestricted shell for a claims verifier.- The parser only collects rows matching one regex (
verify-claims.mjs:12) and silently ignores malformed/unexpected/duplicate rows. It can exit 0 after verifying a subset while claiming “every row”. Validate the header, exact expected IDs/order, uniqueness, columns, and that every table row is consumed; add focused parser tests. - The new verifier and
examples/reflection.jsare not executed by the current CI workflow. The green checks cover the existing 73 source checks and package test, but notnode scripts/verify-claims.mjsorvalidateScripton the reflection example. Add both checks to CI and provide exact-head output. examples/reflection.js:12-15gives the revision agent the draft and critique but not the originaltask/materialpassed to the critic. The runtime contract says agents are self-contained and do not inherit the parent conversation; the revision cannot independently verify the critique evidence. Pass the original material (or a bounded equivalent) to revision and add a regression assertion.
The current [code]smith check is SKIPPED and is not evidence. Reconcile the ownership issue and add the missing executable CI evidence before approval.
- VERIFICATION.md gains a verify-fenced claims table; the new scripts/verify-claims.mjs executes each row and exits 0 only when all match (1 first mismatch, 2 tool failure) - examples/reflection.js: draft -> independent critique -> revision formation with a checkpoint snapshot, styled after audit.js
8d4a56e to
b3b8acc
Compare
…elated to this PR)
|
Note on the failing |
|
Thank you for the exact-head review — all five blockers are actionable. Point-by-point:
Rework incoming on this branch; will re-request review against the exact new head. |
|
Follow-up review of current head
Validation boundary: the reflection probe checks orchestration and delivered inputs, not real-model output quality. The current public CI failure in run 35304360094 is in CLI Agent Bridge; I am not attributing it to this PR. Clarification of the earlier review: external contributor authorship is not itself a blocker, and executing explicitly invoked, trusted repository verification commands through a shell is not by itself a demonstrated security vulnerability. The POSIX-only dependency should nevertheless be disclosed in |
|
Thank you for the follow-up and the two clarifications. Rework pushed at exact head
Ownership: understood and noted for future submissions — thank you for the explicit clarification that external authorship is not itself a blocker. Full suite at this head: 82/82 plugin checks, packaged MCP smoke 1/1, |
|
Additional evidence at exact head |
…nput contract - scripts/verify-claims.mjs: claims are fixed argv data spawned directly (no shell, node resolved to process.execPath); the markdown table is a human-readable mirror, never an execution source - checks/claims.check.mjs: mirror equals the executable claims exactly (header, order, uniqueness, columns, full consumption); six negative parser tests prove malformed/duplicate/smuggled/reordered rows fail loudly instead of being silently skipped - examples/reflection.js: critique and revision agents each receive the original task and material (agents are self-contained; the revision can independently verify critique evidence against the source) - checks/examples.check.mjs: end-to-end input-contract regression through the real Engine (deterministic executor, no model calls); both new checks run in the standard suite, so CI covers them via npm test - VERIFICATION.md: mirror semantics + POSIX/macOS portability note
bdf7717 to
e931a3a
Compare
|
Self-audit correction: exact head is now |
…s-tree flake, evidence on MiniMax-AI#43)
|
For the record on the |
hetaoBackend
left a comment
There was a problem hiding this comment.
Re-reviewed at ee3964c. The previous findings are resolved: verification uses fixed argv with strict mirror validation; critique and revise receive the original task/material; platform limitations are documented. Focused claims/example tests pass (9/9), and the latest-head CI checks are green. Approved for merge.
What changes
VERIFICATION.md+scripts/verify-claims.mjs): the claims a machine can re-check are tabulated in a```verifyfenced block. The runner executes each row via/bin/sh -c, prints one PASS/FAIL line per claim, and exits 0 only when every row matches its expected exit status (1 on the first mismatch, 2 when the tool itself cannot run). Rows needing development dependencies say so (npm cifirst); non-mechanical claims (dashboard acceptance, real-agent calls, platform coverage) intentionally stay prose.examples/reflection.js): draft → independent critique → revision, three phases with separate agents, a checkpoint snapshot of the draft, and explicit open-issue handling — styled afterexamples/audit.js.Verification
node scripts/verify-claims.mjs: 4/4 PASS on a committed tree. V-04 failing on a dirty tree is the feature: it detects drifted committed assets.examples/reflection.jspassesvalidateScriptnpm run check: greenNeed help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.