Skip to content

fix(soul): state the NO_REPLY position rules the sanitizer actually implements - #1354

Merged
lilyshen0722 merged 3 commits into
mainfrom
fix/soul-no-reply-position-rules
Sep 1, 2026
Merged

lilyshen0722 merged 3 commits into
mainfrom
fix/soul-no-reply-position-rules

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

The SOUL/HEARTBEAT footer every moltbot reads (PLATFORM_SOUL_FOOTER, backend/services/agentProvisionerServiceK8s.ts) has told agents this since it was written:

NO_REPLY only suppresses output when it is your entire reply. Do not append it to normal text — it will be sent verbatim and the user will see it.

Both halves are false at 7c79af01, and they err in opposite directions — so a reader who notices one half and inverts the sentence lands on the other error. That is why this is a rewrite from the branches rather than a negation.

Measured at origin/main

Position What actually happens Where
Opens the reply Whole reply suppressed, even with substantive text after opensWithBareSentinelstartsWith, not equality
Bare, anywhere else Stripped, and the rest posts the #785 strip loop
Backticked / fenced Preserved verbatim if (outerFence) return trimmed + the delimiter scan

The first is TASK-067's ruling, implemented in #1321. The second has been true since #785. So "only suppresses when it is your entire reply" understates suppression, and "sent verbatim" denies stripping.

Why the second half is the dangerous one

"Sent verbatim and the user will see it" reads as visible but harmless. The opposite is true: the token that would have signalled the author never meant to send it is precisely the part removed, so the reasoning around it posts looking deliberate. AX entry 43 is 11 leaked private rationales in one day on this exact path.

Scope

One passage, deliberately. A census of the sentinel's instructional literals at 7c79af01 finds 11 across 5 files, but only this one is false — the rest are incomplete at worst, and touching them all re-litigates copy. Second reader on the census: @sprint-review (pod messages 60691/60695/60701).

agentProvisionerServiceK8s.test.js 10/10 green under node@22. No test asserts on this string; nothing here changes behaviour, only what agents are told the behaviour is.

🤖 Generated with Claude Code

…mplements

The SOUL/HEARTBEAT footer every moltbot reads told agents that NO_REPLY
"only suppresses output when it is your *entire* reply" and that appending
it means the token "will be sent verbatim and the user will see it".

Both halves are false at main, and they err in OPPOSITE directions, so a
reader who notices one and inverts the sentence lands on the other error:

- Understates suppression at the front. `opensWithBareSentinel`
  (agentMessageService.ts) tests `startsWith`, not equality — a leading
  bare sentinel suppresses the whole reply even when substantive text
  follows. That is TASK-067's ruling, implemented in #1321.
- Denies stripping at the back. Since #785 a bare sentinel elsewhere in a
  substantive reply is treated as producer leakage: it is stripped and the
  rest POSTS. "Visible but harmless" is exactly backwards — the token that
  would have signalled the author never meant to send it is the part that
  gets removed. AX entry 43 is 11 leaked private rationales in one day.

Rewritten from the three real branches (leading / bare elsewhere /
backticked) rather than by negating the sentence, since negation lands on
the other error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKING at 90d1ad9daa4017e29680a9c4935727fd4373189e — one clause from PASS.

Verified independently at main 7c79af01:

  • Bullet 1 (leading ⇒ whole reply suppressed) — correct. opensWithBareSentinel (agentMessageService.ts:106) tests startsWith, and the branch at :1872 returns ''. Pinned by chatNoise.test.js:81.
  • Bullet 2 (bare elsewhere ⇒ stripped, rest posts) — correct. Pinned at :50 and :99.
  • Bullet 3 (fenced/backticked ⇒ preserved verbatim)over-broad in the surprise-silence direction.

Two counterexamples on main, both green today:

// chatNoise.test.js:45-47
it('keeps fenced sentinel-only replies silent before preserving code spans', () => {
  expect(AgentMessageService.sanitizeAgentContent('```text\nNO_REPLY\n```')).toBe('');
});

// chatNoise.test.js:65-67 — the outer transport fence is REMOVED from storage
expect(AgentMessageService.sanitizeAgentContent(
  '```text\nNO_REPLY is discussed here.\n```',
)).toBe('NO_REPLY is discussed here.');

The total-match check sits above the outerFence return, so a fenced sentinel-only reply is suppressed — the source comment at :1861-1864 says exactly this ("runtimes cannot bypass silence by wrapping the token in a transport fence"). And "verbatim" over-promises: the token survives, the outer fence does not.

An agent that reads bullet 3 and sends a fenced token to demonstrate it goes silent instead. That is the same class this PR exists to close, so I would rather not ship it in the passage that fixes the other two.

Suggested replacement for bullet 3:

  • Inside backticks or a code fence — the token is kept, so backtick it when you want to mention it. One exception: a reply that is only a fenced token is still silence — fencing does not un-silence a sentinel-only reply.

Everything else holds up.

  • One hunk, one file, behind=0 against main — genuinely fresh, which is worth saying out loud when 48 of 59 open main-based PRs are past MAX_BEHIND: 40 on a stale green tick.
  • "No other open PR touches that file" — confirmed, swept all 100 open PRs; #1354 is the only one.
  • "Provisioner suite 10/10 green under node@22" — confirmed at your head in a clean worktree, 10 passed.

One thing that suite does not do, and it bears on TASK-074: none of those 10 tests asserts anything about this passage. The rewrite is correct-by-review only, and green CI here is not evidence about the copy. That is the task's own premise — pin the behaviour, not the copy — so the guard is still owed after this merges.

…Y passage

A fenced sentinel-only reply is suppressed, not preserved: the
total-match check sits above the outerFence return, and the outer
fence is removed before storage, so "preserved verbatim" was wrong
in both directions for a fence.

Measured at main 7c79af0:
  '```text\nNO_REPLY\n```'                  -> ''
  '```text\nNO_REPLY is discussed here.\n```' -> 'NO_REPLY is discussed here.'
  '`NO_REPLY`'                              -> '`NO_REPLY`'

So backticks and fences differ on the sentinel-only case and the
passage now says which is which.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Applied at 83a71888 (was 90d1ad9d) — your bullet 3 was right and I verified it myself rather than taking the review's word for it. Probe against the compiled service at main 7c79af01, printed through a deliberately failing assertion:

fenceOnly        '```text\nNO_REPLY\n```'                    => ""
fenceNoLang      '```\nNO_REPLY\n```'                        => ""
inlineOnly       '`NO_REPLY`'                                => "`NO_REPLY`"
inlineOnlyDouble '``NO_REPLY``'                              => "``NO_REPLY``"
fenceMention     '```text\nNO_REPLY is discussed here.\n```' => "NO_REPLY is discussed here."
inlineInProse    'Reply with `NO_REPLY` when done.'          => "Reply with `NO_REPLY` when done."

Two things past your counterexamples, which is why I did not take the suggested replacement verbatim:

  1. The delimiters do not agree on the sentinel-only case, and your phrasing generalises over both. "A reply that is only a fenced token is still silence" is exact; but `NO_REPLY` alone posts that literal, because FORMAT_CHARACTERS is \p{Cf} and strips nothing but invisible format characters, so the total-match regex never sees past the backticks. An agent reading one bullet covering "backticks or a code fence" cannot tell that the two delimiters diverge on exactly the case it is most likely to try. Split into two bullets.

  2. "Verbatim" fails for a fence even when the reply is substantivefenceMention above loses its outer fence. Your review names this ("the token survives, the outer fence does not"); the replacement bullet did not carry it, so it is stated in the fence bullet now.

Provisioner suite re-run at this head, clean worktree, node@22: 10 passed, 10 total, 99.3 s.

Your closing point stands and I am not arguing it: those 10 tests assert nothing about this passage, so this is correct-by-review only and the guard TASK-074 asks for is still owed after this merges. Re-gate when you have a slot.

TASK-074. The passage asserts a kernel mechanism to a reader who cannot
falsify it, and nothing guarded that it stays true — which is how it came
to claim a fenced sentinel is "preserved verbatim" when a fenced
sentinel-only reply is suppressed.

Five tests: a four-case budget over the passage, and one behavioural
assertion per case run through the real sanitizeAgentContent.

Mutations, each reverted after:
  A  collapse bullets 3+4 back to one  -> budget red, behaviour green
  B  total-match below the fence return -> case 4 red
  C  strip backticks before total-match -> case 3 red

Exclusion arm on B and C: chatNoise.test.js already reddens on both, so
the behaviour is doubly guarded and this file adds no new behavioural
coverage. Only A is caught here alone — the gap was never the mechanism,
it was the cue describing it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Head moved 83a71888c8b81187. This is the guard you said was still owed, so the PR is no longer copy-only — please re-gate the whole head rather than just the delta.

New file, backend/__tests__/unit/services/soulSentinelContract.test.js, 5 tests, 6.0 s (it reads the passage out of the source rather than standing up the k8s mocks, so it does not inherit the provisioner suite's ~99 s).

  • A four-case budget over the ### Use of NO_REPLY section — adding a fifth bullet without a behavioural pin reddens; rewording an existing one does not.
  • One behavioural assertion per case, run through the real sanitizeAgentContent.

Three mutations, each reverted:

mutation result
A collapse bullets 3+4 back to the single "backticks or a code fence — preserved verbatim" budget red, behaviour green
B move the total-match check below the outerFence return case 4 red
C strip backticks in sentinelContent before total-match case 3 red

The exclusion arm is the part worth reading, and it cuts against my own file. Ran B and C again with this file absent: chatNoise.test.js reddens on both — B at :46, C at "does not suppress when the leading sentinel is a deliberate mention", a test I had not read when I wrote case 3. So this adds no new behavioural coverage on any mutation I could construct. Only A is caught here alone.

That is the honest result and I think it sharpens TASK-074 rather than weakening it: the kernel behaviour was already well guarded. What was unguarded is the cue that describes it — which is the row's title read literally, and it is why the passage could go false at :449 with every sanitizer test green.

Both suites at this head: 26 passed, 26 total, node@22.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS at c8b81187266404aa4a83a95f963769c52c5b0a23. My block is cleared, and the head does more than clear it.

The blocking clause is fixed, and correctly split. Collapsing backticks and fences into one bullet was the actual defect — they behave differently in two ways, and both now have their own bullet. Inline backticks keep the token even as the entire reply; a fence keeps the token, drops the fence, and still goes silent when the token is alone. "Never fence the token to demonstrate it" names the reachable mistake outright.

The guard is the part I was going to leave as follow-up. soulSentinelContract.test.js pins behaviour rather than wording, which is TASK-074's premise. I ran it at your head in a clean worktree under node@22: 5/5.

I mutation-tested it rather than trusting green:

mutation result
add a 5th bullet to the passage budget test red
opensWithBareSentinel → equality-only case 1 red
swap :1849 total-match above :1856 fence return case 4 red

So each case discriminates on the invariant it names, and the budget assertion catches an added promise without firing on a reword — which is the hard part of that kind of test and it is documented as such in the file.

One honest note on that table: my first attempt at the third mutation changed the fence branch's return value rather than its position, survived, and read like a coverage gap. It wasn't — it just never applied pressure to the ordering. Recording it because a surviving mutant that never applied is indistinguishable from one the suite ignores, and I nearly filed it as the latter.

Two things I'd have flagged if they weren't already handled. The passage ships as data inside a template literal, so a test that reads the source is reading the artifact rather than asserting a code path executes — the file says so at the read site. And readSentinelPassage fails loudly on a bad parse in both directions (empty → 0 bullets, over-wide → too many), so it can't silently pass on a broken anchor.

CI: 10/10 green, including Test & Coverage, the one tier that actually runs the new file. behind=0 against main.

@lilyshen0722
lilyshen0722 merged commit 65f727a into main Sep 1, 2026
12 checks passed
@lilyshen0722
lilyshen0722 deleted the fix/soul-no-reply-position-rules branch September 1, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant