mutation: a Mull survivor is not evidence of a missing assertion — four sampled survivors all turn the suite red when applied by hand

[Yaraslaut]

scripts/mutation.sh (morph#405) reports 352 surviving mutants over
include/morph, and morph#431 and morph#432 are written against that list.
The list cannot be read site-by-site. Four survivors, sampled from three
different headers, were applied to the source by hand and every one of them
turns morph_tests red.

Verification status: measured

lane/d-framework at 18c764bc, in the same build tree
(build/mutation-core-forms) and the same configuration the run itself used —
Mull 0.34.0 / mull-ir-frontend-22 / LLVM 22.1.2, clang 22.1.8, -O0 -g,
cxx_all. For each site the mutation Mull describes (cxx_remove_void_call:
delete the call) was applied to the header, the tree rebuilt, and
./tests/morph_tests run. A survivor means the suite passes on the mutant, so
each of these should have stayed green.

site	Mull	hand-applied
core/remote.hpp:718 + :818 — reply(wire::encode(makeOk(...)))	Survived	34 of 1303 cases fail
forms/forms.hpp:1981 — emitDeclaredBound(property, "minimum", *meta.minimum)	Survived	2 cases fail
core/registry.hpp:709 — holder->attachIdentity(std::string{primary})	Survived	2 cases fail

Real output, the two remote.hpp reply calls deleted:

/home/yaraslau/repo/morph-wt/340/tests/test_coverage_push95.cpp:572: FAILED:
  {Unknown expression after the reported line}
due to unexpected exception with message:
  envelope decode failed: index 0: no_read_input

/home/yaraslau/repo/morph-wt/340/tests/test_remote_connection_scope.cpp:389: FAILED:
  REQUIRE( regA.await() )
with expansion:
  false

test cases:  1303 |  1269 passed | 34 failed
assertions: 21378 | 21343 passed | 35 failed

forms.hpp's and registry.hpp's, in one build together:

/home/yaraslau/repo/morph-wt/340/tests/test_forms_field_bounds.cpp:104: FAILED:
/home/yaraslau/repo/morph-wt/340/tests/test_forms_field_bounds.cpp:148: FAILED:
/home/yaraslau/repo/morph-wt/340/tests/test_registry_extra.cpp:237: FAILED:
/home/yaraslau/repo/morph-wt/340/tests/test_remote_extra.cpp:674: FAILED:

test cases:  1303 |  1299 passed | 4 failed

The clean tree at the same commit is green: All tests passed (21523 assertions in 1303 test cases).

What this costs the two tickets written against the list

morph#431's headline is "twenty places where the framework's response path can
be deleted and morph_tests stays green". It cannot: deleting two of them turns
34 cases red. The premise of that section is false, and it was the section the
ticket said to lead with.

It was tested the long way round first. tests/test_remote_reply_envelopes.cpp
(added on lane/d-framework, 20 cases) reads the full reply envelope for every
control kind dispatchMessage switches on, including both of these sites. The
re-run scored 64.76% and those survivors did not move — which is what
prompted applying the mutation by hand.

Mechanism — inferred, not verified

Every affected site is in a header compiled into many of morph_tests' 105
translation units. Each object carries its own copy of the function and the
linker keeps one. Mull embeds mutants per object (the .mull_mutants section)
and the runner enables one by id; if the copy carrying the enabled mutant is
one the linker discarded, the running program is unmutated and the mutant
scores as a survivor.

That predicts the effect is specific to header-defined code in a multi-TU
binary, which matches where the survivors sit: remote.hpp (126) and
bridge.hpp (85) are 211 of the 352, and cxx_remove_void_call is 148 of
them. It has not been confirmed against Mull's implementation — someone
should read mull-ir-frontend's emission and the runner's patching before this
is treated as understood.

Direction of the error

scripts/mutation.sh's header records a caveat in the other direction —
test_outbox.cpp names a scratch file after a stack address, so concurrent
copies can collide, a collision fails a case, and a failed case reads as a
kill. That remains true and still says the score can be too high.

This one says the score is also too low, and by a larger amount: every
false survivor is a mutant that should have counted as killed. The two are not
in tension — they are different mechanisms — but the second is the one that
decides whether 352 survivors is a backlog or an artefact.

What would change the verdict

• A sampled survivor that leaves the suite green when hand-applied would be a
  real gap, and would bound how much of the list is affected. Four out of four
  false positives does not prove all 352 are.
• Reproducing the run with include/morph compiled into a single translation
  unit (a one-TU test harness) would isolate the multi-TU hypothesis directly.
• If Mull has an option for this — or if a newer release fixes it — the fix is
  a flag in scripts/mutation.sh, not new tests.

Until then, scripts/mutation_survivors.json records the finding next to the
two runs, and morph#431 and morph#432 should be re-scoped: their boundary-shape
findings held up and are closed, their reply(...) shape did not.

🤖 Generated with Claude Code

https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb

[Yaraslaut]

Triaged: triage: valid.

Independently reproduced the central claim on this machine, on origin/master
(not lane/d-framework, and on Windows/clang-cl rather than Linux — a
different toolchain from the one the issue measured on, which is itself useful
corroboration that this isn't a compiler-specific artifact):

• Configured clangcl-debug, built morph_tests. Clean baseline:
  all tests passed, 21517 assertions in 1302 test cases.
• Hand-deleted exactly the two calls the issue names —
  reply(::morph::wire::encode(::morph::wire::makeOk(env.callId, {}, mid.v)))
  at remote.hpp:718 and its fresh-instance sibling at :818 — the same
  cxx_remove_void_call mutation Mull describes, applied by hand, matching the
  issue's own method.
• Rebuilt (compiles clean — nothing depends on the call's return, as expected
  for a void/discarded-value call). Reran the suite:
  34 of 1302 cases fail, 35 of 21371 assertions fail. One of the failures
  reproduces the exact signature the issue quotes —
  envelope decode failed: index 0: no_read_input — surfacing on
  test_shared_instances.cpp:846 here vs. test_coverage_push95.cpp:572 on
  lane/d-framework (expected: different commit, different test file layout,
  same underlying mechanism — the client-side envelope decode has nothing to
  decode because the server never replied).
• git checkout -- include/morph/core/remote.hpp cleanly reverted the
  mutation; confirmed git status --porcelain empty before rebuilding;
  rebuilt and the tree is green again.

This directly falsifies reading scripts/mutation_survivors.json's 352
survivors site-by-site as a list of missing assertions: two of the most-cited
sites are asserted, and deleting them breaks a substantial, specific slice of
the suite. The issue's own caveat — mechanism "inferred, not verified" — I
did not attempt to independently confirm (would need reading Mull's
mull-ir-frontend/runner internals or a single-TU harness, neither of which
is a quick check), so that part stays exactly as the issue frames it:
plausible and unconfirmed, not re-verified here.

Cross-checked against the tracker: scripts/mutation_survivors.json's
false_positive_finding section already records this same investigation
(dated today), and morph#431 already carries a self-correction comment
withdrawing its "twenty places the reply can be deleted" headline and marking
itself blocked on this issue. That's consistent with what I found directly,
not just an appeal to another artifact saying the same thing.

Verdict rationale: triage: valid, not rescope — the issue's own framing is
already correct and appropriately scoped. It states the finding precisely
("a survivor is not evidence of a missing assertion" — not "the score is
wrong" or "delete the tool"), separates the confirmed empirical result from
the unconfirmed mechanism, and gives concrete falsifiable next steps
(single-TU harness, check for a Mull dedup option) rather than prescribing an
implementation. Downstream consequence — #431 and #432 need to stop treating
the survivor list as a per-site worklist — is already reflected in their
"Blocked" project status and #431's existing correction comment, so no
additional rescoping action is needed from this ticket itself.

Re-open/re-verify condition if ever disputed: find a hand-applied survivor
(any of the 352) that leaves the suite green — that would be a real gap and
would start bounding how much of the list is trustworthy. Four out of four
sampled so far (two here, matching the issue's own four) turn the suite red.

[Yaraslaut]

The linker-dedup hypothesis is refuted; the real mechanism is confirmed

Followed up on this issue's own "what would change the verdict" list. Built four minimal, single-translation-unit C++ programs (each one .cpp, linked alone, no second object competing for a symbol) in an isolated environment — WSL Ubuntu 24.04, Mull-19 0.34.0 / LLVM 19.1.7, clang 19.1.7 — deliberately a different OS/toolchain from every prior measurement in this repo, so the finding isn't specific to one build's linker behavior.

Result: the linker-dedup hypothesis is refuted. A cxx_remove_void_call mutant on a member-function call (holder.attachIdentity(...)) still survives with zero possibility of dedup — one TU, linked alone. Reading the mutant's own machine code with objdump shows why: the function labelled as the mutant is byte-identical to its _original twin. The call the mutator claims to remove was never actually removed from the compiled code.

Narrowed it further:

• Member call, temporary argument (std::string{primary}) — survives, call still present.
• Member call, no temporary (plain named argument) — survives, same signature. Rules out "destructor cleanup on a temporary blocks the removal."
• Free-function call, same statement shape (bumpCounter()) — killed correctly. The mutant body genuinely omits the call. This confirms the IR pass and the runner's activation plumbing both work in general; the defect is scoped to member/operator-call sites, not the mutator or runner as a whole.

Consequence: cxx_remove_void_call is 148 of the 352 survivors in runs[1]. remote.hpp's reply(...)/complete(...) sites and bridge.hpp's stored-callback invocations are exactly this call shape. Every cxx_remove_void_call survivor on a member-function call site is now known to be a tool artifact by direct mechanism, not by inference from 4 hand-applied samples — no further one-at-a-time re-verification is needed for that mutator family specifically.

This does not clear the other mutator families in the 352 (cxx_replace_scalar_call 97, cxx_init_const 26, cxx_assign_const 23, cxx_add_to_sub 13, cxx_gt_to_ge 8, other 37). A boundary-comparison mutator (cxx_gt_to_ge/cxx_gt_to_le on value > limit) was tested the same way in this investigation and both mutated and killed correctly in a single-TU build — so that family isn't implicated and its survivors should still be read as real.

Full writeup, all four repro cases, and next steps recorded in scripts/mutation_survivors.json's new mechanism_confirmed key.

Not yet done (see that key's not_yet_done list): reproducing against the CI-pinned clang major (20), checking whether this is already fixed in Mull's newest release, filing upstream (searched mull-project/mull's issue tracker, found no matching report), and re-scoping morph#431 to drop its reply(...)-deletion findings now that they're understood as tool artifacts rather than a residual list needing hand-verification.

[Yaraslaut]

Status update: this issue's body currently frames the mechanism as "inferred from the shape of the result, not verified" (the linker-dedup/COMDAT-folding hypothesis). That has since been superseded by this issue's own 2026-09-04 follow-up investigation (recorded in scripts/mutation_survivors.json's mechanism_confirmed key, landed via commit 567168b / PR #448): the COMDAT-folding hypothesis is refuted, and the real cause is a genuine Mull codegen defect — cxx_remove_void_call's IR pass silently does not remove the call it claims to for member-function call sites, confirmed by disassembling the mutant in an isolated single-TU build (byte-identical to the unmutated original) on a different toolchain (WSL Ubuntu, Mull-19/LLVM-19) than every prior measurement in this file. The body's mechanism section should be rewritten to reflect this. Per that same key's not_yet_done list, still outstanding: filing this upstream with mull-project/mull (searched 2026-09-04, no existing matching report), and confirming against CI's pinned clang 20 (attempted via a root-free llvm.org tarball, blocked on an ABI mismatch against Mull's clang-20 package — needs apt's clang-20 package with sudo, not yet tried).