Repository navigation
Make existing Engine tests fail when the code they run is broken - #1
Merged
rogers-andrew merged 1 commit intoNov 16, 2026
Merged
Conversation
rogers-andrew
force-pushed
the
strengthen-engine-tests
branch
from
November 12, 2026 19:55
65b387f to
4a9117a
Compare
[Low impact] The PR appears safe to merge; no actionable issues were identified.
|
rogers-andrew
force-pushed
the
strengthen-engine-tests
branch
from
November 16, 2026 18:46
4a9117a to
8d7026d
Compare
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the test-strengthening changes.
No open issues 🎉
What changed in this PR
Strengthens existing Engine tests so they detect regressions identified by mutation testing. The changes add assertions and input cases across seven crates without adding test functions or changing production code.
Changes:
- Expands boundary, input-validation, and diagnostic checks.
- Strengthens task coordination, cancellation, store-event, and tool-origin assertions.
- Documents updated mutation-testing results and deferred coverage.
| File | Description |
|---|---|
vibe/2026-11-12-1-strengthen-engine-tests.md |
Records scope, mutation results, and remaining coverage gaps. |
crates/runvil-plugin/src/service.rs |
Adds service-ID hash assertions. |
crates/runvil-internal/types/src/untrusted_inventory.rs |
Checks delimiter lookalikes containing CJK characters. |
crates/runvil-internal/types/src/names-tests.rs |
Checks the boundary between control and invalid charset errors. |
crates/runvil-internal/types/src/models.rs |
Checks rejection of an absent model ID. |
crates/runvil-internal/types/src/error-tests.rs |
Checks cause text beginning with a separator. |
crates/runvil-internal/runtime/src/subst-tests.rs |
Adds escaping, length-boundary, and lone-brace cases. |
crates/runvil-internal/runtime/src/execute/tests/waits.rs |
Checks ready-task ordering and nested task depth. |
crates/runvil-internal/runtime/src/execute/tests/run_inputs.rs |
Strengthens assertions on seed-free debug output. |
crates/runvil-internal/runtime/src/execute/tests/observations.rs |
Checks additional store operations and lifecycle events. |
crates/runvil-internal/runtime/src/execute/tests/happens_before.rs |
Adds nested cancellation and write-visibility checks. |
crates/runvil-internal/runtime/src/execute/run/tests-steer.rs |
Checks main-inbox delivery leaves task requests untouched. |
crates/runvil-internal/runtime/src/execute/run/tests-inbox.rs |
Checks the section reported for inbox delivery. |
crates/runvil-internal/runtime/src/execute/run/tests-drops.rs |
Adds a combined child-answer and timer-drop case. |
crates/runvil-internal/prompt/src/tests/section_fences.rs |
Expands fence-boundary and error-location cases. |
crates/runvil-internal/prompt/src/contract/tests.rs |
Checks duplicate-role errors name the role. |
crates/runvil-internal/prompt/src/body_text.rs |
Adds string equality and inequality assertions. |
crates/runvil-internal/model-client/src/client-tests.rs |
Checks the argument-name iterator’s size hint. |
crates/runvil-internal/lua/src/tools/tests-stats.rs |
Checks counters seeded after installation. |
crates/runvil-internal/lua/src/tools/tests-offering.rs |
Checks negative array lengths offer no entries. |
crates/runvil-internal/lua/src/tests/sys_and_var.rs |
Checks table reassignment of var is rejected. |
crates/runvil-internal/lua/src/tests/store.rs |
Checks numbered-read padding below the hundred boundary. |
crates/runvil-internal/lua/src/tests/store_errors.rs |
Adds conflict-error field and message checks. |
crates/runvil-internal/lua/src/tests/json/limits.rs |
Checks encoding at the exact held-key limit. |
crates/runvil-internal/lua/src/tests/json/encode.rs |
Checks object nesting against the depth limit. |
crates/runvil-internal/lua/src/tests/json/debug.rs |
Checks depth placeholders for invalid-key tables. |
crates/runvil-internal/lua/src/tests/diagnostics.rs |
Checks incomplete chunk owners leave diagnostics unchanged. |
crates/runvil-internal/lua/src/tests/cancellation.rs |
Checks cancellation takes precedence over a failed model answer. |
crates/runvil-internal/lua/src/protocol/tests/parse_tasks.rs |
Adds the minimum positive whole-number float case. |
crates/runvil-internal/lua/src/program.rs |
Checks runtime mapping of errors without a location. |
crates/runvil-internal/lua/src/models-tests.rs |
Checks whole-number token limits and unrelated userdata rejection. |
crates/runvil-internal/lua/src/models_loop/machine-tests.rs |
Checks excess batch results after partial completion. |
crates/runvil-internal/lua/src/models_loop-tests.rs |
Checks failed answers to the final inbox read. |
crates/runvil-internal/lua/src/messages-tests-list.rs |
Checks bounds validation precedes record validation. |
crates/runvil-internal/lua/src/error-tests.rs |
Checks empty line markers remain in failure text. |
crates/runvil-internal/lua/src/coro.rs |
Checks model selection rejects unrelated userdata. |
crates/runvil-engine/tests/suite/vfs.rs |
Checks the source line recorded for a tool call’s origin. |
Review effort: Balanced
Mutation testing makes one small change to the production code at a time and reports each change that every test still passes. Outside `runvil-vfs`, 89 such changes sat on a line that an existing test runs. At `a449a1bcc`, 84 of them are still in the code and 78 still pass every test: the test passes with the code it runs broken. Checks and input cases added to 41 existing tests now make 55 of the 78 fail a test. No test is added, and no production code changes. - `runvil-lua`: 24 of 40. - `runvil-runtime`: 15 of 18, three of them through a test in the `runvil-engine` suite. - `runvil-prompt`: 10 of 10. - `runvil-types`: 4 of 7. - `runvil-plugin`: 1 of 2, and `runvil-model-client`: 1 of 1. - The 41 tests are 19 in `runvil-lua`, 9 in `runvil-runtime`, 6 in `runvil-prompt`, 4 in `runvil-types`, and 1 each in `runvil-plugin`, `runvil-model-client`, and `runvil-engine`. - Of the 23 left, 18 alter no behavior, so no test can tell them apart, and 5 are reached only by tests about something else. The plan names each one. - The Engine has 2,958 tests before and after. - `cargo dylint` did not run here, because it is not installed on this machine. Plan: /plan/2026-11-12-2-strengthen-engine-tests.plan.md Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
rogers-andrew
force-pushed
the
strengthen-engine-tests
branch
from
November 16, 2026 19:39
8d7026d to
db4cf1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces cppalliance/runvil#212, which was opened against the repository the Engine has since left.
Existing tests only. No test is added, and no production code changes.
Rebased on
a449a1bccon 2026-11-16. Every figure below was measured again on that commit, and "What the rebase changed" lists the differences from the first version.Why
A test that runs a line does not always fail when the line is wrong. Mutation testing finds those cases: it makes one small change to the production code at a time (a swapped operator, a deleted
matcharm, a function body replaced by a default value) and reports each change that every test still passes. Per-test coverage then names the tests that run the changed line and pass anyway.Outside
runvil-vfs, 89 such changes sat on a line that an existing Engine test runs. Ata449a1bcc, 84 of them are still in the code and 78 still pass every test. For each one, this strengthens the test whose subject is that code: it adds a check on a value the test already produces, or an input case the test did not try.runvil-luarunvil-runtimerunvil-promptrunvil-typesrunvil-pluginrunvil-model-clientThe last three columns add up to the fourth.
What changed
41 existing tests gain checks or input cases, in 35 files. The workspace has 3,931 tests before and after, 2,958 of them the Engine's. Three examples:
a_duplicate_role_key_is_rejectedchecked only that an error came back. It now checks that the error names the role, so replacingContractKey::shownwith an empty string fails it.one_exact_fence_is_the_prologue_and_two_can_surround_empty_prosetried twoluafences with a blank line between them. It now also tries them with none, whichopening < posas<=insplit_section_blocksturns into an internal error.an_operation_watcher_sees_a_tool_calls_write_then_read_under_the_calls_originchecked the label and the file of a tool call's origin. It now checks the line too, sofirst_chunk_linereturning 0 or 1 fails it.Hunks worth a closer look
Most hunks are one or two assertions. These change more of a test's setup:
status_reports_a_parked_task_and_then_a_finished_onegains two fixture sections, so that a task at depth 2 exists.a_canceled_tasks_partial_write_is_readable_after_the_joingains a second trace.a_message_to_one_task_does_not_withdraw_another_chains_requestnow ends withdone, to read the unread messages.What the rebase changed
The first version was made against
cc51d454, 28 commits earlier. There, 63 of the 89 failed a test and 26 passed every test, with 45 tests strengthened in 37 files.route_store_to_shims,default_matches,yaml_type, the H1 filter ofPrompt::parse_body, and theis_h1guard ofScheduler::apply_settled. The other 84 are on the same source line as before.a449a1bccwithout this change. Tests added or reworked sincecc51d454catch them.the_records_span_the_complete_runtime_lifecycle, and themiddlecase ofmalformed_section_phases_report_locations. The five changes they were written for are among the 6, and fail tests inobservations-store.rsandpreamble_fences.rs.an_arg_default_must_match_the_declared_typeanda_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_not. Their checks are gone with them.ceb5da405rewrote. It is now the check inan_operation_watcher_sees_a_tool_calls_write_then_read_under_the_calls_origin, in therunvil-enginesuite, which is the test that reads a call's origin.Scheduler::finishthat ends the run when a chain ends with a determinism failure (scheduler/chain.rs:201). The prelude test was the only test that ended a chain that way. With apanic!in the guarded arm, all 3,931 tests still pass.a449a1bccin three files: no title line in a prompt source,scheduler_scenarioin place oftrace_scenario, and&json!({})as a block's arguments.The 23 left
Vec::new()asvec![],|as^on two values with no bit in common, and the library flags of a temporary VM that only parses a chunk.The plan,
plan/2026-11-12-2-strengthen-engine-tests.plan.md, names all 23 with the reason for each. It wasvibe/2026-11-12-1-strengthen-engine-tests.mdin the first version.Checks run
cargo nextest run --locked --workspace --all-features: 3,931 passed.cargo clippy --locked --workspace --all-targets --all-featureswithCARGO_BUILD_WARNINGS=deny.cargo fmt --all --check,cargo test -p build-xtask, andnode --test tools/docs-claims.mjs.#[test]added or removed, and no line touched outside test code.a449a1bcc: 6 fail a test and 78 pass every test. Applied to this tree: 61 fail a test, the 6 and the 55, and 23 pass every test, the 23 above.cargo dylint,cargo deny, the doc builds, the facade surface check, and the Windows tests. CI covers them.One of the 78, the guard in
awaited_round(scheduler/effects.rs:290), failsa_message_to_one_task_does_not_withdraw_another_chains_requestin some runs of unchangeda449a1bcc, 15 of 40, because it takes the first match in aHashMap. With this change it failed in 40 of 40.Where the numbers come from
The Engine was measured in
cppalliance/runvilwithcargo-mutants27.1.0, atfa9936f64and again at2d40bafa2. This repository atcc51d454held the same Engine crates, byte for byte, as that second commit. On 2026-11-16 the 89 were matched toa449a1bccby file, function, and source line, and tested there with and without this change.Not in this change
runvil-vfs.cc51d454anda449a1bcc, andrunvil-markdown. Neither has been measured.🤖 Generated with Claude Code
Summary by CodeRabbit