Skip to content

Make existing Engine tests fail when the code they run is broken - #1

Merged
rogers-andrew merged 1 commit into
cppalliance:masterfrom
rogers-andrew:strengthen-engine-tests
Nov 16, 2026
Merged

rogers-andrew merged 1 commit into
cppalliance:masterfrom
rogers-andrew:strengthen-engine-tests

Conversation

@rogers-andrew

@rogers-andrew rogers-andrew commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Replaces cppalliance/runvil#212, which was opened against the repository the Engine has since left.

Existing tests only. No test is added, and no production code changes.

Rebased on a449a1bcc on 2026-11-16. Every figure below was measured again on that commit, and "What the rebase changed" lists the differences from the first version.

Why

A test that runs a line does not always fail when the line is wrong. Mutation testing finds those cases: it makes one small change to the production code at a time (a swapped operator, a deleted match arm, a function body replaced by a default value) and reports each change that every test still passes. Per-test coverage then names the tests that run the changed line and pass anyway.

Outside runvil-vfs, 89 such changes sat on a line that an existing Engine test runs. At a449a1bcc, 84 of them are still in the code and 78 still pass every test. For each one, this strengthens the test whose subject is that code: it adds a check on a value the test already produces, or an input case the test did not try.

Crate Still in the code Fail a test without this change Pass every test Now fail a test Alter no behavior No fitting test
runvil-lua 41 1 40 24 14 2
runvil-runtime 22 4 18 15 2 1
runvil-prompt 11 1 10 10 0 0
runvil-types 7 0 7 4 1 2
runvil-plugin 2 0 2 1 1 0
runvil-model-client 1 0 1 1 0 0
Total 84 6 78 55 18 5

The last three columns add up to the fourth.

What changed

41 existing tests gain checks or input cases, in 35 files. The workspace has 3,931 tests before and after, 2,958 of them the Engine's. Three examples:

  • a_duplicate_role_key_is_rejected checked only that an error came back. It now checks that the error names the role, so replacing ContractKey::shown with an empty string fails it.
  • one_exact_fence_is_the_prologue_and_two_can_surround_empty_prose tried two lua fences with a blank line between them. It now also tries them with none, which opening < pos as <= in split_section_blocks turns into an internal error.
  • an_operation_watcher_sees_a_tool_calls_write_then_read_under_the_calls_origin checked the label and the file of a tool call's origin. It now checks the line too, so first_chunk_line returning 0 or 1 fails it.

Hunks worth a closer look

Most hunks are one or two assertions. These change more of a test's setup:

  • status_reports_a_parked_task_and_then_a_finished_one gains two fixture sections, so that a task at depth 2 exists.
  • a_canceled_tasks_partial_write_is_readable_after_the_join gains a second trace.
  • a_message_to_one_task_does_not_withdraw_another_chains_request now ends with done, to read the unread messages.

What the rebase changed

The first version was made against cc51d454, 28 commits earlier. There, 63 of the 89 failed a test and 26 passed every test, with 45 tests strengthened in 37 files.

  • 5 of the 89 are gone with the code they changed: route_store_to_shims, default_matches, yaml_type, the H1 filter of Prompt::parse_body, and the is_h1 guard of Scheduler::apply_settled. The other 84 are on the same source line as before.
  • 6 of the 84 fail a test at a449a1bcc without this change. Tests added or reworked since cc51d454 catch them.
  • Two hunks were dropped because they caught nothing more: the store calls added to the_records_span_the_complete_runtime_lifecycle, and the middle case of malformed_section_phases_report_locations. The five changes they were written for are among the 6, and fail tests in observations-store.rs and preamble_fences.rs.
  • Two strengthened tests no longer exist: an_arg_default_must_match_the_declared_type and a_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_not. Their checks are gone with them.
  • The check on an origin's line read it through the watcher of a store, in a test that ceb5da405 rewrote. It is now the check in an_operation_watcher_sees_a_tool_calls_write_then_read_under_the_calls_origin, in the runvil-engine suite, which is the test that reads a call's origin.
  • One change that the first version caught passes every test again: the guard in Scheduler::finish that ends the run when a chain ends with a determinism failure (scheduler/chain.rs:201). The prelude test was the only test that ended a chain that way. With a panic! in the guarded arm, all 3,931 tests still pass.
  • The added cases were adapted to a449a1bcc in three files: no title line in a prompt source, scheduler_scenario in place of trace_scenario, and &json!({}) as a block's arguments.

The 23 left

  • 18 alter no behavior, so no test can tell them apart. Examples: Vec::new() as vec![], | as ^ on two values with no bit in common, and the library flags of a temporary VM that only parses a chunk.
  • 5 are reached only by tests about something else, where a check would make the test's name untrue. They are missing coverage, not weak tests.

The plan, plan/2026-11-12-2-strengthen-engine-tests.plan.md, names all 23 with the reason for each. It was vibe/2026-11-12-1-strengthen-engine-tests.md in the first version.

Checks run

  • cargo nextest run --locked --workspace --all-features: 3,931 passed.
  • cargo clippy --locked --workspace --all-targets --all-features with CARGO_BUILD_WARNINGS=deny.
  • cargo fmt --all --check, cargo test -p build-xtask, and node --test tools/docs-claims.mjs.
  • A script over the diff: no #[test] added or removed, and no line touched outside test code.
  • The 84 changes applied to a449a1bcc: 6 fail a test and 78 pass every test. Applied to this tree: 61 fail a test, the 6 and the 55, and 23 pass every test, the 23 above.
  • vibelight gives the same output with and without the change.
  • Not run locally: cargo dylint, cargo deny, the doc builds, the facade surface check, and the Windows tests. CI covers them.

One of the 78, the guard in awaited_round (scheduler/effects.rs:290), fails a_message_to_one_task_does_not_withdraw_another_chains_request in some runs of unchanged a449a1bcc, 15 of 40, because it takes the first match in a HashMap. With this change it failed in 40 of 40.

Where the numbers come from

The Engine was measured in cppalliance/runvil with cargo-mutants 27.1.0, at fa9936f64 and again at 2d40bafa2. This repository at cc51d454 held the same Engine crates, byte for byte, as that second commit. On 2026-11-16 the 89 were matched to a449a1bcc by file, function, and source line, and tested there with and without this change.

Not in this change

  • runvil-vfs.
  • 54 changes on lines that no test runs. Those need new tests, which is separate work.
  • Code added between cc51d454 and a449a1bcc, and runvil-markdown. Neither has been measured.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Expanded coverage for Lua input validation, error reporting, cancellation, task coordination, and file operations.
    • Added checks for JSON depth and size limits, model options, prompt parsing, and runtime event details.
    • These test updates verify existing behavior; no user-facing functionality changes are included.
  • Documentation
    • Added a reference document summarizing test coverage, verification results, and known testing limitations.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

This pull request expands existing tests across Lua, prompt, runtime, types, model-client, and plugin code. Added cases cover edge inputs, error mapping, limits, cancellation, task scheduling, VFS observations, and value contracts. A plan records mutation-testing results and scope. No production changes are reported.

Changes: Engine test strengthening

Layer / File(s) Summary
Lua API and control-flow test cases
crates/runvil-internal/lua/src/coro.rs, crates/runvil-internal/lua/src/messages-tests-list.rs, crates/runvil-internal/lua/src/models-tests.rs, crates/runvil-internal/lua/src/models_loop-tests.rs, crates/runvil-internal/lua/src/models_loop/machine-tests.rs, crates/runvil-internal/lua/src/protocol/tests/parse_tasks.rs, crates/runvil-internal/lua/src/tests/cancellation.rs, crates/runvil-internal/lua/src/program.rs, crates/runvil-internal/lua/src/error-tests.rs, crates/runvil-internal/lua/src/tests/diagnostics.rs
Tests add cases for rejected Lua value types, model options and loop responses, integral concurrency limits, cancellation outcomes, error mapping, and source markers.
Lua JSON, store, and tool test cases
crates/runvil-internal/lua/src/tests/json/*, crates/runvil-internal/lua/src/tests/store.rs, crates/runvil-internal/lua/src/tests/store_errors.rs, crates/runvil-internal/lua/src/tests/sys_and_var.rs, crates/runvil-internal/lua/src/tools/tests-offering.rs, crates/runvil-internal/lua/src/tools/tests-stats.rs
Tests add JSON depth and cumulative-key boundary cases, store read and conflict cases, global reassignment cases, and tool-offer and statistics cases.
Prompt parsing and contract test cases
crates/runvil-internal/prompt/src/body_text.rs, crates/runvil-internal/prompt/src/contract/tests.rs, crates/runvil-internal/prompt/src/tests/section_fences.rs
Tests add borrowed BodyText equality checks, argument-default and duplicate-role cases, and Lua-fence parsing and error cases.
Runtime task, wait, and cancellation test cases
crates/runvil-internal/runtime/src/execute/run/tests-drops.rs, crates/runvil-internal/runtime/src/execute/run/tests-inbox.rs, crates/runvil-internal/runtime/src/execute/run/tests-steer.rs, crates/runvil-internal/runtime/src/execute/tests/happens_before.rs, crates/runvil-internal/runtime/src/execute/tests/preludes.rs, crates/runvil-internal/runtime/src/execute/tests/waits.rs
Tests add cases for timed waits, inbox delivery, message steering, nested cancellation, fanout determinism, join ordering, and task depth.
Runtime VFS and execution observation cases
crates/runvil-internal/runtime/src/execute/tests/observations.rs, crates/runvil-internal/runtime/src/execute/tests/suite/vfs.rs, crates/runvil-internal/runtime/src/execute/tests/run_inputs.rs, crates/runvil-internal/runtime/src/subst-tests.rs
Tests add VFS lifecycle observations, verify read and write origins during a section handoff, and extend checks for debug output and substitution previews.
Type, identity, and mutation-test records
crates/runvil-internal/types/src/error-tests.rs, crates/runvil-internal/types/src/models.rs, crates/runvil-internal/types/src/names-tests.rs, crates/runvil-internal/types/src/untrusted_inventory.rs, crates/runvil-plugin/src/service.rs, crates/runvil-internal/model-client/src/client-tests.rs, vibe/2026-11-12-1-strengthen-engine-tests.md
Tests add cases for error causes, catalog membership, name validation, inventory matching, service-ID hashing, and iterator size hints. The plan records mutation-testing findings, verification details, and scope.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Other

Suggested reviewers: vinniefalco

Merge Risk: Minimal · up to 4a911

This change only strengthens existing tests and adds a planning document, so it has no runtime impact. It carries no actionable merge risk beyond normal CI checks.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title says the change makes existing Engine tests fail when the code they run is broken. The PR strengthens 45 existing tests with added checks and input cases so they catch production-code mutati…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title says the change makes existing Engine tests fail when the code they run is broken. The PR strengthens 45 existing tests with added checks and input cases so they catch production-code mutations, which matches the title. The title is specific and describes the main change. It does not name the mutation-testing method, but it does not need to cover every detail.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a656600d-77be-4627-9a50-0d09d7147d30
Commits

Reviewing files that changed from the base of the PR and between cc51d45 and 4a9117a.

Files selected for processing: 38

  • crates/runvil-internal/lua/src/coro.rs
  • crates/runvil-internal/lua/src/error-tests.rs
  • crates/runvil-internal/lua/src/messages-tests-list.rs
  • crates/runvil-internal/lua/src/models-tests.rs
  • crates/runvil-internal/lua/src/models_loop-tests.rs
  • crates/runvil-internal/lua/src/models_loop/machine-tests.rs
  • crates/runvil-internal/lua/src/program.rs
  • crates/runvil-internal/lua/src/protocol/tests/parse_tasks.rs
  • crates/runvil-internal/lua/src/tests/cancellation.rs
  • crates/runvil-internal/lua/src/tests/diagnostics.rs
  • crates/runvil-internal/lua/src/tests/json/debug.rs
  • crates/runvil-internal/lua/src/tests/json/encode.rs
  • crates/runvil-internal/lua/src/tests/json/limits.rs
  • crates/runvil-internal/lua/src/tests/store.rs
  • crates/runvil-internal/lua/src/tests/store_errors.rs
  • crates/runvil-internal/lua/src/tests/sys_and_var.rs
  • crates/runvil-internal/lua/src/tools/tests-offering.rs
  • crates/runvil-internal/lua/src/tools/tests-stats.rs
  • crates/runvil-internal/model-client/src/client-tests.rs
  • crates/runvil-internal/prompt/src/body_text.rs
  • crates/runvil-internal/prompt/src/contract/tests.rs
  • crates/runvil-internal/prompt/src/tests/section_fences.rs
  • crates/runvil-internal/runtime/src/execute/run/tests-drops.rs
  • crates/runvil-internal/runtime/src/execute/run/tests-inbox.rs
  • crates/runvil-internal/runtime/src/execute/run/tests-steer.rs
  • crates/runvil-internal/runtime/src/execute/tests/happens_before.rs
  • crates/runvil-internal/runtime/src/execute/tests/observations.rs
  • crates/runvil-internal/runtime/src/execute/tests/preludes.rs
  • crates/runvil-internal/runtime/src/execute/tests/run_inputs.rs
  • crates/runvil-internal/runtime/src/execute/tests/suite/vfs.rs
  • crates/runvil-internal/runtime/src/execute/tests/waits.rs
  • crates/runvil-internal/runtime/src/subst-tests.rs
  • crates/runvil-internal/types/src/error-tests.rs
  • crates/runvil-internal/types/src/models.rs
  • crates/runvil-internal/types/src/names-tests.rs
  • crates/runvil-internal/types/src/untrusted_inventory.rs
  • crates/runvil-plugin/src/service.rs
  • vibe/2026-11-12-1-strengthen-engine-tests.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@greptile-apps

greptile-apps Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge; no actionable issues were identified.

Summary

Strengthens existing Engine tests with more assertions and input cases. No production code changes.

  • Existing Engine tests catch more breaks in the code they exercise.

Reviews (3) · Last reviewed commit: "Make existing Engine tests fail when the..." · Reviewed by Greptile

@rogers-andrew
rogers-andrew force-pushed the strengthen-engine-tests branch from 4a9117a to 8d7026d Compare November 16, 2026 18:46
Copilot AI balanced review requested due to automatic review settings November 16, 2026 18:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the test-strengthening changes.

No open issues 🎉

What changed in this PR

Strengthens existing Engine tests so they detect regressions identified by mutation testing. The changes add assertions and input cases across seven crates without adding test functions or changing production code.

Changes:

  • Expands boundary, input-validation, and diagnostic checks.
  • Strengthens task coordination, cancellation, store-event, and tool-origin assertions.
  • Documents updated mutation-testing results and deferred coverage.
File Description
vibe/​2026-11-12-1-strengthen-engine-tests.md Records scope, mutation results, and remaining coverage gaps.
crates/​runvil-plugin/​src/​service.rs Adds service-ID hash assertions.
crates/​runvil-internal/​types/​src/​untrusted_inventory.rs Checks delimiter lookalikes containing CJK characters.
crates/​runvil-internal/​types/​src/​names-tests.rs Checks the boundary between control and invalid charset errors.
crates/​runvil-internal/​types/​src/​models.rs Checks rejection of an absent model ID.
crates/​runvil-internal/​types/​src/​error-tests.rs Checks cause text beginning with a separator.
crates/​runvil-internal/​runtime/​src/​subst-tests.rs Adds escaping, length-boundary, and lone-brace cases.
crates/​runvil-internal/​runtime/​src/​execute/​tests/​waits.rs Checks ready-task ordering and nested task depth.
crates/​runvil-internal/​runtime/​src/​execute/​tests/​run_inputs.rs Strengthens assertions on seed-free debug output.
crates/​runvil-internal/​runtime/​src/​execute/​tests/​observations.rs Checks additional store operations and lifecycle events.
crates/​runvil-internal/​runtime/​src/​execute/​tests/​happens_before.rs Adds nested cancellation and write-visibility checks.
crates/​runvil-internal/​runtime/​src/​execute/​run/​tests-steer.rs Checks main-inbox delivery leaves task requests untouched.
crates/​runvil-internal/​runtime/​src/​execute/​run/​tests-inbox.rs Checks the section reported for inbox delivery.
crates/​runvil-internal/​runtime/​src/​execute/​run/​tests-drops.rs Adds a combined child-answer and timer-drop case.
crates/​runvil-internal/​prompt/​src/​tests/​section_fences.rs Expands fence-boundary and error-location cases.
crates/​runvil-internal/​prompt/​src/​contract/​tests.rs Checks duplicate-role errors name the role.
crates/​runvil-internal/​prompt/​src/​body_text.rs Adds string equality and inequality assertions.
crates/​runvil-internal/​model-client/​src/​client-tests.rs Checks the argument-name iterator’s size hint.
crates/​runvil-internal/​lua/​src/​tools/​tests-stats.rs Checks counters seeded after installation.
crates/​runvil-internal/​lua/​src/​tools/​tests-offering.rs Checks negative array lengths offer no entries.
crates/​runvil-internal/​lua/​src/​tests/​sys_and_var.rs Checks table reassignment of var is rejected.
crates/​runvil-internal/​lua/​src/​tests/​store.rs Checks numbered-read padding below the hundred boundary.
crates/​runvil-internal/​lua/​src/​tests/​store_errors.rs Adds conflict-error field and message checks.
crates/​runvil-internal/​lua/​src/​tests/​json/​limits.rs Checks encoding at the exact held-key limit.
crates/​runvil-internal/​lua/​src/​tests/​json/​encode.rs Checks object nesting against the depth limit.
crates/​runvil-internal/​lua/​src/​tests/​json/​debug.rs Checks depth placeholders for invalid-key tables.
crates/​runvil-internal/​lua/​src/​tests/​diagnostics.rs Checks incomplete chunk owners leave diagnostics unchanged.
crates/​runvil-internal/​lua/​src/​tests/​cancellation.rs Checks cancellation takes precedence over a failed model answer.
crates/​runvil-internal/​lua/​src/​protocol/​tests/​parse_tasks.rs Adds the minimum positive whole-number float case.
crates/​runvil-internal/​lua/​src/​program.rs Checks runtime mapping of errors without a location.
crates/​runvil-internal/​lua/​src/​models-tests.rs Checks whole-number token limits and unrelated userdata rejection.
crates/​runvil-internal/​lua/​src/​models_loop/​machine-tests.rs Checks excess batch results after partial completion.
crates/​runvil-internal/​lua/​src/​models_loop-tests.rs Checks failed answers to the final inbox read.
crates/​runvil-internal/​lua/​src/​messages-tests-list.rs Checks bounds validation precedes record validation.
crates/​runvil-internal/​lua/​src/​error-tests.rs Checks empty line markers remain in failure text.
crates/​runvil-internal/​lua/​src/​coro.rs Checks model selection rejects unrelated userdata.
crates/​runvil-engine/​tests/​suite/​vfs.rs Checks the source line recorded for a tool call’s origin.

Review effort: Balanced

Mutation testing makes one small change to the production code at a time and reports each change that every test still passes. Outside `runvil-vfs`, 89 such changes sat on a line that an existing test runs. At `a449a1bcc`, 84 of them are still in the code and 78 still pass every test: the test passes with the code it runs broken. Checks and input cases added to 41 existing tests now make 55 of the 78 fail a test. No test is added, and no production code changes.

- `runvil-lua`: 24 of 40.
- `runvil-runtime`: 15 of 18, three of them through a test in the `runvil-engine` suite.
- `runvil-prompt`: 10 of 10.
- `runvil-types`: 4 of 7.
- `runvil-plugin`: 1 of 2, and `runvil-model-client`: 1 of 1.
- The 41 tests are 19 in `runvil-lua`, 9 in `runvil-runtime`, 6 in `runvil-prompt`, 4 in `runvil-types`, and 1 each in `runvil-plugin`, `runvil-model-client`, and `runvil-engine`.
- Of the 23 left, 18 alter no behavior, so no test can tell them apart, and 5 are reached only by tests about something else. The plan names each one.
- The Engine has 2,958 tests before and after.
- `cargo dylint` did not run here, because it is not installed on this machine.

Plan: /plan/2026-11-12-2-strengthen-engine-tests.plan.md

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@rogers-andrew
rogers-andrew force-pushed the strengthen-engine-tests branch from 8d7026d to db4cf1f Compare November 16, 2026 19:39
Copilot AI balanced review requested due to automatic review settings November 16, 2026 19:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for these regression-test improvements.

No open issues 🎉

Review effort: Balanced

@awheeler294 awheeler294 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good

@rogers-andrew
rogers-andrew merged commit db4cf1f into cppalliance:master Nov 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants