Repository navigation
Conversation
A mutant such as `i -= 1` -> `i += 1` in a loop that appends to a vector can use up the machine's memory long before the timeout stops it. Each test now runs its mutants under a memory limit: by default four times the peak memory of its baseline, at least 512 MB, or --memory-limit MB (0 for none). - ctest.run_test waits with os.wait4 on a thread, which gives the peak resident set size of CTest and the test while keeping the timeout. The output goes to a temporary file. - Programs that call __asan_init get the limit through ASAN_OPTIONS (hard_rss_limit_mb, with the user's own options after it), because ASan fails to start under a data segment limit. Other programs run under `ulimit -d`, which CTest and the test inherit. - A run that fails after reaching the limit, by ASan's "hard rss limit exhausted", std::bad_alloc or its measured peak, is the new killed_by_resource_limit state. It counts as detected, and the console, reports and TUI show it. - The baseline line shows the memory used and the limit. The baseline event records the peak. Fixes #40. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lefticus
force-pushed
the
fix/issue-40-memory-limit
branch
from
November 18, 2026 03:08
15033fb to
1bd8fc8
Compare
This was referenced Nov 18, 2026
…he machine ASan checks the resident set size about ten times a second. The test program filled 1 MiB vectors in a tight loop, which a fast runner does at gigabytes a second, so the peak overshot the 256 MiB margin on Ubuntu CI. It now sleeps 5 ms per MiB, at most 200 MiB a second, and the margin is 128 MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #40. Stacked on #48 (#41): both change
run_baseline. Merge #48 first; GitHub then retargets this PR tomain, and the diff shrinks to this commit.What changes
Each test now runs its mutants under a memory limit, so a mutant that allocates forever is stopped before it takes the machine down. Example:
i -= 1→i += 1in a loop that appends to a vector.The limit: by default, four times the peak memory the test used in its baseline, and at least 512 MB.
--memory-limit MBsets it for every test, and--memory-limit 0turns it off.Measuring the peak:
ctest.run_testwaits for CTest withos.wait4on a thread. That reports the peak resident set size of CTest and the test it runs, whichsubprocesscan't, and keeps the timeout. I checked that the peak includes a grandchild. Output goes to a temporary file, with stdout and stderr interleaved as they were written.ASan programs get the limit through
ASAN_OPTIONS=hard_rss_limit_mb=N:allocator_may_return_null=0. The user's ownASAN_OPTIONScome after it, so they win. A program counts as ASan when its binary contains__asan_init. I checked this for both dynamic and staticlibasan; a plain binary doesn't contain it. When strain can't find a test's program, it usesASAN_OPTIONSalone, which is harmless without ASan.Other programs run under
ulimit -d(RLIMIT_DATA) through/bin/sh -c 'ulimit -d …; exec ctest …', which CTest and the test inherit. I used this instead ofpreexec_fn, which isn't safe once threads exist; the TUI runs the runner on a thread. The data segment limit makes an ASan program fail at startup, so it can't apply to ASan programs:New state,
killed_by_resource_limit: counted as detected. A failing run is classified this way when any of these holds:AddressSanitizer: hard rss limit exhausted. This message has noERROR:prefix, so the sanitizer pattern wouldn't catch it.std::bad_alloc.The check runs before the sanitizer check. Without a limit,
bad_allocstays an ordinary kill.Display: the baseline line now reads
baseline passed in 0.02s using 45 MB; mutants time out after 10s or stop at 512 MB, and the baseline event recordspeak_memory. A mutant stopped by the limit saysreached the 512 MB memory limit. The per-test summary line addsmemory Nonly when N > 0, so existing output is unchanged. The TUI shows the state with the short labelmemory.README: a Memory limit section, a row in the result states table, the score formula and the CLI table.
The console's state column widens from 19 to 24 characters, the length of the new state's name.
test_consoleis updated for that.TUI
(The clipped
tcolumn in the Mutants table predates this PR; I'll fix it separately.)Platforms
Both limits are enforced on Linux. On macOS,
ulimit -dis accepted but not enforced. As far as I know, ASan checkshard_rss_limit_mbfrom a background thread that only Linux and NetBSD start; I haven't verified that on macOS. A runaway mutant there still stops at its timeout, as before, and the Linux-only tests skip.Tests
-=→+=mutant appends forever finishes askilled_by_resource_limit. This is checked with the default ASan/UBSan build and with a build without sanitizers. With the limit disabled in the code, the same mutant runs to its 10 s timeout and the test fails.tests/test_ctest.py: builds a test that allocates 1 MB at a time forever, with and without ASan. Under a 256 MB limit, each is classifiedkilled_by_resource_limit, and the peak stays under the limit plus a 256 MB margin. ASan samples RSS about ten times a second, so it can overshoot. The tests also check that the baseline measures a 200 MB peak, thatASAN_OPTIONSare merged, that theulimitwrapper is built correctly, and that ASan detection handles a missing file and a directory.tests/test_results.py: the three ways a run is classified as reaching the limit, a passing run near the limit, andbad_allocwithout a limit.tests/test_cli.py:--memory-limit,0, and rejection of negative values.python -W error -m pytest: 271 passed, 9 skipped.ruff format --check,ruff checkand lizard are clean.🤖 Generated with Claude Code