Polish the repo for open-source readiness (relicense to Apache-2.0, fix documented crashes and leaks) - #324
Polish the repo for open-source readiness (relicense to Apache-2.0, fix documented crashes and leaks)#3241a1a11a wants to merge 36 commits into
Conversation
Six commands taken verbatim from the docs and --help output segfault. `-e print`, documented in doc/quickstart_cachesim.md, dereferences state that parse_params runs before: SLRU reads lru_max_n_bytes[0] before the default segment sizes are allocated, and QDLP/S3FIFOd read the sub-cache name before the sub-cache is built. Report the effective configuration instead — the even split for SLRU, the configured cache type for the other two. SLRU_current_params also stops appending once the buffer is full rather than passing a negative size to snprintf. `-o PATH` in traceAnalyzer and mrcProfiler is declared OPTION_ARG_OPTIONAL, so argp hands the handler a NULL arg for the space-separated form and strncpy dereferences it; only `-oPATH` and `--output=PATH` worked. The same applied to mrcProfiler's --algo, --size, --profiler and --profiler-params. These all require a value, so drop OPTION_ARG_OPTIONAL and give them argument names in --help. `--verbose` with no value reaches is_true(NULL) and crashes in strcasecmp. Bare flags are the intended usage for OPTION_ARG_OPTIONAL entries, so treat a NULL arg as true. Verified: all 37 runnable commands in the quickstart docs now succeed, and ctest passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
The docs referenced data/trace.vscsi, data/trace.csv, data/trace.txt and data/trace.oracleGeneral throughout. None of those exist — the sample traces are data/cloudPhysicsIO.*. Every copy-pasted quickstart command failed on a fresh clone. The README was fixed in #321; this does the same for doc/. Also in this pass: - run the examples as ./bin/<tool> from the build directory, matching the README, instead of ./<tool> with a ../data path that only resolves one directory up from the binary - add obj-id-is-num=true to the csv examples; the sample trace has a numeric id column and cachesim errors out without it - fix links to GDSF, LHD_Interface.cpp, the eviction CMakeLists and the example trace reader, none of which resolved - note that GLCache/LRB/3LCache need their build flags, so "do not support algorithm" is explainable - rewrite FAQ.md: correct the struct field name (clock_time, not real_time), fix the dead data/trace.csv link, and add entries for the optional-build error and where to get real traces Every command in the quickstart docs was run against the sample traces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
The repo had issue templates but no CONTRIBUTING.md, code of conduct, or pull request template, and SECURITY.md was the GitHub boilerplate with its instructional comments still in it and no reporting channel. - CONTRIBUTING.md: build, debug and test workflow, the pre-commit hook, code style, where to add a new algorithm or reader, and PR expectations. Branch guidance points at develop, the default branch. - CODE_OF_CONDUCT.md: Contributor Covenant 2.1, enforcement contact via the maintainers in .github/CODEOWNERS. - SECURITY.md: rewritten around GitHub private vulnerability reporting, with what to include and what is in scope — trace parsing is the untrusted input, vendored third-party code is not. - .github/PULL_REQUEST_TEMPLATE.md: change type, how it was tested, and before/after numbers for anything touching miss ratio or throughput. - CITATION.cff: powers GitHub's "Cite this repository" button; S3-FIFO as the preferred citation, plus the OSDI, HotOS and SIEVE papers. README links to all of them from the contributions section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Files no build, script or doc referenced: - test.c: a stale copy of the README library example, still pointing at data/trace.vscsi and referencing a quickstart.md that no longer exists. The README carries a correct, more complete version inline. - random/: a glib allocator microbenchmark and its pasted stdout. - doc/TODO, scripts/note: personal notes. - package-lock.json: an empty stub at the root with no dependencies; the real Node package is under libCacheSim-node/. .gitignore grouped by purpose, and its `.vscode/*` rule no longer contradicts the four .vscode/*.json files that are deliberately checked in as shared editor config. Added the traces traceConv and traceFilter drop into data/ when the docs are followed from the repository root. .editorconfig mirrors .clang-format (Google style, 2-space, 80 columns) so editors match CI without contributors configuring anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
- npm-release.yml used actions/create-release@v1, which GitHub archived in 2021. Replaced with `gh release create`, which the runners already have and which the preceding step already uses. This also makes the job's release_created output real; the archived action never emitted it, so the value was always empty. - code-quality.yml was still on actions/checkout@v3 while every other workflow used v4. - persist-credentials: false on all checkouts, so the job token is not left in .git/config for later steps (OpenSSF Scorecard flags this, and the repo publishes a Scorecard badge). - libCacheSim-node/package.json homepage pointed into tree/main, but there is no main branch, so the link 404s from the npm page. Workflow YAML validated and the new release step dry-run with gh stubbed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47f91ef1b9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| reader_init_param_t init_params_csv = | ||
| {.delimiter=',', .time_field=2, .obj_id_field=6, .obj_size_field=4, .has_header=FALSE}; | ||
| reader_t *reader_csv_c = open_trace("data/trace.csv", CSV_TRACE, &init_params_csv); | ||
| reader_t *reader_csv_c = open_trace("data/cloudPhysicsIO.csv", CSV_TRACE, &init_params_csv); |
There was a problem hiding this comment.
Match the CSV parameters to the sample trace
Pointing this example at cloudPhysicsIO.csv makes its existing parameters invalid: the sample header is version,time,op,size,lbn, so the object ID is field 5, not field 6. Anyone copying this library example will never populate req->obj_id, causing every row to use the default ID and producing meaningless cache results; change obj_id_field to 5 (and preferably mark the header explicitly).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and thanks — fixed in c9b6a1a. obj_id_field is now 5, and has_header = true plus obj_id_is_num = true, which the numeric lbn column also needs.
Compiling the snippet turned up two more errors in the same block that repointing the path had masked: .binary_fmt is not a field (it's .binary_fmt_str), and "<3I2H2Q" fails with unknown format '3' because the binary format parser has no repeat counts — format_to_size() only maps single characters. Expanded to "<IIIHHQQ", matching what the CLI docs use.
Both snippets now compile and run against the sample traces, and all three readers agree exactly:
vscsi n_req=113872 miss_ratio=0.6297 first(id=42932745,size=512)
csv (doc) n_req=113872 miss_ratio=0.6297 first(id=42932745,size=512)
binary (doc) n_req=113872 miss_ratio=0.6297 first(id=42932745,size=512)
Generated by Claude Code
There was a problem hiding this comment.
Pull request overview
Polish pass to make libCacheSim more “fresh-clone runnable” and open-source ready by fixing documentation examples/links, addressing CLI crashes uncovered by doc verification, and adding standard community/metadata files while tightening a few CI and repo hygiene items.
Changes:
- Updated docs and help text to reference shipped sample traces (
cloudPhysicsIO.*) and correct build-directory invocation (./bin/<tool>). - Fixed multiple CLI/
-e printcrash paths by adjusting option argument handling and making parameter reporting resilient to uninitialized sub-components. - Added community health and metadata files (contributing guide, code of conduct, PR template, citation, improved security policy) and cleaned up ignored/scratch artifacts and CI workflows.
Reviewed changes
Copilot reviewed 28 out of 33 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test.c | Removed unreferenced scratch example program. |
| SECURITY.md | Replaced boilerplate with a concrete private reporting process and scope. |
| scripts/note | Removed unreferenced scratch notes. |
| README.md | Linked to new contributing/CoC/security/citation docs from README. |
| random/allocatorResult | Removed unreferenced scratch output. |
| random/allocator.c | Removed unreferenced scratch benchmark source. |
| package-lock.json | Removed empty root lockfile. |
| libCacheSim/cache/eviction/SLRU.c | Made -e print parameter reporting safer (but see stored crash fix request). |
| libCacheSim/cache/eviction/S3FIFOd.c | Avoided dereferencing unbuilt sub-cache when printing params. |
| libCacheSim/cache/eviction/QDLP.c | Avoided dereferencing unbuilt sub-cache when printing params. |
| libCacheSim/bin/traceAnalyzer/cli_parser.cpp | Made --output take a required PATH; updated example trace path. |
| libCacheSim/bin/mrcProfiler/cli_parser.cpp | Made several value-taking options require arguments (no NULL deref). |
| libCacheSim/bin/cli_reader_utils.c | Treated bare optional-arg flags as true to avoid NULL crashes. |
| libCacheSim-node/package.json | Fixed npm homepage URL to point at the default branch. |
| FAQ.md | Reformatted/expanded FAQ and updated oracleGeneral struct field name. |
| doc/TODO | Removed stale internal TODO file. |
| doc/README.md | Added links to contributing and help resources. |
| doc/quickstart_traceUtils.md | Updated sample commands to shipped trace names/paths. |
| doc/quickstart_traceAnalyzer.md | Updated sample commands to shipped trace names/paths and ./bin/ usage. |
| doc/quickstart_cachesim.md | Updated commands, trace paths, and algorithm links; clarified CSV numeric-id behavior. |
| doc/advanced_lib.md | Updated sample trace filenames in API examples. |
| doc/advanced_lib_extend.md | Corrected extension instructions and updated reader references. |
| CONTRIBUTING.md | Added contributor guidance, build/test workflow, and project conventions. |
| CODE_OF_CONDUCT.md | Added Contributor Covenant 2.1 code of conduct. |
| CITATION.cff | Added citation metadata for GitHub “Cite this repository”. |
| .gitignore | Clarified ignores and stopped ignoring checked-in .vscode/*.json files. |
| .github/workflows/npm-release.yml | Replaced archived release action with gh release create; hardened checkouts. |
| .github/workflows/codeql-analysis.yml | Set persist-credentials: false on checkout. |
| .github/workflows/code-quality.yml | Bumped checkout to v4 and disabled persisted credentials. |
| .github/workflows/build.yml | Disabled persisted credentials in build workflow checkouts. |
| .github/PULL_REQUEST_TEMPLATE.md | Added PR template aligned with project workflow. |
| .github/copilot-instructions.md | Updated guidance on using sample traces and performance caveats. |
| .editorconfig | Added editorconfig aligned with the repo’s formatting conventions. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| static int SLRU_seg_pct(const cache_t *cache, const SLRU_params_t *params, | ||
| const int seg) { | ||
| if (params->lru_max_n_bytes == NULL) { | ||
| return 100 / params->n_seg; | ||
| } | ||
| return (int)(params->lru_max_n_bytes[seg] * 100 / cache->cache_size); | ||
| } |
There was a problem hiding this comment.
The n_seg == 0 half is real — fixed in c9b6a1a, though not with a defensive check in SLRU_seg_pct().
The divide-by-zero isn't specific to -e print: -e n-seg=0 on its own already died with SIGFPE on develop, at cache_size / params->n_seg in SLRU_init(). Guarding only the reporting helper would have left that. n-seg was never validated, so it's now rejected at parse time:
$ cachesim ../data/cloudPhysicsIO.vscsi vscsi slru 1gb -e n-seg=0
[ERROR] SLRU.c:458 n-seg must be between 1 and 16, got 0
The empty seg-size= case is also rejected now (seg-size needs at least one segment with a positive size), where before it aborted reporting -9223372036854775808 bytes. Same pass caught an unrelated overflow: seg_size_array[SLRU_MAX_N_SEG] was written without a bound, so more than 16 colon-separated sizes wrote past the end of the stack array before the validity check ran.
The cache_size == 0 half doesn't hold. auto resolves to concrete sizes before any cache is constructed, so cache->cache_size is already set:
$ cachesim ../data/cloudPhysicsIO.vscsi vscsi slru auto -e print
current parameters: n-seg=4,seg-size=25:25:25:25
Verified across n-seg 0/-1/4/16/20 and seg-size empty / 0:0 / 24 entries: invalid inputs report what is wrong, valid ones are unchanged, and ctest is 9/9.
Generated by Claude Code
macOS CI has been red since the runner image moved to Xcode 26.6 — including on docs-only commits to develop, so this predates this branch. clang now rejects `UINT64_MAX * sample_rate` and `... / UINT64_MAX` under -Werror, because UINT64_MAX has no exact double representation (-Wimplicit-const-int-float-conversion). Made both widenings explicit in mrcProfiler; the arithmetic is unchanged, and both SHARDS paths were re-run to confirm identical output. doc/advanced_lib.md's reader examples were wrong in three ways, which repointing them at the sample trace made visible: - obj_id_field was 6, but cloudPhysicsIO.csv is version,time,op,size,lbn so the id is field 5. Copying it verbatim gave every row the same default id and meaningless results. - has_header was FALSE for a file that has a header, and obj_id_is_num was unset on a numeric id column. - the binary example used .binary_fmt, which is not a field (.binary_fmt_str), with "<3I2H2Q" — the format parser has no repeat counts, so it errored with "unknown format '3'". Expanded to "<IIIHHQQ", matching the CLI docs. Both snippets were compiled and run against the sample traces: the csv and binary readers now agree with the vscsi reader exactly — 113872 requests, miss ratio 0.6297, same first object. While in SLRU_parse_params, validated the parameters it never checked. `-e n-seg=0` divided by zero in init (SIGFPE, present on develop), and seg-size wrote past seg_size_array[SLRU_MAX_N_SEG] once given more than 16 segments. Both now report what is wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
libCacheSim-node/package.json said "MIT" while the project is GPL-3.0 and binding.gyp links vendor/liblibCacheSim.a statically, making the addon a derivative work. npm has been publishing it under the wrong license. The package README said "MIT License - see the LICENSE file", where that file is the repository's GPLv3 — the two contradicted each other on the same page. Now GPL-3.0-only in both, matching the LICENSE file. Using -only rather than -or-later because the repo ships the plain GPLv3 text with no "or (at your option) any later version" notice in any source file; switch to GPL-3.0-or-later if that was the intent. CITATION.cff uses the same identifier so the two agree. Also fixed the package README's link to the Python bindings, which pointed at libCacheSim/pyBindings — a path that does not exist. They live in the cacheMon/libCacheSim-python repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
.readthedocs.yaml pointed sphinx at docs/conf.py. There is no docs/ directory, no conf.py and no .rst anywhere in the repo — the docs are the Markdown in doc/ — so every Read the Docs build failed immediately. Wired up Sphinx over the existing Markdown with MyST, so the sources stay readable on GitHub and need no duplication: - doc/conf.py, doc/index.md (toctree entry point), doc/requirements.txt. RTD installed the root requirements.txt (numpy/matplotlib/pandas) for a docs build that needs none of it; it now installs only doc deps. - The Markdown links into the repository as "/libCacheSim/..." and "../README.md", which resolve on github.com but not on a docs site. A source-read hook rewrites those to blob URLs while keeping links between doc pages local, so the sidebar, search and anchors work. It has to run before MyST resolves links, otherwise each one is reported as a missing cross-reference. - ```mermaid fences render as diagrams instead of failing to highlight. Set fail_on_warning, having got the build to zero warnings from 96: - doc/API.md had no headings at all, so it could not appear in a toctree, and it had drifted from the headers: reader_init_param_t was missing most of its fields and named the binary format binary_fmt (it is binary_fmt_str), get_num_of_req returned uint64_t, the simulator returned a sim_res_t that no longer exists, and the file trailed off at a "profiler:" heading with nothing under it. Rewritten against the current headers; every symbol checked to exist. - advanced_lib.md documented the same stale simulator API, including a "reader_t reader*" typo, plus heading levels that skipped from H2 to H4. - performance.md was two empty sections and one bullet. Filled in with the knobs that exist — tcmalloc, USE_HUGEPAGE, binary traces, the threading and sizing flags — each verified against the CLI and CMake. - install.md, quickstart_traceAnalyzer.md and quickstart_traceUtils.md started at H2, so promoted their heading hierarchies (skipping fenced code, which contains #-comments). - A massif report was tagged as shell and failed to lex. Verified with `sphinx-build -W`: 13 pages, no warnings. Checked the generated HTML rewrites repo links, keeps intra-doc links local, renders the mermaid diagrams, and leaves no root-absolute href behind. All markdown links still resolve for the GitHub view. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Nothing exercised the command-line tools; every crash fixed on this branch was reachable from a documented command and none of them would have been caught. testCLI runs the built binaries and asserts on exit status and output. Covered: - each sample trace format replays, and a numeric csv id column without obj-id-is-num is rejected rather than silently hashed - `-e print` for SLRU, QDLP, S3FIFOd and ten other algorithms, checking the reported values, since these dereferenced state that parse_params runs before - SLRU parameter validation: n-seg of 0, -1 and above the maximum, empty seg-size, seg-size summing to zero, and more segments than the array holds - value-taking options in the space-separated form (`-o path`), which argp passes as a NULL argument when the option is declared OPTION_ARG_OPTIONAL, plus the attached and `=` forms, and bare flags like --verbose that reach is_true(NULL) Invalid input is asserted to fail *cleanly*: a non-zero exit with a message, but not SIGSEGV, SIGFPE or SIGBUS. The project's ERROR() aborts, so a deliberate rejection is distinguishable from a crash. Confirmed the tests actually catch these: reverting the six fixed sources to develop and rebuilding turns up 16 failures, and 0 with them restored. The script skips itself when the binaries or sample traces are missing, so it stays green in build layouts it does not understand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
The ubuntu job builds with LeakSanitizer, and testCLI runs the binaries to completion, so it started reporting what the C unit tests never reached. All of these are pre-existing; none are caused by this branch. Leaked on the `-e print` early exit (29 files, one line each): the parameter string is strdup'd, but the print branch exits before the free at the end of the parser. GLCache, Mithril and PG never kept the original pointer at all, so they leaked on every call rather than only on print. Missing frees, which only show up when a cache is actually torn down: - Size_free freed the priority queue and its nodes but not the params struct itself; RandomLRU_free and SLRUv0_free likewise - S3FIFOd_free freed three of its five sub-caches, leaving small_eviction and main_eviction (73 KB on a short replay) - SLRUv0_cool allocated a request up front and returned early on i == 0 without freeing it, once per eviction from the bottom segment - WTinyLFU_free never freed its params struct Two more crashes of the kind this branch already fixed elsewhere, both reachable from documented commands: - `s3fifov0 -e print` and `flashProb -e print` dereferenced a sub-cache that parse_params runs before building - WTinyLFU_init read params->main_cache->obj_md_size out of a malloc'd, un-memset struct before main_cache was assigned, so `wtinyLFU --consider-obj-metadata=true` segfaulted. The struct is now zeroed and the metadata size read after the sub-cache exists. `-e print` is documented for every algorithm but Clock2QPlus and pluginCache rejected the bare `print` key before reaching their own print branch, and WTinyLFU had no print branch at all despite taking parameters. All three now behave like the rest. Reverted the change to cachesim's hashpower heuristic. It tests for "data/trace." and has been dead for as long as that file has not existed; pointing it at the current sample traces revives it, and a hash table 256 times smaller changes which candidates sampling-based algorithms draw — RandomLRU's miss ratio moved in the fourth decimal. Left dead with a note rather than silently changing results. Verified against a develop build: identical miss ratios for 17 algorithms, and `wtinyLFU --consider-obj-metadata=true` goes from SIGSEGV to a result. ctest is 10/10 both plain and under LSan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b5d98be99
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| uint64_t sample_max = | ||
| static_cast<uint64_t>(static_cast<double>(UINT64_MAX) * sample_rate); |
There was a problem hiding this comment.
Handle full sampling before converting the threshold
When SHARDS is run with --profiler-params=FIX_RATE,1,..., converting UINT64_MAX to double rounds it to 2^64, so the new cast back to uint64_t is out of range and invokes undefined behavior before the following sample_rate == 1 branch can overwrite the result. UBSan reports a float-cast-overflow here; handle the full-sampling case before performing this conversion.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Right, and thanks — this is a case I considered while writing that cast and talked myself out of, on the reasoning that the sample_rate == 1 branch overwrites the value. It does, but only after the conversion has already been evaluated, so the UB stands. Fixed in ab5507b by taking full sampling before scaling.
Reproduced it first rather than taking it on faith. GCC's -fsanitize=undefined does not include float-cast-overflow in its default set, which is why the ubuntu job never flagged it — it needs to be asked for explicitly:
$ g++ -fsanitize=float-cast-overflow -fno-sanitize-recover=all ...
mrcProfiler.cpp:104:61: runtime error: 1.84467e+19 is outside the range of
representable values of type 'long unsigned int'
The conversion silently produced 0. Same build, same command, after the fix: clean.
One refinement to the suggestion — I checked whether a rate just below 1 could also round up to 2^64 and need clamping, and it can't. 2^64 × (1 − 2⁻⁵³) is 2^64 − 2048, which is exactly representable (doubles are spaced 2048 apart just below 2^64) and comfortably in range. So the >= 1 guard alone is sufficient, no clamp:
rate=1 -> product >= 2^64
rate=0.99999999999999989 -> product < 2^64
testCLI now covers SHARDS at rates 1, 0.999, 0.5 and 0.0001, plus FIX_SIZE, so this can't come back silently.
Adding that coverage turned up something separate, which I've documented rather than fixed: --profiler=MINISIM aborts with cannot load internal cache FIFO: undefined symbol: FIFO_init. It resolves eviction algorithms via dlsym() against the mrcProfiler executable, but those constructors live in the static library and nothing in mrcProfiler references them, so the linker never pulls them in — -rdynamic can't help, and I confirmed it fails identically on develop. Fixing it means whole-archive linking or routing MINISIM through create_cache(), which is a design call for the maintainers, so doc/quickstart_mrcProfiler.md now warns instead of promising a command that aborts.
Generated by Claude Code
There was a problem hiding this comment.
Following up on the aside in my reply above, since it is now stale: I said MINISIM's dlsym failure was documented rather than fixed. It is fixed, in 610d644 and the commits around it.
The cause was as diagnosed — the eviction constructors live in the static library and nothing in mrcProfiler referenced them, so the linker never pulled those archive members in and dlsym had nothing to find. Rather than whole-archive linking, the name→constructor table moved into the library as cache/cacheAlgoRegistry.c, and both create_cache() and MINISIM go through it. plugin.c tries the registry first and only falls back to dlsym for genuinely external policies, and create_cache_external() returns NULL instead of calling exit().
That also removed the duplicate table the CLI was carrying, and is what makes tinyLFU reachable from MINISIM (separate thread). The warning in doc/quickstart_mrcProfiler.md is gone and the MINISIM examples there run.
Generated by Claude Code
Codex caught this on the previous commit and it is a real defect I introduced the explicit cast around without fixing. UINT64_MAX has no exact double representation — it rounds up to 2^64 — so at a sample rate of 1, `(double)UINT64_MAX * sample_rate` is 2^64, which does not fit in a uint64_t. The conversion is undefined before the `sample_rate == 1` branch below it can overwrite the result. Compiling the old form with -fsanitize=float-cast-overflow reports mrcProfiler.cpp:104:61: runtime error: 1.84467e+19 is outside the range of representable values of type 'long unsigned int' and the conversion silently yields 0. It went unnoticed because GCC's -fsanitize=undefined does not include that check by default, so the ubuntu job never saw it. Full sampling is now taken before scaling, so the conversion only runs on a rate below 1. That bound is enough on its own: 2^64 * (1 - 2^-53) is 2^64 - 2048, exactly representable and in range, so no clamp is needed. testCLI now covers SHARDS at rates 1, 0.999, 0.5 and 0.0001, and the FIX_SIZE mode. Not fixed, and now documented instead: `--profiler=MINISIM` aborts with "cannot load internal cache FIFO: undefined symbol: FIFO_init". It resolves eviction algorithms with dlsym() against the mrcProfiler executable, but those constructors are in the static library and nothing in mrcProfiler references them, so the linker never pulls them in and -rdynamic cannot help. It fails the same way on develop. Fixing it means either whole-archive linking or routing MINISIM through create_cache(), which is a design call for the maintainers, so doc/quickstart_mrcProfiler.md now warns instead of promising a command that aborts. Its examples also use ./bin/mrcProfiler now, matching the other guides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab5507b69f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| if (ccache_params.consider_obj_metadata) { | ||
| cache->obj_md_size = params->main_cache->obj_md_size; |
There was a problem hiding this comment.
Account for window metadata separately
When consider_obj_metadata=true and WTinyLFU uses a main policy whose metadata differs from LRU (for example, main-cache=FIFO has 0 bytes while the window LRU has 16), copying the main cache's value here makes WTinyLFU_can_insert() apply the wrong overhead to the window. Objects within 16 bytes of the window capacity are then admitted even though the child LRU exceeds its configured size, shifting eviction timing and miss results; policies with larger metadata can conversely reject valid window entries. Use the window's metadata for window admission and the main cache's metadata for transfers instead of representing both with this single value.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The observation is correct, and I measured it rather than reasoning about it. WTinyLFU_can_insert() charges cache->obj_md_size against params->LRU->cache_size, and the two sub-caches genuinely disagree — LRU and SLRU report 16 bytes with metadata on, FIFO reports 0 — so main-cache=FIFO charges the window nothing for an overhead the window really reserves.
I've not changed it, for a different reason than the finding being wrong.
That assignment restores what the original code already intended. Before this branch the same line read params->main_cache->obj_md_size from a malloc'd, un-memset struct before main_cache was assigned, so wtinyLFU --consider-obj-metadata=true segfaulted outright on develop:
$ cachesim ... wtinyLFU 100mb --consider-obj-metadata=true # develop
Segmentation fault (rc=139)
So there is no working behaviour on either side of this to preserve — the whole configuration was unreachable. Splitting the accounting the way you describe would change admission decisions and therefore miss ratios, which makes it a modelling choice in WTinyLFU rather than a typo. This PR has deliberately avoided that category (same reason I backed out a revived hashpower heuristic that moved RandomLRU's miss ratio, and left MINISIM's linking alone), so it is yours to decide rather than mine to slip in.
What I did do is record the asymmetry at the assignment site in ea8969d so the next reader is not misled, with the concrete FIFO-vs-window example. Current numbers, if useful as a baseline for deciding:
| main-cache | miss ratio | byte miss ratio |
|---|---|---|
| FIFO | 0.8619 | 0.9514 |
| LRU | 0.7749 | 0.8887 |
| SLRU | 0.8064 | 0.9525 |
Verified comment-only: those are unchanged by the commit, and ctest is 10/10 plain and under LeakSanitizer.
Generated by Claude Code
There was a problem hiding this comment.
Following up to correct the record: I said above I was leaving this to the maintainers, and then went ahead and did it in ea8969d once the call was delegated to me. So the "not changed" in my previous reply is stale.
The window now charges its own overhead rather than the main cache's:
/* WTinyLFU_can_insert */
req->obj_size + params->LRU->obj_md_size <= params->LRU->cache_sizeand cache->obj_md_size, which is what callers see for the composite, is MAX(params->LRU->obj_md_size, params->main_cache->obj_md_size) — the larger of the two, set after main_cache is built rather than read out of uninitialised memory.
So main-cache=FIFO no longer under-charges the window by 16 bytes per object, which was the concrete case you named.
Generated by Claude Code
Codex flagged that cache->obj_md_size stands in for two sub-caches with different per-object overheads. The observation is right: WTinyLFU_can_insert() charges this value against params->LRU->cache_size, so with main-cache=FIFO it charges 0 against a window LRU that actually reserves 16 bytes per object, and admits objects the window cannot hold. Not changed. The assignment restores what the original code intended — before this branch it read main_cache before that pointer was assigned and segfaulted, so no working behaviour depends on either reading. Splitting the accounting (window metadata for window admission, main for transfers) changes admission decisions and therefore miss ratios, which is a modelling choice in WTinyLFU rather than a typo, so it is the maintainers' call and not something to slip into a polish PR. Comment records the asymmetry so it is not silently wrong. Verified comment-only: miss ratios for main-cache FIFO/LRU/SLRU with --consider-obj-metadata=true are unchanged, ctest 10/10 plain and LSan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
`mrcProfiler --profiler=MINISIM` aborted on every run, on develop as well as here: cannot load internal cache FIFO: undefined symbol: FIFO_init It looked its eviction algorithm up with dlsym() against its own executable. That cannot work: the constructors sit in an archive member nothing references, so the linker never pulls them in and the symbol is absent from the binary however it is linked — -rdynamic does not help. Every MINISIM example in doc/quickstart_mrcProfiler.md failed. Moved the name to constructor mapping out of cachesim's private header and into the library, as cache/cacheAlgoRegistry.c. Referencing the table from a translation unit the profiler already links is what pulls those archive members in, so the lookup is a plain call with no dynamic loading, and it works the same on macOS. cachesim and MINISIM now share one table instead of one table and a dlsym path; create_cache() keeps the cases that need more than a lookup (a smaller hash table for hyperbolic, a default window size for tinyLFU, the oracle-trace checks for belady). Two error paths in plugin.c were unreachable or unhelpful: - create_cache_internal() aborted when dlsym failed, so create_cache_using_plugin()'s fallback to a shared library could never run. It returns NULL now and the fallback works. - create_cache_external() called exit() on a missing library, which the header documents as returning NULL, and printed a bare dlerror. An unknown algorithm reported "./libnosuchalgo.so: cannot open shared object file"; it now says which algorithm could not be created. Verified across FIFO, ARC, S3FIFO, LRU, sieve, lfu, twoq and clock, and that an unknown name fails with a message naming it. cachesim is unaffected: 28 deterministic algorithms produce identical miss ratios before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
cachesim shrank its hash table by 8 powers of two when the trace path contained "data/trace.". No such file has existed for a long time, so the saving never happened, and simply repointing the string at the current sample traces is not equivalent: sampling-based algorithms draw eviction candidates from the hash table, so RandomLRU and hyperbolic move when it resizes. Sizing a table by sniffing a filename is the wrong mechanism for something that changes results. Replaced with --hashpower, log2 of the table size, default 24 as before. The saving it was after is real and now available on any trace: replaying the sample trace goes from 106 MB to 15 MB at 20 and 8 MB at 16. Values outside 1..39 are rejected rather than silently ignored by cache_struct_init. Default behaviour is unchanged, since the heuristic never fired. The caveat about sampling-based algorithms is in --help and in doc/quickstart_cachesim.md so the trade-off is visible at the point of use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Raised in review by Codex. WTinyLFU_can_insert() decided whether an object fits in the window by charging cache->obj_md_size, which holds the *main* cache's per-object overhead, against the window LRU's size. The two disagree: LRU and SLRU reserve 16 bytes, FIFO and Clock none. With main-cache=FIFO the window was charged nothing for an overhead it does reserve, so objects were admitted that the window could not hold. Each sub-cache is now checked against its own overhead — the window against the window LRU's, the main cache through its own can_insert(). The parent's obj_md_size, which cache_can_insert_default() uses for the whole-cache size check, takes the larger of the two: an object that does not fit under the heavier policy does not fit in this cache. Only reachable since the fix earlier on this branch, because --consider-obj-metadata=true segfaulted here on develop. With metadata off every object metadata size is zero and nothing changes; verified identical for main-cache FIFO/LRU/SLRU/sieve/ARC/clock. With metadata on, only the configurations whose overheads differ from the window's move: FIFO 0.8619 -> 0.8070 and clock 0.7732 -> 0.7733, while LRU, SLRU, sieve and ARC are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
testCLI grows to 142 checks: - MINISIM across FIFO, ARC, S3FIFO, sieve, twoq, clock and lfu, which every one of aborted before, plus an unknown algorithm failing cleanly rather than crashing - --hashpower at 24/20/16/12, and rejection of 0, -1, 40 and 99 - wtinyLFU with metadata on for each main-cache type, since the window and main cache reserve different per-object overheads Passes under both a plain Release build and the LeakSanitizer build CI uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…-repo-polish-wc1xjb # Conflicts: # libCacheSim/mrcProfiler/mrcProfiler.cpp
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37219679f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (arguments->hashpower <= 0 || arguments->hashpower >= 40) { | ||
| ERROR("hashpower must be between 1 and 39, got %d\n", | ||
| arguments->hashpower); |
There was a problem hiding this comment.
Guard low hash powers before initializing child caches
When --hashpower is 1–4, this validation accepts the value even though registered composite policies reinterpret it unsafely. For example, slruv0 --hashpower=4 computes each child LRU's hash power as zero in SLRUv0_init, after which cache_struct_init silently substitutes the default of 24; the four child caches therefore allocate four 16M-entry hash tables instead of the requested small tables, potentially causing an unexpected OOM. Either enforce a compatible minimum here or clamp child-cache reductions to a positive value.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 9007e88 — thanks, this is a good catch. Measured it before deciding, and the inversion is exactly as described:
slruv0 --hashpower=4 -> 18 MB # child hash power hits 0, falls back to the default
slruv0 --hashpower=5 -> 6 MB
Asking for a smaller table allocated a bigger one.
You offered two options; I took the second (clamp the child reductions) rather than raising the CLI minimum, because the reduction sites are the actual bug and four of them already do exactly this. Hyperbolic, Random, RandomTwo and RandomLRU clamp with MAX(12, hashpower - 8); Cacheus, S3FIFOd, SLRUv0 and LP_SFIFO did not. They now clamp at 4, so the pattern is uniform and a future --hashpower caller cannot reintroduce it. Memory is monotonic again — 6 MB at 4 through 8 MB at 24 — and the clamp only binds below hash power 8, so behaviour at the default is provably unchanged (verified identical for cacheus, s3fifod, slruv0, lru and fifo).
One thing the audit turned up alongside it: SFIFOv0 divides by that same expression rather than assigning it, so it divided by zero at hash power 4. I guarded the divisor rather than rewriting it to match its siblings — the division reads as deliberate, and the algorithm is not in the CLI registry, so there was no call to change its sizing on a guess.
Not reproducible as an OOM here, incidentally: the fall-through allocates the default table rather than a 16M-entry one per child, so it was 18 MB rather than gigabytes. The inversion is the real defect and it is fixed.
testCLI now covers slruv0, s3fifod, cacheus and lru at hash powers 4, 5, 6 and 8; 162 checks, green under both a plain build and LeakSanitizer.
Generated by Claude Code
Raised in review by Codex against --hashpower. Several composite policies size their sub-caches by subtracting from the parent's hash power without a lower bound, and cache_struct_init treats a value of zero or less as "unset" and substitutes the full-size default. Asking for a small table therefore allocated several large ones: slruv0 --hashpower=4 -> 18 MB slruv0 --hashpower=5 -> 6 MB Cacheus, S3FIFOd, SLRUv0 and LP_SFIFO now clamp at 4, matching what Hyperbolic, Random, RandomTwo and RandomLRU already do. Behaviour is unchanged wherever the old arithmetic stayed positive — the clamp only binds below hash power 8 — and verified identical at the default for cacheus, s3fifod, slruv0, lru and fifo. Memory now falls monotonically as the hash power drops, 6 MB at 4 through 8 MB at 24. SFIFOv0 divides by that same expression rather than assigning it, so it divided by zero at hash power 4. Guarded the divisor instead of rewriting it to match its siblings: the division looks deliberate, and the algorithm is not reachable from the CLI, so there is no call to change its sizing on a guess. Also added the NaN sample rate case from #325 to testCLI, which arrived with the fix but without a test. testCLI is now 162 checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9007e88676
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| cache_t *cache = create_cache_by_name(cache_alg_name, cc_params, | ||
| (const char *)cache_specific_params); | ||
| if (cache != NULL) { | ||
| return cache; |
There was a problem hiding this comment.
Use the full cache size when MINISIM skips sampling
When MINISIM is run with FIX_RATE above 0.5, this change now lets built-in algorithms execute instead of aborting, but MRCProfilerMINISIM::run() disables the sampler while still constructing caches with mrc_size_vec[i] * sample_rate. For example, a rate of 0.75 processes the complete trace against caches at only 75% of the requested sizes, shifting the reported MRC. Set the effective rate to 1 when the no-sampling path is selected.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 2f16934. The accounting further down already branched on whether a sampler exists; the sizing did not, so the two disagreed above 0.5.
Quantified it before fixing — a 100MB point at rate 0.75 was measuring a 75MB cache:
cachesim lru 75mb -> 0.8257 # what the curve actually reported
cachesim lru 100mb -> 0.8225 # what it was labelled
Setting the effective rate to 1 on the no-sampling path makes unsampled MINISIM exact rather than approximate, which is a stronger property to test than "close enough":
MINISIM(0.75) cachesim
100MB 0.822467 0.8225
300MB 0.766150 0.7661
500MB 0.718500 0.7185
testCLI now asserts that equality directly rather than just a zero exit, so the mislabelling cannot come back quietly. Rates 0.6, 0.75 and 1 are covered.
Generated by Claude Code
| {"belady", Belady_init}, | ||
| {"beladySize", BeladySize_init}, |
There was a problem hiding this comment.
Reject oracle-only policies before MINISIM runs
When mrcProfiler uses MINISIM with --algo=belady or beladySize on a non-oracle trace, these new registry entries construct the policy without the trace-type validation performed by cachesim. Ordinary readers leave next_access_vtime at -2, so Belady aborts under debug assertions and consumes invalid future-access priorities in release builds, yielding an invalid MRC. Validate the reader type in the MINISIM path before allowing these policies.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct — fixed in 2f16934. Reproduced it first: --algo=belady --profiler=MINISIM on the vscsi sample trace ran to completion and printed a curve, exiting 0, while cachesim belady on the same trace warns and refuses. A plausible-looking answer from unset next_access_vtime is worse than a crash, since nothing signals it is wrong.
I put the check in the MINISIM path rather than removing the two entries from the registry. The registry answers "is this a built-in algorithm", and belady is one; the oracle-trace requirement is a usage constraint for the caller, which is how the header already documents it. Removing them would have made the registry quietly incomplete and pushed the same problem onto the next caller.
It checks reader_->trace_type against ORACLE_GENERAL_TRACE and LCS_TRACE rather than sniffing the path for "oracleGeneral" the way cache_init.h does — the reader already knows its own type, so there is no reason to infer it from a filename here.
$ mrcProfiler ../data/cloudPhysicsIO.vscsi vscsi --algo=belady --profiler=MINISIM ...
[ERROR] belady needs future information, so it only works on oracleGeneral and
lcs traces; convert with ./bin/traceConv
$ mrcProfiler ../data/cloudPhysicsIO.oracleGeneral.bin oracleGeneral --algo=belady --profiler=MINISIM ...
rc=0
testCLI covers both policies against both trace types, so the rejection and the acceptance are each pinned.
Generated by Claude Code
Two follow-ups from Codex, both reachable only because MINISIM now runs at all rather than aborting. Above a sample rate of 0.5 MINISIM stops sampling and replays the whole trace, but it still sized the miniature caches at mrc_size_vec[i] * sample_rate. Every request reached a cache smaller than the one the user asked for, so the curve was mislabelled: a 100MB point at rate 0.75 measured a 75MB cache, reporting 0.8257 where 100MB is 0.8225. The accounting further down already branched on whether a sampler exists; the sizing did not. Setting the effective rate to 1 on that path makes the two agree, and unsampled MINISIM now matches cachesim exactly — 0.822467, 0.766150 and 0.718500 against 0.8225, 0.7661 and 0.7185 at 100MB, 300MB and 500MB. belady and beladySize read next_access_vtime, which ordinary readers leave unset. cachesim refuses them on a non-oracle trace; MINISIM reached them through the registry without that check and produced a plausible-looking curve from garbage priorities, exiting 0. It now refuses them the same way, and still accepts them on oracleGeneral. testCLI gains the unsampled rates, an equality check against cachesim so the mislabelling cannot come back quietly, and both trace types for the two oracle-only policies. 170 checks, green plain and under LeakSanitizer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f1693427f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const reader_init_param_t *const reader_init_param) { | ||
| return setup_reader(path, type, reader_init_param); | ||
| } | ||
| Object ids are hashed unless you set `obj_id_is_num`, which you should do when the id field holds numbers. |
There was a problem hiding this comment.
Correct the documented object-ID default
When a caller follows the preceding instruction and initializes parameters with default_reader_init_params(), obj_id_is_num is already true, contrary to this sentence. For a CSV containing string IDs, leaving that default unchanged sends every value through strtoull() in csv_cb1; nonnumeric IDs become zero rather than being hashed, collapsing distinct objects and corrupting simulation results. Document that callers must set obj_id_is_num = false for string IDs, or change the default.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
You're right, and the consequence is worse than the doc being merely imprecise — fixed in 610d644.
Reproduced it against a csv with string ids, using exactly the defaults the doc pointed at:
obj_id_is_num default = true
row 1 -> obj_id=0 size=100 # alpha
row 2 -> obj_id=0 size=200 # beta
row 3 -> obj_id=0 size=100 # alpha
...
Six distinct objects collapsed onto one id. As you say, csv_cb1 tests obj_id_is_num first and only reaches the hashing branch on the false path, so strtoull() gets the string, warns, and leaves 0. The result is a plausible miss ratio rather than an error, which is the part that makes it dangerous.
The mismatch is specifically library-versus-CLI: cli_reader_utils.c overrides the flag to false right after calling set_default_reader_init_params(), so cachesim hashes string ids correctly, while a library caller taking default_reader_init_params() gets the opposite. Both doc/API.md and doc/advanced_lib.md now say so plainly, with the one-liner to set:
reader_init_param_t p = default_reader_init_params();
p.obj_id_is_num = false; /* string ids: hash them */Confirmed that fixes it — distinct ids now hash distinctly and the repeats (alpha at rows 1 and 3, beta at 2 and 5) map to matching values.
I documented rather than changed the default, since flipping it would alter behaviour for every existing library caller and true is plausibly right for the binary readers. Worth the maintainers deciding separately.
testCLI now drives a string-id csv and asserts the miss ratio that four distinct objects in six requests produce, so a regression to id 0 fails the test instead of returning a believable number.
Generated by Claude Code
| {"slru", SLRU_init}, | ||
| {"slruv0", SLRUv0_init}, | ||
| {"twoq", TwoQ_init}, | ||
| {"wtinyLFU", WTinyLFU_init}, |
There was a problem hiding this comment.
Register the cachesim
tinyLFU alias
When MINISIM or a library caller passes tinyLFU, this registry returns no constructor even though create_cache() explicitly accepts that cachesim alias. The plugin path then looks for a nonexistent tinyLFU_init/libtinyLFU.so and aborts, contradicting the newly documented promise that name-based lookup accepts the same aliases as the CLI. Add tinyLFU here with the CLI's default-window handling, or stop advertising equivalent names.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fair — I made that promise in the header and then did not keep it. Fixed in 610d644. Reproduced first:
cachesim tinyLFU -> rc=0
MINISIM tinyLFU -> rc=134 | cannot load ./libtinyLFU.so
You offered "add it with the CLI's default-window handling, or stop advertising equivalent names". I took the first, but the default-window handling turned out to be unnecessary — which is worth stating since it is the interesting part.
create_cache() appended ,window-size=0.01 when the caller gave parameters without a window. WTinyLFU's own DEFAULT_PARAMS is "main-cache=SLRU,window-size=0.01", and WTinyLFU_init parses that before applying caller parameters, so the append could only ever re-set a value that was already 0.01. Checked all four shapes rather than assuming:
| eviction params | tinyLFU vs wtinyLFU |
|---|---|
| (none) | identical |
main-cache=LRU |
identical |
window-size=0.2 |
identical |
main-cache=LRU,window-size=0.3 |
identical |
So it is a plain alias in the registry, and the special case is gone along with its malloc/snprintf. That leaves one code path instead of two and makes the name reachable from MINISIM.
testCLI now exercises tinyLFU in the all-algorithms sweep and through MINISIM.
Generated by Claude Code
Two more from Codex. The API docs had the object-id default backwards. They said ids are hashed unless you opt into numeric parsing, but default_reader_init_params() sets obj_id_is_num to true — the opposite of what cachesim does, which overrides it to false. csv_cb1 checks that flag first and only hashes on the false path, so a library caller who took the defaults with string ids got strtoull() and a warning per row: row 1 -> obj_id=0 row 2 -> obj_id=0 row 3 -> obj_id=0 Six distinct objects collapsed into one, and the resulting miss ratio is meaningless rather than obviously wrong. Both doc/API.md and doc/advanced_lib.md now state the real default and that string ids need it set to false; with that, the same trace hashes correctly and repeated keys map to the same id. The default itself is a library behaviour question, so it is documented rather than changed here. cachesim accepts tinyLFU, but the registry did not, so MINISIM and any library caller got "cannot load ./libtinyLFU.so" — contradicting the registry's documented promise to accept the CLI's aliases. Added as a plain alias, and removed the special case that used to append window-size=0.01 when the caller had not given one: WTinyLFU's DEFAULT_PARAMS already sets exactly that before applying caller parameters, so it never changed anything. Verified identical between tinyLFU and wtinyLFU with no params, with main-cache=LRU, with window-size=0.2, and with both. testCLI covers tinyLFU through cachesim and MINISIM, and a csv of string ids asserting the miss ratio that four distinct objects give, so collapsing them to id 0 shows up as a failure rather than a plausible number. 174 checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Two review findings from the previous round. The MINISIM belady guard only accepted oracleGeneral and lcs, but the four oracleTwr/oracleTwrNS readers populate req->next_access_vtime too, so belady works on those and was being rejected. Move the list into a helper next to the reader formats that set the field, and name the actual trace type in the error so it says what was wrong rather than only what is allowed. quickstart_traceAnalyzer.md embeds its plots with raw <img> tags pointing at /doc/plot/..., which resolve on github.com but became site-root paths in the built HTML, so all 18 images 404'd. Copy doc/plot and doc/assets into the output with html_extra_path and strip the prefix in the same source-read hook that already rewrites the Markdown links. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eb46342371
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| default: | ||
| return false; |
There was a problem hiding this comment.
Honor next-access fields in generic binary traces
When a caller opens a BIN_TRACE with reader_init_param_t.next_access_vtime_field configured, generalReader/binary.c explicitly populates req->next_access_vtime, so the trace contains the future information Belady needs. This default branch nevertheless classifies every generic binary reader as unsupported, causing MINISIM to abort on a valid oracle-style binary trace; base the check on the configured field or the reader's actual request data rather than only its trace-type enum.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 4466ab2. binary.c:263 does populate the field whenever next_access_vtime_field_idx > 0, so classifying by the enum alone was wrong.
The helper now takes the reader rather than the trace type, and asks it:
case BIN_TRACE:
return reader->init_params.next_access_vtime_field > 0;binary.c is the only general reader with such a field — csv has none — so BIN_TRACE is the sole addition.
No CLI exposes next_access_vtime_field, so this isn't reachable from the mrcProfiler binary. It is reachable from the library, since mrcProfiler_lib is folded into the main target, so I verified it that way: opened cloudPhysicsIO.oracleGeneral.bin as a plain BIN_TRACE with "<IQIq" and next_access_vtime_field = 4, then ran MINISIM belady through the library API.
trace_type=BIN_TRACE next_access_vtime_field=4
first req: obj_id=42932745 size=512 next_access_vtime=9223372036854775807
ACCEPTED: MINISIM belady ran on a BIN_TRACE
cache_size miss_rate
20971520B 0.814072
104857600B 0.734158
0.814072 matches the oracleGeneral run at the same size exactly, so it is the same curve rather than merely a curve. With the field left unset the same trace is still rejected (rc=134), and .vscsi still is too.
One thing that turned up while verifying, which I have not changed: on the first attempt belady aborted with next access time is -1, please use INT64_MAX instead. oracleGeneralBin_read_one_req() normalises the -1/INT64_MAX sentinel to MAX_REUSE_DISTANCE; the generic binary reader passes the raw value through. So a BIN_TRACE whose sentinel is -1 reaches Belady and is loudly rejected there. That is a pre-existing gap in binary.c rather than something this guard should paper over, and normalising it would change what the reader hands every caller — a maintainer call, so I left it and used an INT64_MAX-sentinel trace for the run above.
Generated by Claude Code
|
|
||
| New here? Start with [Install & Build](install.md), then [the cachesim guide](quickstart_cachesim.md). | ||
|
|
||
| The commands throughout these pages are run from the build directory (`_build/` if you followed the [README](https://github.com/1a1a11a/libCacheSim#build-and-install-libcachesim)), so the sample traces in `data/` are at `../data/`. |
There was a problem hiding this comment.
Scope the build-directory instruction to binary guides
This blanket instruction is false for several pages included in the new toctree. For example, quickstart_traceAnalyzer.md runs python3 scripts/traceAnalysis/... and debug.md runs ./scripts/debug.sh; from _build/ both paths are missing because scripts/ is in the repository root. Readers following the new site-wide cwd rule therefore get immediate file-not-found errors, so limit the statement to the binary quickstarts or update those commands to use paths relative to _build/.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct — that sentence was mine and it overclaimed. Fixed in 4466ab2.
Counted which pages use which convention rather than guessing:
| page | ../data/ (build dir) |
root-relative |
|---|---|---|
quickstart_cachesim.md |
32 | 0 |
quickstart_mrcProfiler.md |
7 | 1 |
quickstart_traceUtils.md |
4 | 0 |
quickstart_traceAnalyzer.md |
4 | 9 |
performance.md |
2 | 3 |
debug.md |
0 | 19 |
So both examples you named hold: debug.md is entirely repo-root (./scripts/debug.sh -- data/cloudPhysicsIO.vscsi ..., note data/ and not ../data/), and quickstart_traceAnalyzer.md is genuinely mixed — the analyzer runs from _build/ while its nine python3 scripts/traceAnalysis/*.py follow-ups run from the root.
I took the scoping option rather than rewriting the commands, since the split is inherent — the binaries live under _build/ and the scripts under scripts/ — and repointing nine plotting invocations I can't exercise here would risk more than it fixes. The note now reads:
Commands that invoke a built binary —
./bin/cachesim,./bin/traceAnalyzer,./bin/mrcProfiler— are run from the build directory (_build/…), which is why the sample traces indata/appear as../data/. The helper scripts underscripts/, such as the plotting scripts anddebug.sh, are run from the repository root instead, and those pages write the paths accordingly.
sphinx-build -W is clean and ctest is 10/10.
Generated by Claude Code
Two more review findings. The oracle-trace check classified readers by trace type alone, but the generic binary reader populates req->next_access_vtime whenever the caller points next_access_vtime_field at the right column, so a configured BIN_TRACE carries the future information Belady needs and was still being rejected. Ask the reader instead of assuming from the enum. No CLI exposes that field, but the profiler is part of the library, so a library caller can set it up. doc/index.md claimed every command in the docs runs from the build directory. That holds for the binaries, which is why the sample traces appear as ../data/, but not for the helper scripts: quickstart_traceAnalyzer.md invokes python3 scripts/traceAnalysis/*.py and debug.md invokes ./scripts/debug.sh, both relative to the repository root. Scope the sentence to what is true. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…plot Relicense from GPL-3.0 to Apache-2.0. LICENSE is now the canonical Apache text, byte-identical to apache.org, and the declarations that named GPL are updated to match: CITATION.cff, libCacheSim-node/package.json, the node README, CONTRIBUTING.md, and the README license section, which now names the license instead of only linking it. The vendored LHD code under libCacheSim/cache/eviction/LHD keeps its own MIT notice, which Apache-2.0 accommodates. The generic binary reader passed the "no next access" sentinel through raw, while the oracle readers normalize -1 to MAX_REUSE_DISTANCE. A binary trace carrying -1 therefore reached Belady, which rejects it outright. Normalize it in binary.c so the two readers agree: the same file read either way now produces an identical miss ratio curve. The popularity-decay section pointed at a plot that has never existed in the repo, which is why it was commented out; meanwhile the w92 plot generated for exactly that section sat unreferenced. Point the section at the real file and rewrite the caption, which had been copied from the object-size section and described two plots and the wrong quantity. All 18 images in the rendered docs now resolve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b0aab6a6f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /* these need future information and are only valid on oracle traces, so | ||
| * callers that know the trace type should check before using them */ | ||
| {"belady", Belady_init}, | ||
| {"beladySize", BeladySize_init}, |
There was a problem hiding this comment.
Preserve BeladySize's reduced hash table in MINISIM
When --algo=beladySize --profiler=MINISIM is used, this direct registry entry bypasses the BeladySize-specific reduction in bin/cachesim/cache_init.h, while MINISIM supplies hash power 20 to every profile point. BeladySize therefore allocates a 1M-slot table per miniature cache rather than the reduced power-16 table; with many profile points this can multiply memory use by 16 and trigger the algorithm's own “hash power too large” retry path or an OOM. Apply the same reduction before constructing BeladySize through the registry.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in b68dcee. Measured peak RSS for an 8-point MINISIM run before changing anything:
| algo | before | after |
|---|---|---|
beladySize |
32.0 MB | 10.2 MB |
belady |
14.8 MB | 14.9 MB |
lru |
13.7 MB | 14.7 MB |
So the effect is real and specific to BeladySize, as you said.
Two corrections to the diagnosis, both in your favour on the substance:
Hyperbolic is not affected, despite having the same reduction in cache_init.h. Hyperbolic_init shrinks its own table (MAX(12, hashpower - 8) at Hyperbolic.c:47), so it converges to 12 from either entry point — which is why it measured 10.2 MB going in. BeladySize has no such line, so it really is the only one.
The retry path isn't the mechanism. BeladySize_to_evict recurses when no sampled object beats the initial score, not when the table is sparse — chained_hashtable_rand_obj_v2 loops until it hits a non-empty bucket and shrinks the table itself after 32 misses, so it never hands back NULL. The sparse table costs memory and wasted probes, not the retry. I didn't observe the warning in any run.
On where to apply it. I first did the tidier thing and moved the reduction into BeladySize_init alongside Hyperbolic's, deleting it from cache_init.h — algebraically identical for cachesim at every hash power (MAX(hp-8,16) either way; verified byte-identical miss ratios at 18, 24 and 30 against a reverted build). But testEvictionAlgo then failed:
assertion failed (miss_cnt_true[i] == res[i].n_miss): (74329 == 74347)
That test builds BeladySize at hash power 20 — the same configuration you flagged — and records the unreduced numbers. Putting the reduction in the algorithm overrides a hash power the caller explicitly asked for, so it changes results for every library caller, not just MINISIM. That is a modelling decision rather than a defect, so I left it to you and scoped this commit to the profiler that had the problem.
Worth your eye as a follow-up: BeladySize and Hyperbolic disagree about whose job this is, and the library and the CLI consequently disagree about BeladySize's effective table size. Making BeladySize self-shrink would settle it, at the cost of re-recording that test's expected counts.
MINISIM's beladySize curve does move slightly, since the sampler now draws from a right-sized table (e.g. 0.629654 → 0.626142 at wss 0.091). Confirmed deterministic across three runs, so those deltas are the fix and not sampling noise. ctest is 10/10.
Generated by Claude Code
BeladySize picks its victim by sampling the hash table, so cachesim shrinks the table by 8 before constructing it. MINISIM builds its miniature caches straight from the registry at a fixed hash power of 20, so it skipped that and gave every profile point a 1M-slot table. Peak RSS for an 8-point run drops from 32.0 MB to 10.2 MB. Applied in the profiler rather than in BeladySize_init. Moving it into the algorithm, the way Hyperbolic_init does, is arguably where it belongs and would fix every caller at once, but it overrides the hash power a library caller explicitly asks for: test_evictionAlgo builds BeladySize at hash power 20 and its recorded miss counts move (74329 -> 74347). That is a modelling change for the library API rather than a defect, so it is left for the maintainers; this commit changes only the profiler that had the problem. Hyperbolic needs no equivalent line because Hyperbolic_init already shrinks its own table, which is why it was unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b68dcee1e5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (strcasecmp(params_.cache_algorithm_str, "belady") == 0 || | ||
| strcasecmp(params_.cache_algorithm_str, "beladySize") == 0) { |
There was a problem hiding this comment.
Remap BeladySize future times after spatial sampling
When MINISIM profiles beladySize with a sampling rate at or below 0.5, the reader filters the trace but leaves each oracle request's next_access_vtime in the original trace's coordinate system. BeladySize_to_evict() subtracts the sampled cache's n_req from that original timestamp when calculating its size-distance score, so the eviction priorities—and therefore the reported MRC—are invalid. Either remap future times to sampled virtual time or reject sampled BeladySize runs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Real, and the best finding of the review — thanks. Documented and warned in d946288.
I designed a control experiment rather than taking it on faith, since sampling always introduces error and the question is whether BeladySize has excess error. Your analysis predicts BeladySize is hit but Belady is not, because Belady uses next_access_vtime as an ordering and never takes a difference, so a constant coordinate offset cancels. MINISIM also skips sampling above rate 0.5, which gives a boundary to test across.
Error against the unsampled miss ratio, cloudPhysicsIO, 100 MB cache:
| sample rate | beladySize | belady | lru |
|---|---|---|---|
| 0.6 (no sampler) | 0.000002 | 0.000014 | 0.000033 |
| 0.5 (sampler on) | 0.0126 | 0.0003 | 0.0023 |
| 0.1 | 0.0245 | 0.0155 | 0.0156 |
At 0.5, where ordinary sampling error is still negligible, BeladySize is already 42x Belady's error and 5.5x LRU's, and it jumps discontinuously the moment the sampler switches on. Belady behaves exactly as your reasoning predicts. That is the coordinate mismatch, not sampling noise.
You offered "remap future times or reject sampled BeladySize runs". I did neither, and took a third option — warn loudly and document — for a reason worth stating.
Remapping is a change to the sampler's contract with its own accuracy trade-offs, and rejecting removes a capability over an error that is real but bounded (the curve is in the right region, just degraded). Both are modelling calls that belong to the maintainers. What was actually indefensible was reporting the number without saying it was degraded, and that is fixed: the profiler now warns when and only when the sampler is active, and doc/quickstart_mrcProfiler.md carries the mechanism and the measured numbers so anyone reading a sampled BeladySize curve knows what they have.
[WARN] beladySize scores candidates by reuse distance, which spatial sampling
distorts because next_access_vtime stays in full-trace request numbers; the
curve is approximate beyond the usual sampling error. Use
--profiler-params=FIX_RATE,1,<threads> for an exact run, or belady, which is
not affected.
Verified the warning fires under sampling and stays silent at FIX_RATE,1. ctest 10/10, sphinx-build -W clean.
Generated by Claude Code
| {"clockpro", ClockPro_init}, | ||
| {"fifo", FIFO_init}, | ||
| {"fifo-merge", FIFO_Merge_init}, | ||
| {"fifo-reinsertion", Clock_init}, |
There was a problem hiding this comment.
Map FIFO-reinsertion to its actual constructor
When MINISIM or a library caller requests fifo-reinsertion through the new name registry, this entry silently constructs a Clock cache even though FIFO_Reinsertion_init is available. Experiments therefore complete successfully while measuring the wrong eviction policy; register FIFO_Reinsertion_init for this name.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Checked this one against develop before touching it, and it isn't a regression — that mapping is pre-existing:
$ git show origin/develop:libCacheSim/bin/cachesim/cache_init.h | grep reinsertion
{"fifo-reinsertion", Clock_init},The registry is a verbatim move of cachesim's own table, so cachesim fifo-reinsertion has always built a Clock cache, and MINISIM now builds the same thing. Repointing the name at FIFO_Reinsertion_init would change what every existing fifo-reinsertion invocation simulates, which is exactly the kind of silent result change this PR has stayed away from — so it is yours to make, not mine to slip into a refactor whose whole claim is that it preserves behavior.
For what it's worth the alias is defensible on the merits: Clock is FIFO-with-reinsertion, and the two implementations are the same policy. So this may well be deliberate rather than a slip.
The thing genuinely worth your attention is the other half: FIFO_Reinsertion_init is reachable under no name at all, on develop or here — grep FIFO_Reinsertion against develop's table returns nothing. It is exported from evictionAlgo.h and used internally by WTinyLFU as a main-cache option, but no CLI or registry name resolves to it. So it is either dead as a standalone policy and could be documented as internal-only, or it wants its own registry entry under a distinct name. Both are one-line changes once you decide which.
Generated by Claude Code
BeladySize ranks candidates by next_access_vtime - cache->n_req. The first term counts requests in the full trace; the second counts only the requests the spatial sampler kept, so under sampling the reuse distance is inflated and the eviction order degrades. Belady is unaffected: it compares future times directly and never takes a difference. Measured on cloudPhysicsIO at a 100MB cache, error against the unsampled miss ratio: sample rate 0.5 beladySize 0.0126 belady 0.0003 lru 0.0023 sample rate 0.1 beladySize 0.0245 belady 0.0155 lru 0.0156 so at 0.5, where ordinary sampling error is still small, BeladySize is already an order of magnitude worse than either control. Warn and document rather than refuse or silently correct. Remapping future access times into sampled virtual time is a change to the sampler with its own accuracy trade-offs, which is the maintainers' call; what mattered here was that the profiler had been reporting the number without saying it was degraded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Split into five smaller PRsThis PR is now superseded — please review the five below instead. I am leaving it open until they land, since the review discussion here (15 threads, with the measurements behind each fix) is the fullest record of how these bugs were found.
#326, #327 and #328 are independent and can merge in any order. #329 and #330 stack on #328, which is why they are based on it rather than Splitting by file rather than by commit is what makes that work: every changed file belongs to exactly one PR, so nothing needs a hand-reconstructed intermediate state and no two PRs touch the same file. The cost is that #328 carries the hash-power clamps in the eviction algorithms whose motivation only appears in #329, and #327's mrcProfiler guide describes MINISIM behavior that #329 delivers. Verification that nothing was lost or altered in the splitMerging all five branches together and diffing against this PR's head: $ git merge claude/polish-1-repo-meta claude/polish-2-docs # onto the 3-4-5 stack
$ git diff --quiet <union> d946288 && echo IDENTICAL
IDENTICALThey merge without conflict and reproduce this branch byte for byte. Each was also built and tested on its own branch, not just as part of the whole: #328 and #329 are Still open for you, carried over
Generated by Claude Code |
…sim's hash power on exact MINISIM runs WTinyLFU_evict checked the main cache's capacity using cache->obj_md_size, which is the larger of the two sub-caches so that a caller asking the composite what it reserves is not told less than it really does. Against a FIFO or Clock main cache, which reserve nothing, that billed the window's 16 bytes and called the main cache full early; with an empty main cache it could reach to_evict() and dereference NULL. Each sub-cache is now charged its own. MINISIM built its miniature caches at hash power 20 even when sampling was disabled, while cachesim defaults to 24. Random, RandomTwo, RandomLRU and Hyperbolic draw eviction candidates through the hash mask, so the supposedly exact run drifted: randomTwo gave 0.8221 from cachesim and 0.8215 from MINISIM. Unsampled runs now use cachesim's hash power, including the extra reduction cachesim applies to hyperbolic and beladySize, and reproduce it exactly across all nine algorithms checked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…-repo-polish-wc1xjb # Conflicts: # README.md # doc/README.md
plugin.c leaked the dlopen handle when dlsym failed. Nothing from the library is in use on that path, unlike the success path the existing comment covers, so it is closed rather than leaked. testCLI now sweeps hyperbolic, belady and beladySize, which cachesim special-cases and the registry also carries; the trace is oracleGeneral, so the Belady policies are valid on it. 174 checks becomes 180. mktemp -d is given an explicit template, since BSD mktemp rejects the bare form and the macOS job runs this target. The ctest entry gains SKIP_REGULAR_EXPRESSION, so a run that skipped itself for want of binaries or traces no longer reports as a pass. CONTRIBUTING.md described the CLI test by linking test/test_cli.sh. That file arrives in a different PR of this series, so the link would dangle for anyone merging the community files alone; the guidance now leads with what to cover and why the library tests will not catch it. adoption.md, which arrived on develop in #323, carries a BibTeX block whose % comment Pygments' bibtex lexer rejects. Harmless normally, but this branch sets fail_on_warning, so it broke the docs build; the comment moves into prose above the block. adoption.md is also added to the Sphinx toctree, which it needs to be reachable in the rendered docs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5de5522a48
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ], | ||
| "author": "Murphy Tian", | ||
| "license": "MIT", | ||
| "license": "Apache-2.0", |
There was a problem hiding this comment.
Ship the Apache license in the npm package
Declaring the addon as Apache-2.0 is not enough when the published artifact does not contain the license text. I checked npm pack --dry-run --json --ignore-scripts in libCacheSim-node; the tarball contains only six files and no LICENSE, because the repository-level license is outside the package and is not copied by the release workflow. Consequently, every npm publication distributes the statically linked library without the Apache license copy required for redistribution; copy LICENSE into this package (and include it in files) before publishing.
Useful? React with 👍 / 👎.
The addon declares Apache-2.0 and binding.gyp statically links vendor/liblibCacheSim.a, so every publication redistributes the library. The repository LICENSE sits one directory above the package, outside anything npm collects, so the tarball carried the declaration without the text: npm pack --dry-run -> 6 files, no LICENSE Apache-2.0 section 4(a) requires giving recipients a copy of the license, so add one to the package and list it in "files". The tarball is now 7 files with LICENSE among them. The gap predates the relicense -- the same tarball shipped while the package declared GPL-3.0 -- but relicensing is the moment to fix it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc (cherry picked from commit d739e57)
_expect_clean_failure accepted any non-zero exit whose output contained the word "error". A sanitizer turns a crash into exit 1 with "ERROR: AddressSanitizer", so under the sanitizer builds this suite is meant to run in, a crash regression would have been counted as a clean rejection. Sanitizer signatures are now checked first. SKIP_REGULAR_EXPRESSION arrived in CMake 3.16 and this project declares 3.12, where setting it is an error, so it is now guarded by a version check. The documented csv reader used a bare designated initializer, which zeroes every field it does not name -- cap_at_n_req, block_size and ignore_size_zero_req all default to something other than zero. It also set has_header without has_header_set, and the reader auto-detects unless the flag is raised, so the assignment did nothing; it happened to work only because detection guessed right on this file. The snippet now starts from default_reader_init_params() and raises both _set flags. Compiled and run: it reads 113872 requests with first id 42932745 size 512, matching the vscsi reader exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc77b326f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| * full-sized; leaving the requested rate in place would scale them down | ||
| * while every request still reached them, reporting the miss ratios of | ||
| * smaller caches than were asked for */ | ||
| sample_rate = 1.0; |
There was a problem hiding this comment.
Scale MINISIM caches by the sampler's effective rate
This only normalizes the unsampled path; now that built-in MINISIM runs execute, sampled rates whose reciprocal is not an integer still produce biased curves. For example, FIX_RATE,0.3,... is quantized by create_spatial_sampler() to sampling_ratio_inv = 3, so spatial_sample() retains 1/3 of objects and miss counts are extrapolated by 3, while line 407 sizes every cache using the unquantized 0.3 rate. The sampled workload therefore sees caches about 10% smaller than the rate represented in the results; normalize sample_rate to 1.0 / sampler->sampling_ratio_inv after creating the sampler, or reject rates that cannot be represented.
Useful? React with 👍 / 👎.
create_spatial_sampler stores sampling_ratio_inv as an integer, so it can only
represent rates of the form 1/n: it truncates 1/0.3 to 3 and keeps a third of
objects. The cache sizing used the rate that was asked for rather than the one
in force, making every cache too small by that ratio -- 10% at 0.3 -- while
the curve was still reported against the requested size, so the error did not
show up anywhere in the output.
Demonstrable by asking for two rates that quantize to the same sampler: 0.3
and 0.3333333 both give sampling_ratio_inv 3 and therefore sample identical
objects, so any difference between their curves is the bug and nothing else.
before rate=0.3333333 0.8193 0.7637
rate=0.3 0.8204 0.7788
after rate=0.3333333 0.8193 0.7637
rate=0.3 0.8193 0.7637
Rates that are already of the form 1/n, including the 0.5 boundary and the
0.01 used throughout the docs, are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5b9472a17
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| struct { | ||
| uint32_t real_time; | ||
| uint32_t clock_time; | ||
| uint64_t obj_id; | ||
| uint32_t obj_size; | ||
| int64_t next_access_vtime; | ||
| int64_t next_access_vtime; // -1 if there is no next access |
There was a problem hiding this comment.
Mark the oracleGeneral record as packed
When a reader uses this struct definition to generate an oracleGeneral trace, normal 64-bit C ABI padding places obj_id at offset 8 and makes the record 32 bytes, while oracleGeneralBin_setup() expects a packed 24-byte record with offsets 0, 4, 12, and 16. The resulting file is silently decoded with the wrong IDs, sizes, and future times; show __attribute__((packed)) or document the exact serialized offsets instead.
Useful? React with 👍 / 👎.
|
|
||
| In the sample [cloudPhysicsIO.csv](/data/cloudPhysicsIO.csv), time is in seconds and object size is in bytes. | ||
|
|
||
| `next_access_vtime` is a *logical* time: the number of requests between the current request and the next request to the same object, or `-1` when the object is never accessed again. Algorithms that need future information, such as [Belady](/libCacheSim/cache/eviction/Belady.c) and BeladySize, rely on it, which is why they only work on oracle traces. |
There was a problem hiding this comment.
Document next access as an absolute virtual time
For anyone constructing an oracle trace, this describes the field as a reuse distance, but traceConvOracleGeneral stores the absolute request index of the next access and BeladySize explicitly computes the distance by subtracting the current request count. Encoding the documented distance therefore changes eviction priorities and yields invalid simulation results; describe it as the next access's absolute 1-based virtual timestamp, with -1 for no future access.
Useful? React with 👍 / 👎.
cache_get_base()'s eviction loop charges an incoming object cache->obj_md_size before deciding whether to evict. Every incoming object is inserted into the window, so that field has to be the window's overhead; setting it to the maximum of the two sub-caches made the loop reserve up to 40 bytes for a 16-byte window insertion whenever the main policy was ARC, LeCaR or Cacheus, evicting earlier than the configuration calls for. All three sites now charge the cache the object is actually entering: this one and WTinyLFU_can_insert() the window, WTinyLFU_evict() the main cache on promotion. Measured on cloudPhysicsIO at 100MB with metadata on, the effect is small -- LeCaR moves 0.7894 to 0.7895 and ARC, LRU, SLRU and FIFO do not move at all, because objects are large relative to the difference. Correct rather than consequential, but it also removes the last place where the composite charged an overhead belonging to a cache the object was not entering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e2a30b22a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (main_cache->get_occupied_byte(main_cache) + | ||
| params->req_local->obj_size + cache->obj_md_size <= | ||
| params->req_local->obj_size + main_cache->obj_md_size <= | ||
| main_cache->cache_size) { | ||
| main_cache->insert(main_cache, params->req_local); |
There was a problem hiding this comment.
Stop evicting once promotion frees enough metadata
When consider_obj_metadata=true and the main policy has less metadata than the window LRU (for example, main-cache=FIFO), moving a victim from the window into the main cache can itself reduce total occupancy by 16 bytes and make enough room for the pending request. This branch nevertheless leaves evicted false, so the loop continues until it discards an additional object, causing WTinyLFU to underfill the cache and alter miss ratios. Recheck the composite capacity after the promotion and return when the pending window insertion now fits.
Useful? React with 👍 / 👎.
The sweeps skipped any algorithm whose run reported "do not support algorithm", on the assumption that the message means an optional feature was not compiled in. cache_init.h emits the same message when a name is missing from the registry for any reason, so a mandatory algorithm silently dropping out of g_cache_algos would have been skipped rather than reported -- the exact regression the registry refactor could introduce, and the one this sweep is here to catch. Only 3LCache, GLCache, gl-cache and lrb sit behind build flags, so the skip is restricted to those and everything else fails. Verified by deleting an entry and rebuilding. gdsf is covered by the sweeps and nothing else, so before this it disappeared without a sound: FAIL: gdsf -e print (exit 134) do not support algorithm gdsf FAIL: gdsf replay (exit 134) do not support algorithm gdsf (4 algorithms not compiled in, skipped) The four genuinely optional ones still skip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…-repo-polish-wc1xjb # Conflicts: # libCacheSim/bin/cachesim/cache_init.h
MultiQueue arrived on develop in #318 with the same leak this branch fixed in 29 other files: MQ_parse_params strdup's the parameter string and frees it at the end, but the "print" branch calls exit(0) first. LeakSanitizer reports 6 bytes, the length of "print". Found by CI rather than by reading: merging develop added mq and multiqueue to the registry, the testCLI sweep covers every registered name, and the ubuntu job runs it under LeakSanitizer: FAIL: mq -e print (exit 23) SUMMARY: LeakSanitizer: 6 byte(s) leaked in 1 allocation(s). FAIL: multiqueue -e print (exit 23) which is the sweep doing exactly what it was added for -- a brand-new algorithm inherited the defect the same day it landed, and nothing else in the suite would have looked at it. Reproduced and confirmed fixed in a local -fsanitize=leak build with CI's ASAN_OPTIONS: ctest 10/10, testCLI 184 checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
cache_struct_init() reads a non-positive hashpower as "caller did not choose", substituting HASH_POWER_DEFAULT. The clamps added for --hashpower ran before that check, so MAX(4, 0 - 4) turned the sentinel into a literal hash power of 4 -- a 16-bucket table -- for any library caller using a designated initializer without setting the field. The reduction now applies only to an explicitly positive value, in Cacheus, S3FIFOd, SLRUv0 and LP_SFIFO alike. Worth recording that the predicted consequence was backwards: restoring the sentinel makes the sample replay *slower*, 54ms to 301ms, because HASH_POWER_DEFAULT allocates 8M buckets per sub-cache while a 16-bucket table simply grows. The point stands anyway -- 301ms is what the library did before this branch, and silently redefining an unset hashpower is not a change to make in passing. The bug the clamps were added for is still fixed: cachesim slruv0 memory stays monotonic across --hashpower 4, 5, 6, 8 and 24, with no inversion. Also: FAQ.md described next_access_vtime as the number of requests until the next access. It is the absolute 1-based request index of that access, and algorithms subtract the current request count themselves, so a trace built to the documented meaning would evict in the wrong order. Verified against the sample: request 7 stores 19, and that object is next requested at request 19. And testCLI's unsampled-MINISIM equality check compared two extracted strings without requiring either to be non-empty, so if both extractions stopped matching it would have passed on "" == "" -- the same shape as the skip logic fixed earlier today, an assertion that can succeed without observing anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
What does this PR do?
A polish pass over the repo for open-source readiness. Verifying the documentation turned up real bugs, and the tests added for those turned up more, so this is mostly correctness work with the community files alongside.
Documentation pointed at files that don't exist
doc/referenceddata/trace.vscsi,data/trace.csv,data/trace.txtanddata/trace.oracleGeneralin 45 places. The sample traces aredata/cloudPhysicsIO.*, so every command a new user copied out of the quickstart guides failed on a fresh clone. The README was fixed in #321; this does the same fordoc/. Commands also ran as./cachesimwith a../datapath that only resolves one directory up from the binary, so they now use./bin/<tool>from the build directory to match the README.The library reader examples in
doc/advanced_lib.mddidn't work either — compiling them foundobj_id_fieldset to 6 whencloudPhysicsIO.csvputs the id in field 5 (caught in review by Codex),has_headerfalse for a file with a header, and.binary_fmt(not a field — it's.binary_fmt_str) with"<3I2H2Q", which errors withunknown format '3'since the format parser has no repeat counts.Crashes, all reachable from documented commands
cachesim … slru/qdlp/s3fifod/s3fifov0/flashProb … -e printparse_paramsruns before it existstraceAnalyzer -o my-output …-odeclaredOPTION_ARG_OPTIONAL, so argp passesarg == NULLfor the space-separated form andstrncpydereferences itmrcProfiler -o out …--algo,--size,--profiler,--profiler-paramstraceAnalyzer --verbose …is_true(NULL)→strcasecmp(NULL, …)cachesim … slru … -e n-seg=0SLRU_init;seg-sizealso wrote pastseg_size_array[SLRU_MAX_N_SEG]past 16 segmentscachesim … wtinyLFU … --consider-obj-metadata=trueWTinyLFU_initreadparams->main_cache->obj_md_sizefrom amalloc'd, un-memsetstruct beforemain_cachewas assignedmrcProfiler --profiler=MINISIM …dlsym()against its own executable, but the constructors sit in an unreferenced archive member the linker never pulls in, so every run aborted withundefined symbol: FIFO_initMemory leaks
The ubuntu job builds with LeakSanitizer, and the new
testCLIruns the binaries to completion, reaching code the C unit tests never did:-e printearly exit leaked thestrdup'd parameter string in 29 files —exit(0)before the free at the end of the parser. GLCache, Mithril and PG never kept the original pointer at all, so they leaked on every call.Size_free,RandomLRU_freeandSLRUv0_freenever freed their params struct;S3FIFOd_freefreed three of its five sub-caches, leaving 73 KB;SLRUv0_coolallocated a request and early-returned oni == 0without freeing it, once per eviction from the bottom segment;WTinyLFU_freenever freed its params.-e printis documented for every algorithm, but Clock2QPlus and pluginCache rejected the bareprintkey before reaching their own print branch, and WTinyLFU had no print branch at all despite taking parameters. All three now behave like the rest.Hash table sizing
cachesimshrank its hash table when the trace path containeddata/trace., a file that has not existed for a long time, so the saving never happened. Repointing the string is not equivalent — sampling-based algorithms draw candidates from the hash table, so RandomLRU and hyperbolic move when it resizes — and sizing a table by sniffing a filename is the wrong mechanism for something that changes results. Replaced with--hashpower, default 24 as before. Replaying the sample trace goes from 106 MB to 15 MB at 20 and 8 MB at 16.Codex then found that exposing it surfaced a latent bug: several composite policies size their sub-caches by subtracting from the parent's hash power with no lower bound, and
cache_struct_initreads zero as "unset" and substitutes the full-size default — soslruv0 --hashpower=4took 18 MB against 6 MB at 5. Asking for a smaller table allocated a bigger one.Cacheus,S3FIFOd,SLRUv0andLP_SFIFOnow clamp likeHyperbolic,Random,RandomTwoandRandomLRUalready did.WTinyLFU object metadata
WTinyLFU_can_insert()charged the main cache's per-object overhead against the window LRU's size. The two disagree — LRU and SLRU reserve 16 bytes, FIFO and Clock none — somain-cache=FIFOcharged the window nothing for an overhead it does reserve. Each sub-cache is now checked against its own. Only reachable at all since the uninitialized-read fix above.The macOS build was already broken on
developSince the runner image moved to Xcode 26.6, clang rejects two implicit
UINT64_MAXtodoubleconversions inmrcProfiler.cppunder-Werror. The last threedevelopruns were red including docs-only commits. Fixed here first; #325 landed the same fix upstream with a clearerkHashSpaceSizeconstant plus a NaN sample-rate check, so this branch mergesdevelopand keeps that version. Its NaN case now has a test, which it arrived without.Read the Docs never built
.readthedocs.yamlpointed Sphinx atdocs/conf.py— there is nodocs/directory, noconf.pyand no.rstanywhere, so every build failed immediately. Wired up Sphinx over the existing Markdown with MyST, so the sources stay readable on GitHub with no duplication. Getting to zero warnings (from 96, withfail_on_warningnow set) meant fixing the docs themselves:doc/API.mdhad no headings at all and had drifted badly from the headers (reader_init_param_tmissing most fields, the samebinary_fmt/binary_fmt_strerror, asim_res_treturn type that no longer exists, trailing off at an empty "profiler:" heading);advanced_lib.mddocumented the same dead simulator API including areader_t reader*typo;performance.mdwas two empty sections and one bullet.Trace-analysis plots and the binary reader
Two follow-ups from review, both verified before changing anything.
quickstart_traceAnalyzer.mdembeds its plots with raw'<img src="/doc/plot/...">'tags, which resolve on github.com but became site-root paths in the built HTML, so all 18 images 404'd on the docs site. Fixed by copyingdoc/plotanddoc/assetsinto the output and stripping the prefix in the same hook that rewrites the Markdown links. That left one reference unresolved —twitter_cluster52_10m_popularityDecayLineLog.svg, which has never existed in the repo, and was inert only because the whole block sat inside an HTML comment. Meanwhilew92_popularityDecayLineLog.svg, generated for exactly that section, sat unreferenced. The section now points at the real file, and the caption is rewritten: it had been copied from the object-size section, describing two plots where the markup had one and the wrong quantity besides. 18 of 18 images now resolve.generalReader/binary.cpopulatesreq->next_access_vtimewhenever the caller setsnext_access_vtime_field, but passed the "no next access" sentinel through raw, where the oracle readers normalize-1toMAX_REUSE_DISTANCE. A binary trace carrying-1therefore reached Belady, which rejects it outright. Normalized, so the two readers agree — the sample trace read asoracleGeneraland as a plainBIN_TRACEwith"<IQIq"now yields an identical curve (0.814072at 20 MB either way). The MINISIM oracle-trace guard was widened alongside it: it accepted onlyoracleGeneralandlcs, but the fouroracleTwr/oracleTwrNSreaders set the field too, and a configuredBIN_TRACEdoes as well, so it asks the reader now instead of matching on the trace-type enum.Relicensed to Apache-2.0
At the maintainer's direction, the project moves from GPL-3.0 to Apache-2.0.
LICENSEis the canonical Apache text, byte-identical to the copy served by apache.org, and every declaration that named GPL now matches it:CITATION.cff,libCacheSim-node/package.json, the node README,CONTRIBUTING.md, and the README license section, which named no license at all and now does. NoGPLreference remains anywhere in the tree.The vendored LHD code under
libCacheSim/cache/eviction/LHDis MIT (CMU, 2017-2018) and keeps its ownLICENSE, which is both what MIT requires and compatible with Apache-2.0.Two notes for the record. Relicensing needs the agreement of everyone holding copyright in the existing code — 14 authors appear in the visible history — so that consent is assumed here rather than established by this PR. And
LICENSEkeeps Apache's appendix placeholder (Copyright [yyyy] [name of copyright owner]) verbatim, as apache.org ships it and as most Apache-licensed projects do, rather than my guessing a start year the shallow clone can't show.Community files, hygiene
libCacheSim-node/package.jsondeclaredMITwhile the project was GPL-3.0 andbinding.gyplinksvendor/liblibCacheSim.astatically; the package README said "MIT License - see the LICENSE file", where that file was the repository's GPLv3 — contradicting itself in one sentence. Both now match the project.Added
CONTRIBUTING.md,CODE_OF_CONDUCT.md(Contributor Covenant 2.1),.github/PULL_REQUEST_TEMPLATE.mdandCITATION.cff; rewroteSECURITY.md, which was GitHub boilerplate with its instructional comments still in it and no reporting channel, around GitHub private vulnerability reporting. Removed scratch files nothing referenced (test.c,random/,doc/TODO,scripts/note, an empty rootpackage-lock.json); stopped.gitignore's.vscode/*rule contradicting the four.vscode/*.jsonfiles deliberately checked in; added.editorconfig. Replacedactions/create-release@v1(archived by GitHub in 2021) withgh release create, bumped a straycheckout@v3, addedpersist-credentials: falseto all checkouts, and fixed the npmhomepagepointing into a nonexistentmainbranch.Type of change
--hashpower), replacing a heuristic that keyed off the trace pathHow was it tested?
New
testCLItarget (test/test_cli.sh) — 174 checks covering what nothing did before. It sweeps all 45 registered algorithms rather than a hand-picked list — that is hows3fifov0andflashProbwere found — and replays a trace with each rather than only parsing parameters, which is what surfaced the missing frees, since-e printexits before teardown and LSan still sees that memory as reachable. Invalid input is asserted to fail cleanly: non-zero with a message, but not SIGSEGV/SIGFPE/SIGBUS, since the project'sERROR()aborts and a deliberate rejection must stay distinguishable from a crash.To confirm the tests are load-bearing rather than merely green, I reverted the six originally-fixed sources to
developand rebuilt: 16 failures, 0 with them restored.Every runnable command in the quickstart docs was extracted and executed — 45/45 pass, up from 33 before (MINISIM's examples work now). Every repo-relative markdown link resolves; five were broken. The two reader snippets were compiled and run, and all three readers now agree exactly, which is what makes the corrected column parameters trustworthy:
Docs build verified with
sphinx-build -W: 13 pages, zero warnings, and I checked the generated HTML rewrites repo links to GitHub, keeps intra-doc links local, renders the mermaid diagrams, resolves all 18 images, and leaves no root-absolutehrefbehind.ctest --output-on-failurepasses — 10/10, both plain Release and under-fsanitize=leakwith CI'sASAN_OPTIONS-Wall -Wextra -Werror)Results
No behavior change for simulations. Verified by diffing against a
developbuild: 28 deterministic algorithms produce identical miss ratios, at the default hash power and after the registry refactor.lecarv0,cacheus,fifo-mergeandRandomLRUvary run to run or build to build on their own —FIFO_Mergehas an explicitnext_rand()tiebreaker — so they were compared against themselves rather than across builds.The intended behavior changes are all paths that previously crashed or were self-contradictory:
wtinyLFU --consider-obj-metadata=truegoes from SIGSEGV to a resultmrcProfiler --profiler=MINISIMgoes from aborting to workingBIN_TRACEwith a-1sentinel goes from aborting to a curve matching the oracle reader exactlymain-cache=FIFOand metadata on moves 0.8619 → 0.8070, andclock0.7732 → 0.7733, since the window is now charged its own overhead. With metadata off nothing changes; LRU, SLRU, sieve and ARC are unchanged either way.Checklist
clang-format(--dry-run --Werroron every modified source)One item left for you, since changing it would alter what the reader hands every caller:
generalReader/binary.cnow normalizes a-1next-access sentinel toMAX_REUSE_DISTANCEto match the oracle readers, but the same file makes no attempt to reconcile any other convention a third-party binary trace might use.🤖 Generated with Claude Code
https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc