Skip to content

Polish the repo for open-source readiness (relicense to Apache-2.0, fix documented crashes and leaks) - #324

Open
1a1a11a wants to merge 36 commits into
developfrom
claude/open-source-repo-polish-wc1xjb
Open

Polish the repo for open-source readiness (relicense to Apache-2.0, fix documented crashes and leaks)#324
1a1a11a wants to merge 36 commits into
developfrom
claude/open-source-repo-polish-wc1xjb

Conversation

@1a1a11a

@1a1a11a 1a1a11a commented Aug 13, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

A polish pass over the repo for open-source readiness. Verifying the documentation turned up real bugs, and the tests added for those turned up more, so this is mostly correctness work with the community files alongside.

Documentation pointed at files that don't exist

doc/ referenced data/trace.vscsi, data/trace.csv, data/trace.txt and data/trace.oracleGeneral in 45 places. The sample traces are data/cloudPhysicsIO.*, so every command a new user copied out of the quickstart guides failed on a fresh clone. The README was fixed in #321; this does the same for doc/. Commands also ran as ./cachesim with a ../data path that only resolves one directory up from the binary, so they now use ./bin/<tool> from the build directory to match the README.

The library reader examples in doc/advanced_lib.md didn't work either — compiling them found obj_id_field set to 6 when cloudPhysicsIO.csv puts the id in field 5 (caught in review by Codex), has_header false for a file with a header, and .binary_fmt (not a field — it's .binary_fmt_str) with "<3I2H2Q", which errors with unknown format '3' since the format parser has no repeat counts.

Crashes, all reachable from documented commands

Command Cause
cachesim … slru/qdlp/s3fifod/s3fifov0/flashProb … -e print reporting path dereferences state parse_params runs before it exists
traceAnalyzer -o my-output … -o declared OPTION_ARG_OPTIONAL, so argp passes arg == NULL for the space-separated form and strncpy dereferences it
mrcProfiler -o out … same, plus --algo, --size, --profiler, --profiler-params
traceAnalyzer --verbose … is_true(NULL)strcasecmp(NULL, …)
cachesim … slru … -e n-seg=0 divide by zero in SLRU_init; seg-size also wrote past seg_size_array[SLRU_MAX_N_SEG] past 16 segments
cachesim … wtinyLFU … --consider-obj-metadata=true WTinyLFU_init read params->main_cache->obj_md_size from a malloc'd, un-memset struct before main_cache was assigned
mrcProfiler --profiler=MINISIM … resolved eviction algorithms with dlsym() against its own executable, but the constructors sit in an unreferenced archive member the linker never pulls in, so every run aborted with undefined symbol: FIFO_init

Memory leaks

The ubuntu job builds with LeakSanitizer, and the new testCLI runs the binaries to completion, reaching code the C unit tests never did:

  • -e print early exit leaked the strdup'd parameter string in 29 files — exit(0) before the free at the end of the parser. GLCache, Mithril and PG never kept the original pointer at all, so they leaked on every call.
  • Missing frees, visible only when a cache is torn down: Size_free, RandomLRU_free and SLRUv0_free never freed their params struct; S3FIFOd_free freed three of its five sub-caches, leaving 73 KB; SLRUv0_cool allocated a request and early-returned on i == 0 without freeing it, once per eviction from the bottom segment; WTinyLFU_free never freed its params.

-e print is documented for every algorithm, but Clock2QPlus and pluginCache rejected the bare print key before reaching their own print branch, and WTinyLFU had no print branch at all despite taking parameters. All three now behave like the rest.

Hash table sizing

cachesim shrank its hash table when the trace path contained data/trace., a file that has not existed for a long time, so the saving never happened. Repointing the string is not equivalent — sampling-based algorithms draw candidates from the hash table, so RandomLRU and hyperbolic move when it resizes — and sizing a table by sniffing a filename is the wrong mechanism for something that changes results. Replaced with --hashpower, default 24 as before. Replaying the sample trace goes from 106 MB to 15 MB at 20 and 8 MB at 16.

Codex then found that exposing it surfaced a latent bug: several composite policies size their sub-caches by subtracting from the parent's hash power with no lower bound, and cache_struct_init reads zero as "unset" and substitutes the full-size default — so slruv0 --hashpower=4 took 18 MB against 6 MB at 5. Asking for a smaller table allocated a bigger one. Cacheus, S3FIFOd, SLRUv0 and LP_SFIFO now clamp like Hyperbolic, Random, RandomTwo and RandomLRU already did.

WTinyLFU object metadata

WTinyLFU_can_insert() charged the main cache's per-object overhead against the window LRU's size. The two disagree — LRU and SLRU reserve 16 bytes, FIFO and Clock none — so main-cache=FIFO charged the window nothing for an overhead it does reserve. Each sub-cache is now checked against its own. Only reachable at all since the uninitialized-read fix above.

The macOS build was already broken on develop

Since the runner image moved to Xcode 26.6, clang rejects two implicit UINT64_MAX to double conversions in mrcProfiler.cpp under -Werror. The last three develop runs were red including docs-only commits. Fixed here first; #325 landed the same fix upstream with a clearer kHashSpaceSize constant plus a NaN sample-rate check, so this branch merges develop and keeps that version. Its NaN case now has a test, which it arrived without.

Read the Docs never built

.readthedocs.yaml pointed Sphinx at docs/conf.py — there is no docs/ directory, no conf.py and no .rst anywhere, so every build failed immediately. Wired up Sphinx over the existing Markdown with MyST, so the sources stay readable on GitHub with no duplication. Getting to zero warnings (from 96, with fail_on_warning now set) meant fixing the docs themselves: doc/API.md had no headings at all and had drifted badly from the headers (reader_init_param_t missing most fields, the same binary_fmt/binary_fmt_str error, a sim_res_t return type that no longer exists, trailing off at an empty "profiler:" heading); advanced_lib.md documented the same dead simulator API including a reader_t reader* typo; performance.md was two empty sections and one bullet.

Trace-analysis plots and the binary reader

Two follow-ups from review, both verified before changing anything.

quickstart_traceAnalyzer.md embeds its plots with raw '&lt;img src="/doc/plot/..."&gt;' tags, which resolve on github.com but became site-root paths in the built HTML, so all 18 images 404'd on the docs site. Fixed by copying doc/plot and doc/assets into the output and stripping the prefix in the same hook that rewrites the Markdown links. That left one reference unresolved — twitter_cluster52_10m_popularityDecayLineLog.svg, which has never existed in the repo, and was inert only because the whole block sat inside an HTML comment. Meanwhile w92_popularityDecayLineLog.svg, generated for exactly that section, sat unreferenced. The section now points at the real file, and the caption is rewritten: it had been copied from the object-size section, describing two plots where the markup had one and the wrong quantity besides. 18 of 18 images now resolve.

generalReader/binary.c populates req->next_access_vtime whenever the caller sets next_access_vtime_field, but passed the "no next access" sentinel through raw, where the oracle readers normalize -1 to MAX_REUSE_DISTANCE. A binary trace carrying -1 therefore reached Belady, which rejects it outright. Normalized, so the two readers agree — the sample trace read as oracleGeneral and as a plain BIN_TRACE with "<IQIq" now yields an identical curve (0.814072 at 20 MB either way). The MINISIM oracle-trace guard was widened alongside it: it accepted only oracleGeneral and lcs, but the four oracleTwr/oracleTwrNS readers set the field too, and a configured BIN_TRACE does as well, so it asks the reader now instead of matching on the trace-type enum.

Relicensed to Apache-2.0

At the maintainer's direction, the project moves from GPL-3.0 to Apache-2.0. LICENSE is the canonical Apache text, byte-identical to the copy served by apache.org, and every declaration that named GPL now matches it: CITATION.cff, libCacheSim-node/package.json, the node README, CONTRIBUTING.md, and the README license section, which named no license at all and now does. No GPL reference remains anywhere in the tree.

The vendored LHD code under libCacheSim/cache/eviction/LHD is MIT (CMU, 2017-2018) and keeps its own LICENSE, which is both what MIT requires and compatible with Apache-2.0.

Two notes for the record. Relicensing needs the agreement of everyone holding copyright in the existing code — 14 authors appear in the visible history — so that consent is assumed here rather than established by this PR. And LICENSE keeps Apache's appendix placeholder (Copyright [yyyy] [name of copyright owner]) verbatim, as apache.org ships it and as most Apache-licensed projects do, rather than my guessing a start year the shallow clone can't show.

Community files, hygiene

libCacheSim-node/package.json declared MIT while the project was GPL-3.0 and binding.gyp links vendor/liblibCacheSim.a statically; the package README said "MIT License - see the LICENSE file", where that file was the repository's GPLv3 — contradicting itself in one sentence. Both now match the project.

Added CONTRIBUTING.md, CODE_OF_CONDUCT.md (Contributor Covenant 2.1), .github/PULL_REQUEST_TEMPLATE.md and CITATION.cff; rewrote SECURITY.md, which was GitHub boilerplate with its instructional comments still in it and no reporting channel, around GitHub private vulnerability reporting. Removed scratch files nothing referenced (test.c, random/, doc/TODO, scripts/note, an empty root package-lock.json); stopped .gitignore's .vscode/* rule contradicting the four .vscode/*.json files deliberately checked in; added .editorconfig. Replaced actions/create-release@v1 (archived by GitHub in 2021) with gh release create, bumped a stray checkout@v3, added persist-credentials: false to all checkouts, and fixed the npm homepage pointing into a nonexistent main branch.

Type of change

  • Bug fix
  • New eviction / admission / prefetch algorithm
  • New trace reader or trace format
  • Performance improvement
  • Documentation
  • Build / CI
  • Other: relicense to Apache-2.0, and one new CLI option (--hashpower), replacing a heuristic that keyed off the trace path

How was it tested?

New testCLI target (test/test_cli.sh) — 174 checks covering what nothing did before. It sweeps all 45 registered algorithms rather than a hand-picked list — that is how s3fifov0 and flashProb were found — and replays a trace with each rather than only parsing parameters, which is what surfaced the missing frees, since -e print exits before teardown and LSan still sees that memory as reachable. Invalid input is asserted to fail cleanly: non-zero with a message, but not SIGSEGV/SIGFPE/SIGBUS, since the project's ERROR() aborts and a deliberate rejection must stay distinguishable from a crash.

To confirm the tests are load-bearing rather than merely green, I reverted the six originally-fixed sources to develop and rebuilt: 16 failures, 0 with them restored.

Every runnable command in the quickstart docs was extracted and executed — 45/45 pass, up from 33 before (MINISIM's examples work now). Every repo-relative markdown link resolves; five were broken. The two reader snippets were compiled and run, and all three readers now agree exactly, which is what makes the corrected column parameters trustworthy:

vscsi              n_req=113872   miss_ratio=0.6297  first(id=42932745,size=512)
csv (doc)          n_req=113872   miss_ratio=0.6297  first(id=42932745,size=512)
binary (doc)       n_req=113872   miss_ratio=0.6297  first(id=42932745,size=512)

Docs build verified with sphinx-build -W: 13 pages, zero warnings, and I checked the generated HTML rewrites repo links to GitHub, keeps intra-doc links local, renders the mermaid diagrams, resolves all 18 images, and leaves no root-absolute href behind.

  • ctest --output-on-failure passes — 10/10, both plain Release and under -fsanitize=leak with CI's ASAN_OPTIONS
  • Build is warning-free (CI uses -Wall -Wextra -Werror)

Results

No behavior change for simulations. Verified by diffing against a develop build: 28 deterministic algorithms produce identical miss ratios, at the default hash power and after the registry refactor. lecarv0, cacheus, fifo-merge and RandomLRU vary run to run or build to build on their own — FIFO_Merge has an explicit next_rand() tiebreaker — so they were compared against themselves rather than across builds.

The intended behavior changes are all paths that previously crashed or were self-contradictory:

  • wtinyLFU --consider-obj-metadata=true goes from SIGSEGV to a result
  • mrcProfiler --profiler=MINISIM goes from aborting to working
  • Belady on a BIN_TRACE with a -1 sentinel goes from aborting to a curve matching the oracle reader exactly
  • WTinyLFU with main-cache=FIFO and metadata on moves 0.8619 → 0.8070, and clock 0.7732 → 0.7733, since the window is now charged its own overhead. With metadata off nothing changes; LRU, SLRU, sieve and ARC are unchanged either way.

Checklist

  • Formatted with clang-format (--dry-run --Werror on every modified source)
  • Added or updated tests
  • Added or updated documentation
  • New algorithms are registered in the CLI and listed in the README

One item left for you, since changing it would alter what the reader hands every caller: generalReader/binary.c now normalizes a -1 next-access sentinel to MAX_REUSE_DISTANCE to match the oracle readers, but the same file makes no attempt to reconcile any other convention a third-party binary trace might use.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

claude added 5 commits August 13, 2026 00:25
Six commands taken verbatim from the docs and --help output segfault.

`-e print`, documented in doc/quickstart_cachesim.md, dereferences state
that parse_params runs before: SLRU reads lru_max_n_bytes[0] before the
default segment sizes are allocated, and QDLP/S3FIFOd read the sub-cache
name before the sub-cache is built. Report the effective configuration
instead — the even split for SLRU, the configured cache type for the
other two. SLRU_current_params also stops appending once the buffer is
full rather than passing a negative size to snprintf.

`-o PATH` in traceAnalyzer and mrcProfiler is declared
OPTION_ARG_OPTIONAL, so argp hands the handler a NULL arg for the
space-separated form and strncpy dereferences it; only `-oPATH` and
`--output=PATH` worked. The same applied to mrcProfiler's --algo,
--size, --profiler and --profiler-params. These all require a value, so
drop OPTION_ARG_OPTIONAL and give them argument names in --help.

`--verbose` with no value reaches is_true(NULL) and crashes in
strcasecmp. Bare flags are the intended usage for OPTION_ARG_OPTIONAL
entries, so treat a NULL arg as true.

Verified: all 37 runnable commands in the quickstart docs now succeed,
and ctest passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
The docs referenced data/trace.vscsi, data/trace.csv, data/trace.txt and
data/trace.oracleGeneral throughout. None of those exist — the sample
traces are data/cloudPhysicsIO.*. Every copy-pasted quickstart command
failed on a fresh clone. The README was fixed in #321; this does the
same for doc/.

Also in this pass:

- run the examples as ./bin/<tool> from the build directory, matching
  the README, instead of ./<tool> with a ../data path that only resolves
  one directory up from the binary
- add obj-id-is-num=true to the csv examples; the sample trace has a
  numeric id column and cachesim errors out without it
- fix links to GDSF, LHD_Interface.cpp, the eviction CMakeLists and the
  example trace reader, none of which resolved
- note that GLCache/LRB/3LCache need their build flags, so
  "do not support algorithm" is explainable
- rewrite FAQ.md: correct the struct field name (clock_time, not
  real_time), fix the dead data/trace.csv link, and add entries for the
  optional-build error and where to get real traces

Every command in the quickstart docs was run against the sample traces.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
The repo had issue templates but no CONTRIBUTING.md, code of conduct, or
pull request template, and SECURITY.md was the GitHub boilerplate with
its instructional comments still in it and no reporting channel.

- CONTRIBUTING.md: build, debug and test workflow, the pre-commit hook,
  code style, where to add a new algorithm or reader, and PR
  expectations. Branch guidance points at develop, the default branch.
- CODE_OF_CONDUCT.md: Contributor Covenant 2.1, enforcement contact via
  the maintainers in .github/CODEOWNERS.
- SECURITY.md: rewritten around GitHub private vulnerability reporting,
  with what to include and what is in scope — trace parsing is the
  untrusted input, vendored third-party code is not.
- .github/PULL_REQUEST_TEMPLATE.md: change type, how it was tested, and
  before/after numbers for anything touching miss ratio or throughput.
- CITATION.cff: powers GitHub's "Cite this repository" button; S3-FIFO
  as the preferred citation, plus the OSDI, HotOS and SIEVE papers.

README links to all of them from the contributions section.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Files no build, script or doc referenced:

- test.c: a stale copy of the README library example, still pointing at
  data/trace.vscsi and referencing a quickstart.md that no longer
  exists. The README carries a correct, more complete version inline.
- random/: a glib allocator microbenchmark and its pasted stdout.
- doc/TODO, scripts/note: personal notes.
- package-lock.json: an empty stub at the root with no dependencies;
  the real Node package is under libCacheSim-node/.

.gitignore grouped by purpose, and its `.vscode/*` rule no longer
contradicts the four .vscode/*.json files that are deliberately checked
in as shared editor config. Added the traces traceConv and traceFilter
drop into data/ when the docs are followed from the repository root.

.editorconfig mirrors .clang-format (Google style, 2-space, 80 columns)
so editors match CI without contributors configuring anything.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
- npm-release.yml used actions/create-release@v1, which GitHub archived
  in 2021. Replaced with `gh release create`, which the runners already
  have and which the preceding step already uses. This also makes the
  job's release_created output real; the archived action never emitted
  it, so the value was always empty.
- code-quality.yml was still on actions/checkout@v3 while every other
  workflow used v4.
- persist-credentials: false on all checkouts, so the job token is not
  left in .git/config for later steps (OpenSSF Scorecard flags this, and
  the repo publishes a Scorecard badge).
- libCacheSim-node/package.json homepage pointed into tree/main, but
  there is no main branch, so the link 404s from the npm page.

Workflow YAML validated and the new release step dry-run with gh stubbed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
@1a1a11a
1a1a11a requested a review from haochengxia as a code owner August 13, 2026 00:29
Copilot AI lite review requested due to automatic review settings August 13, 2026 00:29

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47f91ef1b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread doc/advanced_lib.md Outdated
reader_init_param_t init_params_csv =
{.delimiter=',', .time_field=2, .obj_id_field=6, .obj_size_field=4, .has_header=FALSE};
reader_t *reader_csv_c = open_trace("data/trace.csv", CSV_TRACE, &init_params_csv);
reader_t *reader_csv_c = open_trace("data/cloudPhysicsIO.csv", CSV_TRACE, &init_params_csv);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Match the CSV parameters to the sample trace

Pointing this example at cloudPhysicsIO.csv makes its existing parameters invalid: the sample header is version,time,op,size,lbn, so the object ID is field 5, not field 6. Anyone copying this library example will never populate req->obj_id, causing every row to use the default ID and producing meaningless cache results; change obj_id_field to 5 (and preferably mark the header explicitly).

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and thanks — fixed in c9b6a1a. obj_id_field is now 5, and has_header = true plus obj_id_is_num = true, which the numeric lbn column also needs.

Compiling the snippet turned up two more errors in the same block that repointing the path had masked: .binary_fmt is not a field (it's .binary_fmt_str), and "<3I2H2Q" fails with unknown format '3' because the binary format parser has no repeat counts — format_to_size() only maps single characters. Expanded to "<IIIHHQQ", matching what the CLI docs use.

Both snippets now compile and run against the sample traces, and all three readers agree exactly:

vscsi              n_req=113872   miss_ratio=0.6297  first(id=42932745,size=512)
csv (doc)          n_req=113872   miss_ratio=0.6297  first(id=42932745,size=512)
binary (doc)       n_req=113872   miss_ratio=0.6297  first(id=42932745,size=512)

Generated by Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Polish pass to make libCacheSim more “fresh-clone runnable” and open-source ready by fixing documentation examples/links, addressing CLI crashes uncovered by doc verification, and adding standard community/metadata files while tightening a few CI and repo hygiene items.

Changes:

  • Updated docs and help text to reference shipped sample traces (cloudPhysicsIO.*) and correct build-directory invocation (./bin/<tool>).
  • Fixed multiple CLI/-e print crash paths by adjusting option argument handling and making parameter reporting resilient to uninitialized sub-components.
  • Added community health and metadata files (contributing guide, code of conduct, PR template, citation, improved security policy) and cleaned up ignored/scratch artifacts and CI workflows.

Reviewed changes

Copilot reviewed 28 out of 33 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
test.c Removed unreferenced scratch example program.
SECURITY.md Replaced boilerplate with a concrete private reporting process and scope.
scripts/note Removed unreferenced scratch notes.
README.md Linked to new contributing/CoC/security/citation docs from README.
random/allocatorResult Removed unreferenced scratch output.
random/allocator.c Removed unreferenced scratch benchmark source.
package-lock.json Removed empty root lockfile.
libCacheSim/cache/eviction/SLRU.c Made -e print parameter reporting safer (but see stored crash fix request).
libCacheSim/cache/eviction/S3FIFOd.c Avoided dereferencing unbuilt sub-cache when printing params.
libCacheSim/cache/eviction/QDLP.c Avoided dereferencing unbuilt sub-cache when printing params.
libCacheSim/bin/traceAnalyzer/cli_parser.cpp Made --output take a required PATH; updated example trace path.
libCacheSim/bin/mrcProfiler/cli_parser.cpp Made several value-taking options require arguments (no NULL deref).
libCacheSim/bin/cli_reader_utils.c Treated bare optional-arg flags as true to avoid NULL crashes.
libCacheSim-node/package.json Fixed npm homepage URL to point at the default branch.
FAQ.md Reformatted/expanded FAQ and updated oracleGeneral struct field name.
doc/TODO Removed stale internal TODO file.
doc/README.md Added links to contributing and help resources.
doc/quickstart_traceUtils.md Updated sample commands to shipped trace names/paths.
doc/quickstart_traceAnalyzer.md Updated sample commands to shipped trace names/paths and ./bin/ usage.
doc/quickstart_cachesim.md Updated commands, trace paths, and algorithm links; clarified CSV numeric-id behavior.
doc/advanced_lib.md Updated sample trace filenames in API examples.
doc/advanced_lib_extend.md Corrected extension instructions and updated reader references.
CONTRIBUTING.md Added contributor guidance, build/test workflow, and project conventions.
CODE_OF_CONDUCT.md Added Contributor Covenant 2.1 code of conduct.
CITATION.cff Added citation metadata for GitHub “Cite this repository”.
.gitignore Clarified ignores and stopped ignoring checked-in .vscode/*.json files.
.github/workflows/npm-release.yml Replaced archived release action with gh release create; hardened checkouts.
.github/workflows/codeql-analysis.yml Set persist-credentials: false on checkout.
.github/workflows/code-quality.yml Bumped checkout to v4 and disabled persisted credentials.
.github/workflows/build.yml Disabled persisted credentials in build workflow checkouts.
.github/PULL_REQUEST_TEMPLATE.md Added PR template aligned with project workflow.
.github/copilot-instructions.md Updated guidance on using sample traces and performance caveats.
.editorconfig Added editorconfig aligned with the repo’s formatting conventions.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +410 to +416
static int SLRU_seg_pct(const cache_t *cache, const SLRU_params_t *params,
const int seg) {
if (params->lru_max_n_bytes == NULL) {
return 100 / params->n_seg;
}
return (int)(params->lru_max_n_bytes[seg] * 100 / cache->cache_size);
}

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The n_seg == 0 half is real — fixed in c9b6a1a, though not with a defensive check in SLRU_seg_pct().

The divide-by-zero isn't specific to -e print: -e n-seg=0 on its own already died with SIGFPE on develop, at cache_size / params->n_seg in SLRU_init(). Guarding only the reporting helper would have left that. n-seg was never validated, so it's now rejected at parse time:

$ cachesim ../data/cloudPhysicsIO.vscsi vscsi slru 1gb -e n-seg=0
[ERROR] SLRU.c:458  n-seg must be between 1 and 16, got 0

The empty seg-size= case is also rejected now (seg-size needs at least one segment with a positive size), where before it aborted reporting -9223372036854775808 bytes. Same pass caught an unrelated overflow: seg_size_array[SLRU_MAX_N_SEG] was written without a bound, so more than 16 colon-separated sizes wrote past the end of the stack array before the validity check ran.

The cache_size == 0 half doesn't hold. auto resolves to concrete sizes before any cache is constructed, so cache->cache_size is already set:

$ cachesim ../data/cloudPhysicsIO.vscsi vscsi slru auto -e print
current parameters: n-seg=4,seg-size=25:25:25:25

Verified across n-seg 0/-1/4/16/20 and seg-size empty / 0:0 / 24 entries: invalid inputs report what is wrong, valid ones are unchanged, and ctest is 9/9.


Generated by Claude Code

claude added 5 commits August 13, 2026 00:37
macOS CI has been red since the runner image moved to Xcode 26.6 —
including on docs-only commits to develop, so this predates this branch.
clang now rejects `UINT64_MAX * sample_rate` and `... / UINT64_MAX` under
-Werror, because UINT64_MAX has no exact double representation
(-Wimplicit-const-int-float-conversion). Made both widenings explicit in
mrcProfiler; the arithmetic is unchanged, and both SHARDS paths were
re-run to confirm identical output.

doc/advanced_lib.md's reader examples were wrong in three ways, which
repointing them at the sample trace made visible:

- obj_id_field was 6, but cloudPhysicsIO.csv is version,time,op,size,lbn
  so the id is field 5. Copying it verbatim gave every row the same
  default id and meaningless results.
- has_header was FALSE for a file that has a header, and obj_id_is_num
  was unset on a numeric id column.
- the binary example used .binary_fmt, which is not a field
  (.binary_fmt_str), with "<3I2H2Q" — the format parser has no repeat
  counts, so it errored with "unknown format '3'". Expanded to
  "<IIIHHQQ", matching the CLI docs.

Both snippets were compiled and run against the sample traces: the csv
and binary readers now agree with the vscsi reader exactly — 113872
requests, miss ratio 0.6297, same first object.

While in SLRU_parse_params, validated the parameters it never checked.
`-e n-seg=0` divided by zero in init (SIGFPE, present on develop), and
seg-size wrote past seg_size_array[SLRU_MAX_N_SEG] once given more than
16 segments. Both now report what is wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
libCacheSim-node/package.json said "MIT" while the project is GPL-3.0 and
binding.gyp links vendor/liblibCacheSim.a statically, making the addon a
derivative work. npm has been publishing it under the wrong license. The
package README said "MIT License - see the LICENSE file", where that file
is the repository's GPLv3 — the two contradicted each other on the same
page.

Now GPL-3.0-only in both, matching the LICENSE file. Using -only rather
than -or-later because the repo ships the plain GPLv3 text with no
"or (at your option) any later version" notice in any source file; switch
to GPL-3.0-or-later if that was the intent. CITATION.cff uses the same
identifier so the two agree.

Also fixed the package README's link to the Python bindings, which
pointed at libCacheSim/pyBindings — a path that does not exist. They live
in the cacheMon/libCacheSim-python repository.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
.readthedocs.yaml pointed sphinx at docs/conf.py. There is no docs/
directory, no conf.py and no .rst anywhere in the repo — the docs are the
Markdown in doc/ — so every Read the Docs build failed immediately.

Wired up Sphinx over the existing Markdown with MyST, so the sources stay
readable on GitHub and need no duplication:

- doc/conf.py, doc/index.md (toctree entry point), doc/requirements.txt.
  RTD installed the root requirements.txt (numpy/matplotlib/pandas) for a
  docs build that needs none of it; it now installs only doc deps.
- The Markdown links into the repository as "/libCacheSim/..." and
  "../README.md", which resolve on github.com but not on a docs site. A
  source-read hook rewrites those to blob URLs while keeping links
  between doc pages local, so the sidebar, search and anchors work. It
  has to run before MyST resolves links, otherwise each one is reported
  as a missing cross-reference.
- ```mermaid fences render as diagrams instead of failing to highlight.

Set fail_on_warning, having got the build to zero warnings from 96:

- doc/API.md had no headings at all, so it could not appear in a toctree,
  and it had drifted from the headers: reader_init_param_t was missing
  most of its fields and named the binary format binary_fmt (it is
  binary_fmt_str), get_num_of_req returned uint64_t, the simulator
  returned a sim_res_t that no longer exists, and the file trailed off at
  a "profiler:" heading with nothing under it. Rewritten against the
  current headers; every symbol checked to exist.
- advanced_lib.md documented the same stale simulator API, including a
  "reader_t reader*" typo, plus heading levels that skipped from H2 to H4.
- performance.md was two empty sections and one bullet. Filled in with
  the knobs that exist — tcmalloc, USE_HUGEPAGE, binary traces, the
  threading and sizing flags — each verified against the CLI and CMake.
- install.md, quickstart_traceAnalyzer.md and quickstart_traceUtils.md
  started at H2, so promoted their heading hierarchies (skipping fenced
  code, which contains #-comments).
- A massif report was tagged as shell and failed to lex.

Verified with `sphinx-build -W`: 13 pages, no warnings. Checked the
generated HTML rewrites repo links, keeps intra-doc links local, renders
the mermaid diagrams, and leaves no root-absolute href behind. All
markdown links still resolve for the GitHub view.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Nothing exercised the command-line tools; every crash fixed on this
branch was reachable from a documented command and none of them would
have been caught. testCLI runs the built binaries and asserts on exit
status and output.

Covered:

- each sample trace format replays, and a numeric csv id column without
  obj-id-is-num is rejected rather than silently hashed
- `-e print` for SLRU, QDLP, S3FIFOd and ten other algorithms, checking
  the reported values, since these dereferenced state that parse_params
  runs before
- SLRU parameter validation: n-seg of 0, -1 and above the maximum, empty
  seg-size, seg-size summing to zero, and more segments than the array
  holds
- value-taking options in the space-separated form (`-o path`), which
  argp passes as a NULL argument when the option is declared
  OPTION_ARG_OPTIONAL, plus the attached and `=` forms, and bare flags
  like --verbose that reach is_true(NULL)

Invalid input is asserted to fail *cleanly*: a non-zero exit with a
message, but not SIGSEGV, SIGFPE or SIGBUS. The project's ERROR() aborts,
so a deliberate rejection is distinguishable from a crash.

Confirmed the tests actually catch these: reverting the six fixed sources
to develop and rebuilding turns up 16 failures, and 0 with them restored.
The script skips itself when the binaries or sample traces are missing,
so it stays green in build layouts it does not understand.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
The ubuntu job builds with LeakSanitizer, and testCLI runs the binaries
to completion, so it started reporting what the C unit tests never
reached. All of these are pre-existing; none are caused by this branch.

Leaked on the `-e print` early exit (29 files, one line each): the
parameter string is strdup'd, but the print branch exits before the free
at the end of the parser. GLCache, Mithril and PG never kept the original
pointer at all, so they leaked on every call rather than only on print.

Missing frees, which only show up when a cache is actually torn down:

- Size_free freed the priority queue and its nodes but not the params
  struct itself; RandomLRU_free and SLRUv0_free likewise
- S3FIFOd_free freed three of its five sub-caches, leaving small_eviction
  and main_eviction (73 KB on a short replay)
- SLRUv0_cool allocated a request up front and returned early on i == 0
  without freeing it, once per eviction from the bottom segment
- WTinyLFU_free never freed its params struct

Two more crashes of the kind this branch already fixed elsewhere, both
reachable from documented commands:

- `s3fifov0 -e print` and `flashProb -e print` dereferenced a sub-cache
  that parse_params runs before building
- WTinyLFU_init read params->main_cache->obj_md_size out of a malloc'd,
  un-memset struct before main_cache was assigned, so
  `wtinyLFU --consider-obj-metadata=true` segfaulted. The struct is now
  zeroed and the metadata size read after the sub-cache exists.

`-e print` is documented for every algorithm but Clock2QPlus and
pluginCache rejected the bare `print` key before reaching their own print
branch, and WTinyLFU had no print branch at all despite taking
parameters. All three now behave like the rest.

Reverted the change to cachesim's hashpower heuristic. It tests for
"data/trace." and has been dead for as long as that file has not existed;
pointing it at the current sample traces revives it, and a hash table 256
times smaller changes which candidates sampling-based algorithms draw —
RandomLRU's miss ratio moved in the fourth decimal. Left dead with a note
rather than silently changing results.

Verified against a develop build: identical miss ratios for 17
algorithms, and `wtinyLFU --consider-obj-metadata=true` goes from SIGSEGV
to a result. ctest is 10/10 both plain and under LSan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b5d98be99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libCacheSim/mrcProfiler/mrcProfiler.cpp Outdated
Comment on lines +100 to +101
uint64_t sample_max =
static_cast<uint64_t>(static_cast<double>(UINT64_MAX) * sample_rate);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle full sampling before converting the threshold

When SHARDS is run with --profiler-params=FIX_RATE,1,..., converting UINT64_MAX to double rounds it to 2^64, so the new cast back to uint64_t is out of range and invokes undefined behavior before the following sample_rate == 1 branch can overwrite the result. UBSan reports a float-cast-overflow here; handle the full-sampling case before performing this conversion.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, and thanks — this is a case I considered while writing that cast and talked myself out of, on the reasoning that the sample_rate == 1 branch overwrites the value. It does, but only after the conversion has already been evaluated, so the UB stands. Fixed in ab5507b by taking full sampling before scaling.

Reproduced it first rather than taking it on faith. GCC's -fsanitize=undefined does not include float-cast-overflow in its default set, which is why the ubuntu job never flagged it — it needs to be asked for explicitly:

$ g++ -fsanitize=float-cast-overflow -fno-sanitize-recover=all ...
mrcProfiler.cpp:104:61: runtime error: 1.84467e+19 is outside the range of
representable values of type 'long unsigned int'

The conversion silently produced 0. Same build, same command, after the fix: clean.

One refinement to the suggestion — I checked whether a rate just below 1 could also round up to 2^64 and need clamping, and it can't. 2^64 × (1 − 2⁻⁵³) is 2^64 − 2048, which is exactly representable (doubles are spaced 2048 apart just below 2^64) and comfortably in range. So the >= 1 guard alone is sufficient, no clamp:

rate=1                   -> product >= 2^64
rate=0.99999999999999989 -> product <  2^64

testCLI now covers SHARDS at rates 1, 0.999, 0.5 and 0.0001, plus FIX_SIZE, so this can't come back silently.

Adding that coverage turned up something separate, which I've documented rather than fixed: --profiler=MINISIM aborts with cannot load internal cache FIFO: undefined symbol: FIFO_init. It resolves eviction algorithms via dlsym() against the mrcProfiler executable, but those constructors live in the static library and nothing in mrcProfiler references them, so the linker never pulls them in — -rdynamic can't help, and I confirmed it fails identically on develop. Fixing it means whole-archive linking or routing MINISIM through create_cache(), which is a design call for the maintainers, so doc/quickstart_mrcProfiler.md now warns instead of promising a command that aborts.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the aside in my reply above, since it is now stale: I said MINISIM's dlsym failure was documented rather than fixed. It is fixed, in 610d644 and the commits around it.

The cause was as diagnosed — the eviction constructors live in the static library and nothing in mrcProfiler referenced them, so the linker never pulled those archive members in and dlsym had nothing to find. Rather than whole-archive linking, the name→constructor table moved into the library as cache/cacheAlgoRegistry.c, and both create_cache() and MINISIM go through it. plugin.c tries the registry first and only falls back to dlsym for genuinely external policies, and create_cache_external() returns NULL instead of calling exit().

That also removed the duplicate table the CLI was carrying, and is what makes tinyLFU reachable from MINISIM (separate thread). The warning in doc/quickstart_mrcProfiler.md is gone and the MINISIM examples there run.


Generated by Claude Code

Codex caught this on the previous commit and it is a real defect I
introduced the explicit cast around without fixing.

UINT64_MAX has no exact double representation — it rounds up to 2^64 —
so at a sample rate of 1, `(double)UINT64_MAX * sample_rate` is 2^64,
which does not fit in a uint64_t. The conversion is undefined before the
`sample_rate == 1` branch below it can overwrite the result. Compiling
the old form with -fsanitize=float-cast-overflow reports

  mrcProfiler.cpp:104:61: runtime error: 1.84467e+19 is outside the range
  of representable values of type 'long unsigned int'

and the conversion silently yields 0. It went unnoticed because GCC's
-fsanitize=undefined does not include that check by default, so the
ubuntu job never saw it. Full sampling is now taken before scaling, so
the conversion only runs on a rate below 1. That bound is enough on its
own: 2^64 * (1 - 2^-53) is 2^64 - 2048, exactly representable and in
range, so no clamp is needed.

testCLI now covers SHARDS at rates 1, 0.999, 0.5 and 0.0001, and the
FIX_SIZE mode.

Not fixed, and now documented instead: `--profiler=MINISIM` aborts with
"cannot load internal cache FIFO: undefined symbol: FIFO_init". It
resolves eviction algorithms with dlsym() against the mrcProfiler
executable, but those constructors are in the static library and nothing
in mrcProfiler references them, so the linker never pulls them in and
-rdynamic cannot help. It fails the same way on develop. Fixing it means
either whole-archive linking or routing MINISIM through create_cache(),
which is a design call for the maintainers, so doc/quickstart_mrcProfiler.md
now warns instead of promising a command that aborts. Its examples also
use ./bin/mrcProfiler now, matching the other guides.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab5507b69f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libCacheSim/cache/eviction/WTinyLFU.c Outdated
}

if (ccache_params.consider_obj_metadata) {
cache->obj_md_size = params->main_cache->obj_md_size;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for window metadata separately

When consider_obj_metadata=true and WTinyLFU uses a main policy whose metadata differs from LRU (for example, main-cache=FIFO has 0 bytes while the window LRU has 16), copying the main cache's value here makes WTinyLFU_can_insert() apply the wrong overhead to the window. Objects within 16 bytes of the window capacity are then admitted even though the child LRU exceeds its configured size, shifting eviction timing and miss results; policies with larger metadata can conversely reject valid window entries. Use the window's metadata for window admission and the main cache's metadata for transfers instead of representing both with this single value.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The observation is correct, and I measured it rather than reasoning about it. WTinyLFU_can_insert() charges cache->obj_md_size against params->LRU->cache_size, and the two sub-caches genuinely disagree — LRU and SLRU report 16 bytes with metadata on, FIFO reports 0 — so main-cache=FIFO charges the window nothing for an overhead the window really reserves.

I've not changed it, for a different reason than the finding being wrong.

That assignment restores what the original code already intended. Before this branch the same line read params->main_cache->obj_md_size from a malloc'd, un-memset struct before main_cache was assigned, so wtinyLFU --consider-obj-metadata=true segfaulted outright on develop:

$ cachesim ... wtinyLFU 100mb --consider-obj-metadata=true   # develop
Segmentation fault (rc=139)

So there is no working behaviour on either side of this to preserve — the whole configuration was unreachable. Splitting the accounting the way you describe would change admission decisions and therefore miss ratios, which makes it a modelling choice in WTinyLFU rather than a typo. This PR has deliberately avoided that category (same reason I backed out a revived hashpower heuristic that moved RandomLRU's miss ratio, and left MINISIM's linking alone), so it is yours to decide rather than mine to slip in.

What I did do is record the asymmetry at the assignment site in ea8969d so the next reader is not misled, with the concrete FIFO-vs-window example. Current numbers, if useful as a baseline for deciding:

main-cache miss ratio byte miss ratio
FIFO 0.8619 0.9514
LRU 0.7749 0.8887
SLRU 0.8064 0.9525

Verified comment-only: those are unchanged by the commit, and ctest is 10/10 plain and under LeakSanitizer.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up to correct the record: I said above I was leaving this to the maintainers, and then went ahead and did it in ea8969d once the call was delegated to me. So the "not changed" in my previous reply is stale.

The window now charges its own overhead rather than the main cache's:

/* WTinyLFU_can_insert */
req->obj_size + params->LRU->obj_md_size <= params->LRU->cache_size

and cache->obj_md_size, which is what callers see for the composite, is MAX(params->LRU->obj_md_size, params->main_cache->obj_md_size) — the larger of the two, set after main_cache is built rather than read out of uninitialised memory.

So main-cache=FIFO no longer under-charges the window by 16 bytes per object, which was the concrete case you named.


Generated by Claude Code

claude added 6 commits August 13, 2026 01:47
Codex flagged that cache->obj_md_size stands in for two sub-caches with
different per-object overheads. The observation is right:
WTinyLFU_can_insert() charges this value against params->LRU->cache_size,
so with main-cache=FIFO it charges 0 against a window LRU that actually
reserves 16 bytes per object, and admits objects the window cannot hold.

Not changed. The assignment restores what the original code intended —
before this branch it read main_cache before that pointer was assigned
and segfaulted, so no working behaviour depends on either reading.
Splitting the accounting (window metadata for window admission, main for
transfers) changes admission decisions and therefore miss ratios, which
is a modelling choice in WTinyLFU rather than a typo, so it is the
maintainers' call and not something to slip into a polish PR.

Comment records the asymmetry so it is not silently wrong. Verified
comment-only: miss ratios for main-cache FIFO/LRU/SLRU with
--consider-obj-metadata=true are unchanged, ctest 10/10 plain and LSan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
`mrcProfiler --profiler=MINISIM` aborted on every run, on develop as well
as here:

  cannot load internal cache FIFO: undefined symbol: FIFO_init

It looked its eviction algorithm up with dlsym() against its own
executable. That cannot work: the constructors sit in an archive member
nothing references, so the linker never pulls them in and the symbol is
absent from the binary however it is linked — -rdynamic does not help.
Every MINISIM example in doc/quickstart_mrcProfiler.md failed.

Moved the name to constructor mapping out of cachesim's private header
and into the library, as cache/cacheAlgoRegistry.c. Referencing the table
from a translation unit the profiler already links is what pulls those
archive members in, so the lookup is a plain call with no dynamic
loading, and it works the same on macOS. cachesim and MINISIM now share
one table instead of one table and a dlsym path; create_cache() keeps the
cases that need more than a lookup (a smaller hash table for hyperbolic,
a default window size for tinyLFU, the oracle-trace checks for belady).

Two error paths in plugin.c were unreachable or unhelpful:

- create_cache_internal() aborted when dlsym failed, so
  create_cache_using_plugin()'s fallback to a shared library could never
  run. It returns NULL now and the fallback works.
- create_cache_external() called exit() on a missing library, which the
  header documents as returning NULL, and printed a bare dlerror. An
  unknown algorithm reported "./libnosuchalgo.so: cannot open shared
  object file"; it now says which algorithm could not be created.

Verified across FIFO, ARC, S3FIFO, LRU, sieve, lfu, twoq and clock, and
that an unknown name fails with a message naming it. cachesim is
unaffected: 28 deterministic algorithms produce identical miss ratios
before and after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
cachesim shrank its hash table by 8 powers of two when the trace path
contained "data/trace.". No such file has existed for a long time, so the
saving never happened, and simply repointing the string at the current
sample traces is not equivalent: sampling-based algorithms draw eviction
candidates from the hash table, so RandomLRU and hyperbolic move when it
resizes. Sizing a table by sniffing a filename is the wrong mechanism for
something that changes results.

Replaced with --hashpower, log2 of the table size, default 24 as before.
The saving it was after is real and now available on any trace: replaying
the sample trace goes from 106 MB to 15 MB at 20 and 8 MB at 16. Values
outside 1..39 are rejected rather than silently ignored by
cache_struct_init.

Default behaviour is unchanged, since the heuristic never fired. The
caveat about sampling-based algorithms is in --help and in
doc/quickstart_cachesim.md so the trade-off is visible at the point of
use.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Raised in review by Codex. WTinyLFU_can_insert() decided whether an
object fits in the window by charging cache->obj_md_size, which holds the
*main* cache's per-object overhead, against the window LRU's size. The
two disagree: LRU and SLRU reserve 16 bytes, FIFO and Clock none. With
main-cache=FIFO the window was charged nothing for an overhead it does
reserve, so objects were admitted that the window could not hold.

Each sub-cache is now checked against its own overhead — the window
against the window LRU's, the main cache through its own can_insert().
The parent's obj_md_size, which cache_can_insert_default() uses for the
whole-cache size check, takes the larger of the two: an object that does
not fit under the heavier policy does not fit in this cache.

Only reachable since the fix earlier on this branch, because
--consider-obj-metadata=true segfaulted here on develop. With metadata
off every object metadata size is zero and nothing changes; verified
identical for main-cache FIFO/LRU/SLRU/sieve/ARC/clock. With metadata on,
only the configurations whose overheads differ from the window's move:
FIFO 0.8619 -> 0.8070 and clock 0.7732 -> 0.7733, while LRU, SLRU, sieve
and ARC are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
testCLI grows to 142 checks:

- MINISIM across FIFO, ARC, S3FIFO, sieve, twoq, clock and lfu, which
  every one of aborted before, plus an unknown algorithm failing cleanly
  rather than crashing
- --hashpower at 24/20/16/12, and rejection of 0, -1, 40 and 99
- wtinyLFU with metadata on for each main-cache type, since the window
  and main cache reserve different per-object overheads

Passes under both a plain Release build and the LeakSanitizer build CI
uses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…-repo-polish-wc1xjb

# Conflicts:
#	libCacheSim/mrcProfiler/mrcProfiler.cpp

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37219679f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +174 to +176
if (arguments->hashpower <= 0 || arguments->hashpower >= 40) {
ERROR("hashpower must be between 1 and 39, got %d\n",
arguments->hashpower);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard low hash powers before initializing child caches

When --hashpower is 1–4, this validation accepts the value even though registered composite policies reinterpret it unsafely. For example, slruv0 --hashpower=4 computes each child LRU's hash power as zero in SLRUv0_init, after which cache_struct_init silently substitutes the default of 24; the four child caches therefore allocate four 16M-entry hash tables instead of the requested small tables, potentially causing an unexpected OOM. Either enforce a compatible minimum here or clamp child-cache reductions to a positive value.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 9007e88 — thanks, this is a good catch. Measured it before deciding, and the inversion is exactly as described:

slruv0 --hashpower=4  ->  18 MB      # child hash power hits 0, falls back to the default
slruv0 --hashpower=5  ->   6 MB

Asking for a smaller table allocated a bigger one.

You offered two options; I took the second (clamp the child reductions) rather than raising the CLI minimum, because the reduction sites are the actual bug and four of them already do exactly this. Hyperbolic, Random, RandomTwo and RandomLRU clamp with MAX(12, hashpower - 8); Cacheus, S3FIFOd, SLRUv0 and LP_SFIFO did not. They now clamp at 4, so the pattern is uniform and a future --hashpower caller cannot reintroduce it. Memory is monotonic again — 6 MB at 4 through 8 MB at 24 — and the clamp only binds below hash power 8, so behaviour at the default is provably unchanged (verified identical for cacheus, s3fifod, slruv0, lru and fifo).

One thing the audit turned up alongside it: SFIFOv0 divides by that same expression rather than assigning it, so it divided by zero at hash power 4. I guarded the divisor rather than rewriting it to match its siblings — the division reads as deliberate, and the algorithm is not in the CLI registry, so there was no call to change its sizing on a guess.

Not reproducible as an OOM here, incidentally: the fall-through allocates the default table rather than a 16M-entry one per child, so it was 18 MB rather than gigabytes. The inversion is the real defect and it is fixed.

testCLI now covers slruv0, s3fifod, cacheus and lru at hash powers 4, 5, 6 and 8; 162 checks, green under both a plain build and LeakSanitizer.


Generated by Claude Code

Raised in review by Codex against --hashpower. Several composite policies
size their sub-caches by subtracting from the parent's hash power without
a lower bound, and cache_struct_init treats a value of zero or less as
"unset" and substitutes the full-size default. Asking for a small table
therefore allocated several large ones:

  slruv0 --hashpower=4  ->  18 MB
  slruv0 --hashpower=5  ->   6 MB

Cacheus, S3FIFOd, SLRUv0 and LP_SFIFO now clamp at 4, matching what
Hyperbolic, Random, RandomTwo and RandomLRU already do. Behaviour is
unchanged wherever the old arithmetic stayed positive — the clamp only
binds below hash power 8 — and verified identical at the default for
cacheus, s3fifod, slruv0, lru and fifo. Memory now falls monotonically as
the hash power drops, 6 MB at 4 through 8 MB at 24.

SFIFOv0 divides by that same expression rather than assigning it, so it
divided by zero at hash power 4. Guarded the divisor instead of rewriting
it to match its siblings: the division looks deliberate, and the
algorithm is not reachable from the CLI, so there is no call to change
its sizing on a guess.

Also added the NaN sample rate case from #325 to testCLI, which arrived
with the fix but without a test. testCLI is now 162 checks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9007e88676

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +72 to +75
cache_t *cache = create_cache_by_name(cache_alg_name, cc_params,
(const char *)cache_specific_params);
if (cache != NULL) {
return cache;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the full cache size when MINISIM skips sampling

When MINISIM is run with FIX_RATE above 0.5, this change now lets built-in algorithms execute instead of aborting, but MRCProfilerMINISIM::run() disables the sampler while still constructing caches with mrc_size_vec[i] * sample_rate. For example, a rate of 0.75 processes the complete trace against caches at only 75% of the requested sizes, shifting the reported MRC. Set the effective rate to 1 when the no-sampling path is selected.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 2f16934. The accounting further down already branched on whether a sampler exists; the sizing did not, so the two disagreed above 0.5.

Quantified it before fixing — a 100MB point at rate 0.75 was measuring a 75MB cache:

cachesim lru 75mb   -> 0.8257    # what the curve actually reported
cachesim lru 100mb  -> 0.8225    # what it was labelled

Setting the effective rate to 1 on the no-sampling path makes unsampled MINISIM exact rather than approximate, which is a stronger property to test than "close enough":

              MINISIM(0.75)   cachesim
100MB           0.822467       0.8225
300MB           0.766150       0.7661
500MB           0.718500       0.7185

testCLI now asserts that equality directly rather than just a zero exit, so the mislabelling cannot come back quietly. Rates 0.6, 0.75 and 1 are covered.


Generated by Claude Code

Comment on lines +76 to +77
{"belady", Belady_init},
{"beladySize", BeladySize_init},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject oracle-only policies before MINISIM runs

When mrcProfiler uses MINISIM with --algo=belady or beladySize on a non-oracle trace, these new registry entries construct the policy without the trace-type validation performed by cachesim. Ordinary readers leave next_access_vtime at -2, so Belady aborts under debug assertions and consumes invalid future-access priorities in release builds, yielding an invalid MRC. Validate the reader type in the MINISIM path before allowing these policies.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct — fixed in 2f16934. Reproduced it first: --algo=belady --profiler=MINISIM on the vscsi sample trace ran to completion and printed a curve, exiting 0, while cachesim belady on the same trace warns and refuses. A plausible-looking answer from unset next_access_vtime is worse than a crash, since nothing signals it is wrong.

I put the check in the MINISIM path rather than removing the two entries from the registry. The registry answers "is this a built-in algorithm", and belady is one; the oracle-trace requirement is a usage constraint for the caller, which is how the header already documents it. Removing them would have made the registry quietly incomplete and pushed the same problem onto the next caller.

It checks reader_->trace_type against ORACLE_GENERAL_TRACE and LCS_TRACE rather than sniffing the path for "oracleGeneral" the way cache_init.h does — the reader already knows its own type, so there is no reason to infer it from a filename here.

$ mrcProfiler ../data/cloudPhysicsIO.vscsi vscsi --algo=belady --profiler=MINISIM ...
[ERROR] belady needs future information, so it only works on oracleGeneral and
        lcs traces; convert with ./bin/traceConv

$ mrcProfiler ../data/cloudPhysicsIO.oracleGeneral.bin oracleGeneral --algo=belady --profiler=MINISIM ...
rc=0

testCLI covers both policies against both trace types, so the rejection and the acceptance are each pinned.


Generated by Claude Code

Two follow-ups from Codex, both reachable only because MINISIM now runs
at all rather than aborting.

Above a sample rate of 0.5 MINISIM stops sampling and replays the whole
trace, but it still sized the miniature caches at mrc_size_vec[i] *
sample_rate. Every request reached a cache smaller than the one the user
asked for, so the curve was mislabelled: a 100MB point at rate 0.75
measured a 75MB cache, reporting 0.8257 where 100MB is 0.8225. The
accounting further down already branched on whether a sampler exists;
the sizing did not. Setting the effective rate to 1 on that path makes
the two agree, and unsampled MINISIM now matches cachesim exactly —
0.822467, 0.766150 and 0.718500 against 0.8225, 0.7661 and 0.7185 at
100MB, 300MB and 500MB.

belady and beladySize read next_access_vtime, which ordinary readers
leave unset. cachesim refuses them on a non-oracle trace; MINISIM reached
them through the registry without that check and produced a
plausible-looking curve from garbage priorities, exiting 0. It now
refuses them the same way, and still accepts them on oracleGeneral.

testCLI gains the unsampled rates, an equality check against cachesim so
the mislabelling cannot come back quietly, and both trace types for the
two oracle-only policies. 170 checks, green plain and under
LeakSanitizer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2f1693427f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread doc/API.md Outdated
const reader_init_param_t *const reader_init_param) {
return setup_reader(path, type, reader_init_param);
}
Object ids are hashed unless you set `obj_id_is_num`, which you should do when the id field holds numbers.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct the documented object-ID default

When a caller follows the preceding instruction and initializes parameters with default_reader_init_params(), obj_id_is_num is already true, contrary to this sentence. For a CSV containing string IDs, leaving that default unchanged sends every value through strtoull() in csv_cb1; nonnumeric IDs become zero rather than being hashed, collapsing distinct objects and corrupting simulation results. Document that callers must set obj_id_is_num = false for string IDs, or change the default.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and the consequence is worse than the doc being merely imprecise — fixed in 610d644.

Reproduced it against a csv with string ids, using exactly the defaults the doc pointed at:

obj_id_is_num default = true
row 1 -> obj_id=0 size=100      # alpha
row 2 -> obj_id=0 size=200      # beta
row 3 -> obj_id=0 size=100      # alpha
...

Six distinct objects collapsed onto one id. As you say, csv_cb1 tests obj_id_is_num first and only reaches the hashing branch on the false path, so strtoull() gets the string, warns, and leaves 0. The result is a plausible miss ratio rather than an error, which is the part that makes it dangerous.

The mismatch is specifically library-versus-CLI: cli_reader_utils.c overrides the flag to false right after calling set_default_reader_init_params(), so cachesim hashes string ids correctly, while a library caller taking default_reader_init_params() gets the opposite. Both doc/API.md and doc/advanced_lib.md now say so plainly, with the one-liner to set:

reader_init_param_t p = default_reader_init_params();
p.obj_id_is_num = false;   /* string ids: hash them */

Confirmed that fixes it — distinct ids now hash distinctly and the repeats (alpha at rows 1 and 3, beta at 2 and 5) map to matching values.

I documented rather than changed the default, since flipping it would alter behaviour for every existing library caller and true is plausibly right for the binary readers. Worth the maintainers deciding separately.

testCLI now drives a string-id csv and asserts the miss ratio that four distinct objects in six requests produce, so a regression to id 0 fails the test instead of returning a believable number.


Generated by Claude Code

{"slru", SLRU_init},
{"slruv0", SLRUv0_init},
{"twoq", TwoQ_init},
{"wtinyLFU", WTinyLFU_init},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Register the cachesim tinyLFU alias

When MINISIM or a library caller passes tinyLFU, this registry returns no constructor even though create_cache() explicitly accepts that cachesim alias. The plugin path then looks for a nonexistent tinyLFU_init/libtinyLFU.so and aborts, contradicting the newly documented promise that name-based lookup accepts the same aliases as the CLI. Add tinyLFU here with the CLI's default-window handling, or stop advertising equivalent names.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair — I made that promise in the header and then did not keep it. Fixed in 610d644. Reproduced first:

cachesim tinyLFU  -> rc=0
MINISIM  tinyLFU  -> rc=134 | cannot load ./libtinyLFU.so

You offered "add it with the CLI's default-window handling, or stop advertising equivalent names". I took the first, but the default-window handling turned out to be unnecessary — which is worth stating since it is the interesting part.

create_cache() appended ,window-size=0.01 when the caller gave parameters without a window. WTinyLFU's own DEFAULT_PARAMS is "main-cache=SLRU,window-size=0.01", and WTinyLFU_init parses that before applying caller parameters, so the append could only ever re-set a value that was already 0.01. Checked all four shapes rather than assuming:

eviction params tinyLFU vs wtinyLFU
(none) identical
main-cache=LRU identical
window-size=0.2 identical
main-cache=LRU,window-size=0.3 identical

So it is a plain alias in the registry, and the special case is gone along with its malloc/snprintf. That leaves one code path instead of two and makes the name reachable from MINISIM.

testCLI now exercises tinyLFU in the all-algorithms sweep and through MINISIM.


Generated by Claude Code

Two more from Codex.

The API docs had the object-id default backwards. They said ids are
hashed unless you opt into numeric parsing, but
default_reader_init_params() sets obj_id_is_num to true — the opposite of
what cachesim does, which overrides it to false. csv_cb1 checks that flag
first and only hashes on the false path, so a library caller who took the
defaults with string ids got strtoull() and a warning per row:

  row 1 -> obj_id=0    row 2 -> obj_id=0    row 3 -> obj_id=0

Six distinct objects collapsed into one, and the resulting miss ratio is
meaningless rather than obviously wrong. Both doc/API.md and
doc/advanced_lib.md now state the real default and that string ids need
it set to false; with that, the same trace hashes correctly and repeated
keys map to the same id. The default itself is a library behaviour
question, so it is documented rather than changed here.

cachesim accepts tinyLFU, but the registry did not, so MINISIM and any
library caller got "cannot load ./libtinyLFU.so" — contradicting the
registry's documented promise to accept the CLI's aliases. Added as a
plain alias, and removed the special case that used to append
window-size=0.01 when the caller had not given one: WTinyLFU's
DEFAULT_PARAMS already sets exactly that before applying caller
parameters, so it never changed anything. Verified identical between
tinyLFU and wtinyLFU with no params, with main-cache=LRU, with
window-size=0.2, and with both.

testCLI covers tinyLFU through cachesim and MINISIM, and a csv of string
ids asserting the miss ratio that four distinct objects give, so
collapsing them to id 0 shows up as a failure rather than a plausible
number. 174 checks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Two review findings from the previous round.

The MINISIM belady guard only accepted oracleGeneral and lcs, but the four
oracleTwr/oracleTwrNS readers populate req->next_access_vtime too, so belady
works on those and was being rejected. Move the list into a helper next to
the reader formats that set the field, and name the actual trace type in the
error so it says what was wrong rather than only what is allowed.

quickstart_traceAnalyzer.md embeds its plots with raw <img> tags pointing at
/doc/plot/..., which resolve on github.com but became site-root paths in the
built HTML, so all 18 images 404'd. Copy doc/plot and doc/assets into the
output with html_extra_path and strip the prefix in the same source-read hook
that already rewrites the Markdown links.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eb46342371

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +36 to +37
default:
return false;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor next-access fields in generic binary traces

When a caller opens a BIN_TRACE with reader_init_param_t.next_access_vtime_field configured, generalReader/binary.c explicitly populates req->next_access_vtime, so the trace contains the future information Belady needs. This default branch nevertheless classifies every generic binary reader as unsupported, causing MINISIM to abort on a valid oracle-style binary trace; base the check on the configured field or the reader's actual request data rather than only its trace-type enum.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 4466ab2. binary.c:263 does populate the field whenever next_access_vtime_field_idx > 0, so classifying by the enum alone was wrong.

The helper now takes the reader rather than the trace type, and asks it:

case BIN_TRACE:
  return reader->init_params.next_access_vtime_field > 0;

binary.c is the only general reader with such a field — csv has none — so BIN_TRACE is the sole addition.

No CLI exposes next_access_vtime_field, so this isn't reachable from the mrcProfiler binary. It is reachable from the library, since mrcProfiler_lib is folded into the main target, so I verified it that way: opened cloudPhysicsIO.oracleGeneral.bin as a plain BIN_TRACE with "<IQIq" and next_access_vtime_field = 4, then ran MINISIM belady through the library API.

trace_type=BIN_TRACE next_access_vtime_field=4
first req: obj_id=42932745 size=512 next_access_vtime=9223372036854775807
ACCEPTED: MINISIM belady ran on a BIN_TRACE
cache_size   miss_rate
20971520B    0.814072
104857600B   0.734158

0.814072 matches the oracleGeneral run at the same size exactly, so it is the same curve rather than merely a curve. With the field left unset the same trace is still rejected (rc=134), and .vscsi still is too.

One thing that turned up while verifying, which I have not changed: on the first attempt belady aborted with next access time is -1, please use INT64_MAX instead. oracleGeneralBin_read_one_req() normalises the -1/INT64_MAX sentinel to MAX_REUSE_DISTANCE; the generic binary reader passes the raw value through. So a BIN_TRACE whose sentinel is -1 reaches Belady and is loudly rejected there. That is a pre-existing gap in binary.c rather than something this guard should paper over, and normalising it would change what the reader hands every caller — a maintainer call, so I left it and used an INT64_MAX-sentinel trace for the run above.


Generated by Claude Code

Comment thread doc/index.md Outdated

New here? Start with [Install & Build](install.md), then [the cachesim guide](quickstart_cachesim.md).

The commands throughout these pages are run from the build directory (`_build/` if you followed the [README](https://github.com/1a1a11a/libCacheSim#build-and-install-libcachesim)), so the sample traces in `data/` are at `../data/`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Scope the build-directory instruction to binary guides

This blanket instruction is false for several pages included in the new toctree. For example, quickstart_traceAnalyzer.md runs python3 scripts/traceAnalysis/... and debug.md runs ./scripts/debug.sh; from _build/ both paths are missing because scripts/ is in the repository root. Readers following the new site-wide cwd rule therefore get immediate file-not-found errors, so limit the statement to the binary quickstarts or update those commands to use paths relative to _build/.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct — that sentence was mine and it overclaimed. Fixed in 4466ab2.

Counted which pages use which convention rather than guessing:

page ../data/ (build dir) root-relative
quickstart_cachesim.md 32 0
quickstart_mrcProfiler.md 7 1
quickstart_traceUtils.md 4 0
quickstart_traceAnalyzer.md 4 9
performance.md 2 3
debug.md 0 19

So both examples you named hold: debug.md is entirely repo-root (./scripts/debug.sh -- data/cloudPhysicsIO.vscsi ..., note data/ and not ../data/), and quickstart_traceAnalyzer.md is genuinely mixed — the analyzer runs from _build/ while its nine python3 scripts/traceAnalysis/*.py follow-ups run from the root.

I took the scoping option rather than rewriting the commands, since the split is inherent — the binaries live under _build/ and the scripts under scripts/ — and repointing nine plotting invocations I can't exercise here would risk more than it fixes. The note now reads:

Commands that invoke a built binary — ./bin/cachesim, ./bin/traceAnalyzer, ./bin/mrcProfiler — are run from the build directory (_build/ …), which is why the sample traces in data/ appear as ../data/. The helper scripts under scripts/, such as the plotting scripts and debug.sh, are run from the repository root instead, and those pages write the paths accordingly.

sphinx-build -W is clean and ctest is 10/10.


Generated by Claude Code

claude added 2 commits August 13, 2026 15:54
Two more review findings.

The oracle-trace check classified readers by trace type alone, but the generic
binary reader populates req->next_access_vtime whenever the caller points
next_access_vtime_field at the right column, so a configured BIN_TRACE carries
the future information Belady needs and was still being rejected. Ask the
reader instead of assuming from the enum. No CLI exposes that field, but the
profiler is part of the library, so a library caller can set it up.

doc/index.md claimed every command in the docs runs from the build directory.
That holds for the binaries, which is why the sample traces appear as
../data/, but not for the helper scripts: quickstart_traceAnalyzer.md invokes
python3 scripts/traceAnalysis/*.py and debug.md invokes ./scripts/debug.sh,
both relative to the repository root. Scope the sentence to what is true.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…plot

Relicense from GPL-3.0 to Apache-2.0. LICENSE is now the canonical Apache
text, byte-identical to apache.org, and the declarations that named GPL are
updated to match: CITATION.cff, libCacheSim-node/package.json, the node
README, CONTRIBUTING.md, and the README license section, which now names the
license instead of only linking it. The vendored LHD code under
libCacheSim/cache/eviction/LHD keeps its own MIT notice, which Apache-2.0
accommodates.

The generic binary reader passed the "no next access" sentinel through raw,
while the oracle readers normalize -1 to MAX_REUSE_DISTANCE. A binary trace
carrying -1 therefore reached Belady, which rejects it outright. Normalize it
in binary.c so the two readers agree: the same file read either way now
produces an identical miss ratio curve.

The popularity-decay section pointed at a plot that has never existed in the
repo, which is why it was commented out; meanwhile the w92 plot generated for
exactly that section sat unreferenced. Point the section at the real file and
rewrite the caption, which had been copied from the object-size section and
described two plots and the wrong quantity. All 18 images in the rendered
docs now resolve.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
@1a1a11a 1a1a11a changed the title Polish repo for open source: fix broken docs, CLI crashes, and missing community files Polish the repo for open-source readiness (relicense to Apache-2.0, fix documented crashes and leaks) Aug 13, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b0aab6a6f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

/* these need future information and are only valid on oracle traces, so
* callers that know the trace type should check before using them */
{"belady", Belady_init},
{"beladySize", BeladySize_init},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve BeladySize's reduced hash table in MINISIM

When --algo=beladySize --profiler=MINISIM is used, this direct registry entry bypasses the BeladySize-specific reduction in bin/cachesim/cache_init.h, while MINISIM supplies hash power 20 to every profile point. BeladySize therefore allocates a 1M-slot table per miniature cache rather than the reduced power-16 table; with many profile points this can multiply memory use by 16 and trigger the algorithm's own “hash power too large” retry path or an OOM. Apply the same reduction before constructing BeladySize through the registry.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in b68dcee. Measured peak RSS for an 8-point MINISIM run before changing anything:

algo before after
beladySize 32.0 MB 10.2 MB
belady 14.8 MB 14.9 MB
lru 13.7 MB 14.7 MB

So the effect is real and specific to BeladySize, as you said.

Two corrections to the diagnosis, both in your favour on the substance:

Hyperbolic is not affected, despite having the same reduction in cache_init.h. Hyperbolic_init shrinks its own table (MAX(12, hashpower - 8) at Hyperbolic.c:47), so it converges to 12 from either entry point — which is why it measured 10.2 MB going in. BeladySize has no such line, so it really is the only one.

The retry path isn't the mechanism. BeladySize_to_evict recurses when no sampled object beats the initial score, not when the table is sparse — chained_hashtable_rand_obj_v2 loops until it hits a non-empty bucket and shrinks the table itself after 32 misses, so it never hands back NULL. The sparse table costs memory and wasted probes, not the retry. I didn't observe the warning in any run.

On where to apply it. I first did the tidier thing and moved the reduction into BeladySize_init alongside Hyperbolic's, deleting it from cache_init.h — algebraically identical for cachesim at every hash power (MAX(hp-8,16) either way; verified byte-identical miss ratios at 18, 24 and 30 against a reverted build). But testEvictionAlgo then failed:

assertion failed (miss_cnt_true[i] == res[i].n_miss): (74329 == 74347)

That test builds BeladySize at hash power 20 — the same configuration you flagged — and records the unreduced numbers. Putting the reduction in the algorithm overrides a hash power the caller explicitly asked for, so it changes results for every library caller, not just MINISIM. That is a modelling decision rather than a defect, so I left it to you and scoped this commit to the profiler that had the problem.

Worth your eye as a follow-up: BeladySize and Hyperbolic disagree about whose job this is, and the library and the CLI consequently disagree about BeladySize's effective table size. Making BeladySize self-shrink would settle it, at the cost of re-recording that test's expected counts.

MINISIM's beladySize curve does move slightly, since the sampler now draws from a right-sized table (e.g. 0.629654 → 0.626142 at wss 0.091). Confirmed deterministic across three runs, so those deltas are the fix and not sampling noise. ctest is 10/10.


Generated by Claude Code

BeladySize picks its victim by sampling the hash table, so cachesim shrinks
the table by 8 before constructing it. MINISIM builds its miniature caches
straight from the registry at a fixed hash power of 20, so it skipped that and
gave every profile point a 1M-slot table. Peak RSS for an 8-point run drops
from 32.0 MB to 10.2 MB.

Applied in the profiler rather than in BeladySize_init. Moving it into the
algorithm, the way Hyperbolic_init does, is arguably where it belongs and
would fix every caller at once, but it overrides the hash power a library
caller explicitly asks for: test_evictionAlgo builds BeladySize at hash power
20 and its recorded miss counts move (74329 -> 74347). That is a modelling
change for the library API rather than a defect, so it is left for the
maintainers; this commit changes only the profiler that had the problem.

Hyperbolic needs no equivalent line because Hyperbolic_init already shrinks
its own table, which is why it was unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b68dcee1e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +347 to +348
if (strcasecmp(params_.cache_algorithm_str, "belady") == 0 ||
strcasecmp(params_.cache_algorithm_str, "beladySize") == 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remap BeladySize future times after spatial sampling

When MINISIM profiles beladySize with a sampling rate at or below 0.5, the reader filters the trace but leaves each oracle request's next_access_vtime in the original trace's coordinate system. BeladySize_to_evict() subtracts the sampled cache's n_req from that original timestamp when calculating its size-distance score, so the eviction priorities—and therefore the reported MRC—are invalid. Either remap future times to sampled virtual time or reject sampled BeladySize runs.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Real, and the best finding of the review — thanks. Documented and warned in d946288.

I designed a control experiment rather than taking it on faith, since sampling always introduces error and the question is whether BeladySize has excess error. Your analysis predicts BeladySize is hit but Belady is not, because Belady uses next_access_vtime as an ordering and never takes a difference, so a constant coordinate offset cancels. MINISIM also skips sampling above rate 0.5, which gives a boundary to test across.

Error against the unsampled miss ratio, cloudPhysicsIO, 100 MB cache:

sample rate beladySize belady lru
0.6 (no sampler) 0.000002 0.000014 0.000033
0.5 (sampler on) 0.0126 0.0003 0.0023
0.1 0.0245 0.0155 0.0156

At 0.5, where ordinary sampling error is still negligible, BeladySize is already 42x Belady's error and 5.5x LRU's, and it jumps discontinuously the moment the sampler switches on. Belady behaves exactly as your reasoning predicts. That is the coordinate mismatch, not sampling noise.

You offered "remap future times or reject sampled BeladySize runs". I did neither, and took a third option — warn loudly and document — for a reason worth stating.

Remapping is a change to the sampler's contract with its own accuracy trade-offs, and rejecting removes a capability over an error that is real but bounded (the curve is in the right region, just degraded). Both are modelling calls that belong to the maintainers. What was actually indefensible was reporting the number without saying it was degraded, and that is fixed: the profiler now warns when and only when the sampler is active, and doc/quickstart_mrcProfiler.md carries the mechanism and the measured numbers so anyone reading a sampled BeladySize curve knows what they have.

[WARN] beladySize scores candidates by reuse distance, which spatial sampling
distorts because next_access_vtime stays in full-trace request numbers; the
curve is approximate beyond the usual sampling error. Use
--profiler-params=FIX_RATE,1,<threads> for an exact run, or belady, which is
not affected.

Verified the warning fires under sampling and stays silent at FIX_RATE,1. ctest 10/10, sphinx-build -W clean.


Generated by Claude Code

{"clockpro", ClockPro_init},
{"fifo", FIFO_init},
{"fifo-merge", FIFO_Merge_init},
{"fifo-reinsertion", Clock_init},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Map FIFO-reinsertion to its actual constructor

When MINISIM or a library caller requests fifo-reinsertion through the new name registry, this entry silently constructs a Clock cache even though FIFO_Reinsertion_init is available. Experiments therefore complete successfully while measuring the wrong eviction policy; register FIFO_Reinsertion_init for this name.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this one against develop before touching it, and it isn't a regression — that mapping is pre-existing:

$ git show origin/develop:libCacheSim/bin/cachesim/cache_init.h | grep reinsertion
      {"fifo-reinsertion", Clock_init},

The registry is a verbatim move of cachesim's own table, so cachesim fifo-reinsertion has always built a Clock cache, and MINISIM now builds the same thing. Repointing the name at FIFO_Reinsertion_init would change what every existing fifo-reinsertion invocation simulates, which is exactly the kind of silent result change this PR has stayed away from — so it is yours to make, not mine to slip into a refactor whose whole claim is that it preserves behavior.

For what it's worth the alias is defensible on the merits: Clock is FIFO-with-reinsertion, and the two implementations are the same policy. So this may well be deliberate rather than a slip.

The thing genuinely worth your attention is the other half: FIFO_Reinsertion_init is reachable under no name at all, on develop or here — grep FIFO_Reinsertion against develop's table returns nothing. It is exported from evictionAlgo.h and used internally by WTinyLFU as a main-cache option, but no CLI or registry name resolves to it. So it is either dead as a standalone policy and could be documented as internal-only, or it wants its own registry entry under a distinct name. Both are one-line changes once you decide which.


Generated by Claude Code

BeladySize ranks candidates by next_access_vtime - cache->n_req. The first
term counts requests in the full trace; the second counts only the requests
the spatial sampler kept, so under sampling the reuse distance is inflated and
the eviction order degrades. Belady is unaffected: it compares future times
directly and never takes a difference.

Measured on cloudPhysicsIO at a 100MB cache, error against the unsampled miss
ratio:

  sample rate 0.5   beladySize 0.0126   belady 0.0003   lru 0.0023
  sample rate 0.1   beladySize 0.0245   belady 0.0155   lru 0.0156

so at 0.5, where ordinary sampling error is still small, BeladySize is already
an order of magnitude worse than either control.

Warn and document rather than refuse or silently correct. Remapping future
access times into sampled virtual time is a change to the sampler with its own
accuracy trade-offs, which is the maintainers' call; what mattered here was
that the profiler had been reporting the number without saying it was degraded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

1a1a11a commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

Split into five smaller PRs

This PR is now superseded — please review the five below instead. I am leaving it open until they land, since the review discussion here (15 threads, with the measurements behind each fix) is the fullest record of how these bugs were found.

PR Contents Base Files
#326 Community health files, CI hardening, Apache-2.0 relicense develop 21
#327 Documentation examples and the Read the Docs build develop 17
#328 Crashes and memory leaks reachable from documented commands develop 38
#329 Algorithm registry shared with the MRC profiler, --hashpower #328 10
#330 The testCLI regression target #329 2

#326, #327 and #328 are independent and can merge in any order. #329 and #330 stack on #328, which is why they are based on it rather than develop.

Splitting by file rather than by commit is what makes that work: every changed file belongs to exactly one PR, so nothing needs a hand-reconstructed intermediate state and no two PRs touch the same file. The cost is that #328 carries the hash-power clamps in the eviction algorithms whose motivation only appears in #329, and #327's mrcProfiler guide describes MINISIM behavior that #329 delivers.

Verification that nothing was lost or altered in the split

Merging all five branches together and diffing against this PR's head:

$ git merge claude/polish-1-repo-meta claude/polish-2-docs  # onto the 3-4-5 stack
$ git diff --quiet <union> d946288 && echo IDENTICAL
IDENTICAL

They merge without conflict and reproduce this branch byte for byte. Each was also built and tested on its own branch, not just as part of the whole: #328 and #329 are 9/9 on ctest (the tenth target is testCLI, which arrives in #330), #330 is 10/10, and #327's sphinx-build -W is clean standing alone.

Still open for you, carried over


Generated by Claude Code

claude added 3 commits August 13, 2026 17:35
…sim's hash power on exact MINISIM runs

WTinyLFU_evict checked the main cache's capacity using cache->obj_md_size,
which is the larger of the two sub-caches so that a caller asking the
composite what it reserves is not told less than it really does. Against a
FIFO or Clock main cache, which reserve nothing, that billed the window's 16
bytes and called the main cache full early; with an empty main cache it could
reach to_evict() and dereference NULL. Each sub-cache is now charged its own.

MINISIM built its miniature caches at hash power 20 even when sampling was
disabled, while cachesim defaults to 24. Random, RandomTwo, RandomLRU and
Hyperbolic draw eviction candidates through the hash mask, so the supposedly
exact run drifted: randomTwo gave 0.8221 from cachesim and 0.8215 from
MINISIM. Unsampled runs now use cachesim's hash power, including the extra
reduction cachesim applies to hyperbolic and beladySize, and reproduce it
exactly across all nine algorithms checked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…-repo-polish-wc1xjb

# Conflicts:
#	README.md
#	doc/README.md
plugin.c leaked the dlopen handle when dlsym failed. Nothing from the library
is in use on that path, unlike the success path the existing comment covers,
so it is closed rather than leaked.

testCLI now sweeps hyperbolic, belady and beladySize, which cachesim
special-cases and the registry also carries; the trace is oracleGeneral, so
the Belady policies are valid on it. 174 checks becomes 180. mktemp -d is
given an explicit template, since BSD mktemp rejects the bare form and the
macOS job runs this target. The ctest entry gains SKIP_REGULAR_EXPRESSION, so
a run that skipped itself for want of binaries or traces no longer reports as
a pass.

CONTRIBUTING.md described the CLI test by linking test/test_cli.sh. That file
arrives in a different PR of this series, so the link would dangle for anyone
merging the community files alone; the guidance now leads with what to cover
and why the library tests will not catch it.

adoption.md, which arrived on develop in #323, carries a BibTeX block whose
% comment Pygments' bibtex lexer rejects. Harmless normally, but this branch
sets fail_on_warning, so it broke the docs build; the comment moves into prose
above the block. adoption.md is also added to the Sphinx toctree, which it
needs to be reachable in the rendered docs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5de5522a48

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

],
"author": "Murphy Tian",
"license": "MIT",
"license": "Apache-2.0",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ship the Apache license in the npm package

Declaring the addon as Apache-2.0 is not enough when the published artifact does not contain the license text. I checked npm pack --dry-run --json --ignore-scripts in libCacheSim-node; the tarball contains only six files and no LICENSE, because the repository-level license is outside the package and is not copied by the release workflow. Consequently, every npm publication distributes the statically linked library without the Apache license copy required for redistribution; copy LICENSE into this package (and include it in files) before publishing.

Useful? React with 👍 / 👎.

claude added 2 commits August 13, 2026 17:52
The addon declares Apache-2.0 and binding.gyp statically links
vendor/liblibCacheSim.a, so every publication redistributes the library. The
repository LICENSE sits one directory above the package, outside anything npm
collects, so the tarball carried the declaration without the text:

  npm pack --dry-run  ->  6 files, no LICENSE

Apache-2.0 section 4(a) requires giving recipients a copy of the license, so
add one to the package and list it in "files". The tarball is now 7 files with
LICENSE among them. The gap predates the relicense -- the same tarball shipped
while the package declared GPL-3.0 -- but relicensing is the moment to fix it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
(cherry picked from commit d739e57)
_expect_clean_failure accepted any non-zero exit whose output contained the
word "error". A sanitizer turns a crash into exit 1 with "ERROR:
AddressSanitizer", so under the sanitizer builds this suite is meant to run
in, a crash regression would have been counted as a clean rejection. Sanitizer
signatures are now checked first.

SKIP_REGULAR_EXPRESSION arrived in CMake 3.16 and this project declares 3.12,
where setting it is an error, so it is now guarded by a version check.

The documented csv reader used a bare designated initializer, which zeroes
every field it does not name -- cap_at_n_req, block_size and
ignore_size_zero_req all default to something other than zero. It also set
has_header without has_header_set, and the reader auto-detects unless the flag
is raised, so the assignment did nothing; it happened to work only because
detection guessed right on this file. The snippet now starts from
default_reader_init_params() and raises both _set flags. Compiled and run: it
reads 113872 requests with first id 42932745 size 512, matching the vscsi
reader exactly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc77b326f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

* full-sized; leaving the requested rate in place would scale them down
* while every request still reached them, reporting the miss ratios of
* smaller caches than were asked for */
sample_rate = 1.0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Scale MINISIM caches by the sampler's effective rate

This only normalizes the unsampled path; now that built-in MINISIM runs execute, sampled rates whose reciprocal is not an integer still produce biased curves. For example, FIX_RATE,0.3,... is quantized by create_spatial_sampler() to sampling_ratio_inv = 3, so spatial_sample() retains 1/3 of objects and miss counts are extrapolated by 3, while line 407 sizes every cache using the unquantized 0.3 rate. The sampled workload therefore sees caches about 10% smaller than the rate represented in the results; normalize sample_rate to 1.0 / sampler->sampling_ratio_inv after creating the sampler, or reject rates that cannot be represented.

Useful? React with 👍 / 👎.

create_spatial_sampler stores sampling_ratio_inv as an integer, so it can only
represent rates of the form 1/n: it truncates 1/0.3 to 3 and keeps a third of
objects. The cache sizing used the rate that was asked for rather than the one
in force, making every cache too small by that ratio -- 10% at 0.3 -- while
the curve was still reported against the requested size, so the error did not
show up anywhere in the output.

Demonstrable by asking for two rates that quantize to the same sampler: 0.3
and 0.3333333 both give sampling_ratio_inv 3 and therefore sample identical
objects, so any difference between their curves is the bug and nothing else.

  before   rate=0.3333333  0.8193 0.7637
           rate=0.3        0.8204 0.7788
  after    rate=0.3333333  0.8193 0.7637
           rate=0.3        0.8193 0.7637

Rates that are already of the form 1/n, including the 0.5 boundary and the
0.01 used throughout the docs, are unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5b9472a17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread FAQ.md
Comment on lines 8 to +12
struct {
uint32_t real_time;
uint32_t clock_time;
uint64_t obj_id;
uint32_t obj_size;
int64_t next_access_vtime;
int64_t next_access_vtime; // -1 if there is no next access

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Mark the oracleGeneral record as packed

When a reader uses this struct definition to generate an oracleGeneral trace, normal 64-bit C ABI padding places obj_id at offset 8 and makes the record 32 bytes, while oracleGeneralBin_setup() expects a packed 24-byte record with offsets 0, 4, 12, and 16. The resulting file is silently decoded with the wrong IDs, sizes, and future times; show __attribute__((packed)) or document the exact serialized offsets instead.

Useful? React with 👍 / 👎.

Comment thread FAQ.md Outdated

In the sample [cloudPhysicsIO.csv](/data/cloudPhysicsIO.csv), time is in seconds and object size is in bytes.

`next_access_vtime` is a *logical* time: the number of requests between the current request and the next request to the same object, or `-1` when the object is never accessed again. Algorithms that need future information, such as [Belady](/libCacheSim/cache/eviction/Belady.c) and BeladySize, rely on it, which is why they only work on oracle traces.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document next access as an absolute virtual time

For anyone constructing an oracle trace, this describes the field as a reuse distance, but traceConvOracleGeneral stores the absolute request index of the next access and BeladySize explicitly computes the distance by subtracting the current request count. Encoding the documented distance therefore changes eviction priorities and yields invalid simulation results; describe it as the next access's absolute 1-based virtual timestamp, with -1 for no future access.

Useful? React with 👍 / 👎.

cache_get_base()'s eviction loop charges an incoming object cache->obj_md_size
before deciding whether to evict. Every incoming object is inserted into the
window, so that field has to be the window's overhead; setting it to the
maximum of the two sub-caches made the loop reserve up to 40 bytes for a
16-byte window insertion whenever the main policy was ARC, LeCaR or Cacheus,
evicting earlier than the configuration calls for.

All three sites now charge the cache the object is actually entering: this one
and WTinyLFU_can_insert() the window, WTinyLFU_evict() the main cache on
promotion.

Measured on cloudPhysicsIO at 100MB with metadata on, the effect is small --
LeCaR moves 0.7894 to 0.7895 and ARC, LRU, SLRU and FIFO do not move at all,
because objects are large relative to the difference. Correct rather than
consequential, but it also removes the last place where the composite charged
an overhead belonging to a cache the object was not entering.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e2a30b22a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 300 to 303
if (main_cache->get_occupied_byte(main_cache) +
params->req_local->obj_size + cache->obj_md_size <=
params->req_local->obj_size + main_cache->obj_md_size <=
main_cache->cache_size) {
main_cache->insert(main_cache, params->req_local);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop evicting once promotion frees enough metadata

When consider_obj_metadata=true and the main policy has less metadata than the window LRU (for example, main-cache=FIFO), moving a victim from the window into the main cache can itself reduce total occupancy by 16 bytes and make enough room for the pending request. This branch nevertheless leaves evicted false, so the loop continues until it discards an additional object, causing WTinyLFU to underfill the cache and alter miss ratios. Recheck the composite capacity after the promotion and return when the pending window insertion now fits.

Useful? React with 👍 / 👎.

claude added 4 commits August 13, 2026 18:28
The sweeps skipped any algorithm whose run reported "do not support
algorithm", on the assumption that the message means an optional feature was
not compiled in. cache_init.h emits the same message when a name is missing
from the registry for any reason, so a mandatory algorithm silently dropping
out of g_cache_algos would have been skipped rather than reported -- the exact
regression the registry refactor could introduce, and the one this sweep is
here to catch.

Only 3LCache, GLCache, gl-cache and lrb sit behind build flags, so the skip is
restricted to those and everything else fails.

Verified by deleting an entry and rebuilding. gdsf is covered by the sweeps
and nothing else, so before this it disappeared without a sound:

  FAIL: gdsf -e print (exit 134)   do not support algorithm gdsf
  FAIL: gdsf replay (exit 134)     do not support algorithm gdsf
  (4 algorithms not compiled in, skipped)

The four genuinely optional ones still skip.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
…-repo-polish-wc1xjb

# Conflicts:
#	libCacheSim/bin/cachesim/cache_init.h
MultiQueue arrived on develop in #318 with the same leak this branch fixed in
29 other files: MQ_parse_params strdup's the parameter string and frees it at
the end, but the "print" branch calls exit(0) first. LeakSanitizer reports
6 bytes, the length of "print".

Found by CI rather than by reading: merging develop added mq and multiqueue to
the registry, the testCLI sweep covers every registered name, and the ubuntu
job runs it under LeakSanitizer:

  FAIL: mq -e print (exit 23)
        SUMMARY: LeakSanitizer: 6 byte(s) leaked in 1 allocation(s).
  FAIL: multiqueue -e print (exit 23)

which is the sweep doing exactly what it was added for -- a brand-new
algorithm inherited the defect the same day it landed, and nothing else in the
suite would have looked at it.

Reproduced and confirmed fixed in a local -fsanitize=leak build with CI's
ASAN_OPTIONS: ctest 10/10, testCLI 184 checks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
cache_struct_init() reads a non-positive hashpower as "caller did not choose",
substituting HASH_POWER_DEFAULT. The clamps added for --hashpower ran before
that check, so MAX(4, 0 - 4) turned the sentinel into a literal hash power of
4 -- a 16-bucket table -- for any library caller using a designated
initializer without setting the field. The reduction now applies only to an
explicitly positive value, in Cacheus, S3FIFOd, SLRUv0 and LP_SFIFO alike.

Worth recording that the predicted consequence was backwards: restoring the
sentinel makes the sample replay *slower*, 54ms to 301ms, because
HASH_POWER_DEFAULT allocates 8M buckets per sub-cache while a 16-bucket table
simply grows. The point stands anyway -- 301ms is what the library did before
this branch, and silently redefining an unset hashpower is not a change to
make in passing.

The bug the clamps were added for is still fixed: cachesim slruv0 memory stays
monotonic across --hashpower 4, 5, 6, 8 and 24, with no inversion.

Also: FAQ.md described next_access_vtime as the number of requests until the
next access. It is the absolute 1-based request index of that access, and
algorithms subtract the current request count themselves, so a trace built to
the documented meaning would evict in the wrong order. Verified against the
sample: request 7 stores 19, and that object is next requested at request 19.

And testCLI's unsampled-MINISIM equality check compared two extracted strings
without requiring either to be non-empty, so if both extractions stopped
matching it would have passed on "" == "" -- the same shape as the skip logic
fixed earlier today, an assertion that can succeed without observing anything.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YUq1vM4g82TkaX2jvLmuQc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants