From a3fca4f04e2cf28b4fc6c15cd27ce14f0af40621 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 14 Aug 2026 19:51:50 +0000 Subject: [PATCH] Fix documentation examples and make the Read the Docs build work --- .github/copilot-instructions.md | 2 +- .readthedocs.yaml | 13 +- FAQ.md | 48 ++++- doc/API.md | 311 ++++++++++++++++++++------------ doc/README.md | 6 + doc/adoption.md | 3 +- doc/advanced_lib.md | 106 ++++++----- doc/advanced_lib_extend.md | 6 +- doc/conf.py | 106 +++++++++++ doc/index.md | 58 ++++++ doc/install.md | 16 +- doc/memory_usage_profiling.md | 2 +- doc/performance.md | 61 ++++++- doc/quickstart_cachesim.md | 94 ++++++---- doc/quickstart_traceAnalyzer.md | 47 +++-- doc/quickstart_traceUtils.md | 10 +- doc/requirements.txt | 6 + 17 files changed, 636 insertions(+), 259 deletions(-) create mode 100644 doc/conf.py create mode 100644 doc/index.md create mode 100644 doc/requirements.txt diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index ed9c8e72f..d723f6712 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -17,7 +17,7 @@ - Use a standard out-of-source CMake build for release-style work: `cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release` then `cmake --build build`. - CMake tests are enabled by default. Run `ctest --test-dir _build_dbg --output-on-failure` after the debug build, or the equivalent `build/` test directory if you used a separate build tree. - If you change installation, packaging, or the public library surface, also review `test/test_lib.sh`. -- Use sample traces in `data/` for quick validation unless the task specifically requires the large traces in `2024_google/`. +- Use sample traces in `data/` for quick validation. They are deliberately tiny, so never use them to compare miss ratios between algorithms; larger traces are listed at https://github.com/cacheMon/cache_dataset. ## Project-Specific Conventions - When adding a new eviction algorithm, reader, or plugin, follow `doc/advanced_lib_extend.md` instead of inventing a new integration path. These changes usually require updates to implementation files, registration headers, CMake lists, CLI/parser wiring, and tests. diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 87c2cd65e..7b0379bf2 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -11,13 +11,14 @@ build: tools: python: "3.11" -# Build documentation in the docs/ directory with Sphinx +# The documentation sources are the Markdown files in doc/, rendered with MyST. sphinx: - configuration: docs/conf.py + configuration: doc/conf.py + # The build is warning-free; keep it that way, since a broken cross-reference + # is otherwise easy to miss. + fail_on_warning: true -# We recommend specifying your dependencies to enable reproducible builds: -# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html +# Docs-only dependencies; the root requirements.txt is for the analysis scripts. python: install: - - requirements: requirements.txt - + - requirements: doc/requirements.txt diff --git a/FAQ.md b/FAQ.md index 1ff4995b7..fdeaa8af8 100644 --- a/FAQ.md +++ b/FAQ.md @@ -1,17 +1,47 @@ -## FAQ -1. **how to read OracleGeneral trace，how to transform from csv to it? ** -The [oracleGeneral](/libCacheSim/traceReader/customizedReader/oracle/oracleGeneralBin.h) trace is a binary trace, so you cannot direct read as txt file. Each request uses the following data struct +# FAQ + +### How do I read an oracleGeneral trace, and how do I convert a csv trace into one? + +The [oracleGeneral](/libCacheSim/traceReader/customizedReader/oracle/oracleGeneralBin.h) trace is a binary format, so it cannot be read as a text file. Each request is the following struct: + ```c struct { - uint32_t real_time; + uint32_t clock_time; uint64_t obj_id; uint32_t obj_size; - int64_t next_access_vtime; + int64_t next_access_vtime; // -1 if there is no next access }; ``` -* Read the trace: we have provided a tool `tracePrint` that you can use to print the trace in plain text, it is compiled and under `bin/` -* Convert csv to oracleGeneral: we have provided `traceConv` to convert traces. The help menu should be sufficient to get started. +* **Read the trace**: use `tracePrint` to print the trace as plain text. It is built into `bin/` alongside `cachesim`. + ```bash + ./bin/tracePrint ../data/cloudPhysicsIO.oracleGeneral.bin oracleGeneral + ``` +* **Convert a csv trace**: use `traceConv`. See [quickstart_traceUtils.md](/doc/quickstart_traceUtils.md), or run `./bin/traceConv --help`. + ```bash + ./bin/traceConv ../data/cloudPhysicsIO.csv csv \ + -t "time-col=2,obj-id-col=5,obj-size-col=4,obj-id-is-num=1" \ + --output-format=oracleGeneral + ``` + +oracleGeneral traces are usually stored zstd-compressed, and libCacheSim reads them without decompressing first. + +### What are the units in a trace? + +In the sample [cloudPhysicsIO.csv](/data/cloudPhysicsIO.csv), time is in seconds and object size is in bytes. + +`next_access_vtime` is a *logical* time: the 1-based request index at which this object is next requested — an absolute position in the trace, not the distance to it — or `-1` when the object is never accessed again. Algorithms subtract the current request count themselves, so encoding a distance here silently changes eviction order. In `cloudPhysicsIO.oracleGeneral.bin`, for instance, request 7 stores `19` and that object is next seen at request 19. Algorithms that need future information, such as [Belady](/libCacheSim/cache/eviction/Belady.c) and BeladySize, rely on it, which is why they only work on oracle traces. + +Object ids are hashed unless the reader is told they are already numeric. Pass `obj-id-is-num=true` in `--trace-type-params` when the id column holds numbers — `cachesim` stops with an error if you leave it out on such a trace. + +### Why does `cachesim` say "do not support algorithm X"? + +Some algorithms are behind an optional build flag because they pull in extra dependencies: GLCache (`-DENABLE_GLCACHE=ON`), LRB (`-DENABLE_LRB=ON`), and 3LCache (`-DENABLE_3L_CACHE=ON`). Rebuild with the relevant flag to enable them. See the [README](/README.md#supported-algorithms) for the full list. + +### Where can I get larger traces? + +The traces in [data/](/data/) are samples and are **far too small to compare miss ratios between algorithms**. We maintain a list of open-source cache datasets at [cacheMon/cache_dataset](https://github.com/cacheMon/cache_dataset). + +--- -2. **What are the units in the trace? ** -In the [trace.csv](/data/trace.csv), the time unit is in sec, the next_access_time is the logical time (# requests) between current and the next request (to the same object). The next access time is used by some algorithms that require future information, e.g., Belady. The object id is a hash of raw object id (string or numeric value). +More questions? Check the [documentation index](/doc/README.md), search the [issue tracker](https://github.com/1a1a11a/libCacheSim/issues), or ask in [Discussions](https://github.com/1a1a11a/libCacheSim/discussions). diff --git a/doc/API.md b/doc/API.md index 9a6011d71..14530c5af 100644 --- a/doc/API.md +++ b/doc/API.md @@ -1,148 +1,231 @@ -traceReader: -```C -typedef struct { - int time_field; - int obj_id_field; - int obj_size_field; - int op_field; - int ttl_field; +# C API reference - // csv reader - gboolean has_header; - char delimiter; +The public C API is exposed through a single header: - // binary reader - char binary_fmt[MAX_BIN_FMT_STR_LEN]; -} reader_init_param_t; +```c +#include +``` -typedef struct reader { - char *mapped_file; /* mmap the file, this should not change during runtime */ - uint64_t mmap_offset; +Compile against it with pkg-config: - FILE *file; - size_t file_size; +```bash +gcc your_program.c $(pkg-config --cflags --libs libCacheSim glib-2.0) -o your_program -lm -lzstd +``` - trace_type_e trace_type; /* possible types see trace_type_t */ +See [advanced_lib.md](advanced_lib.md) for a walkthrough and [the example folder](/example) for complete programs. The declarations below are the commonly used subset; the headers under [libCacheSim/include/libCacheSim/](/libCacheSim/include/libCacheSim/) are authoritative. - size_t item_size; /* the size of one record, used to - * locate the memory location of next element, - * when used in vscsiReaser and binaryReader, - * it is a const value, - * when it is used in plainReader or csvReader, - * it is the size of last record, it does not - * include LFCR or \0 */ +--- - uint64_t n_total_req; /* number of requests in the trace */ - uint64_t n_uniq_obj; /* number of objects in the trace */ +## Reading traces - char trace_path[MAX_FILE_PATH_LEN]; - reader_init_param_t init_params; +### Opening a trace - void *reader_params; - void *other_params; /* currently not used */ +`reader_init_param_t` describes how to interpret a trace. Field indices are 1-based, and `0` means the field is absent. Start from `default_reader_init_params()` rather than zero-initializing, so the defaults for delimiter, sampling, and the "was this set by the user" flags are correct. - gint ver; +```c +typedef struct { + bool ignore_obj_size; + bool ignore_size_zero_req; + bool obj_id_is_num; + bool obj_id_is_num_set; // whether the user passed this parameter + int64_t cap_at_n_req; // process at most n requests + + int32_t time_field; + int32_t obj_id_field; + int32_t obj_size_field; + int32_t obj_cost_field; + int32_t op_field; + int32_t ttl_field; + int32_t cnt_field; + int32_t tenant_field; + int32_t next_access_vtime_field; + + int32_t n_feature_fields; + int32_t feature_fields[N_MAX_FEATURES]; + + // block cache; breaks a large request into per-block requests + int32_t block_size; - bool cloned; // true if this is a cloned reader, else false + // csv reader + bool has_header; + bool has_header_set; // false alone cannot distinguish "unset" + char delimiter; + + // skip metadata at the start of a binary trace + ssize_t trace_start_offset; -} reader_t; + // binary reader, a Python struct format string + char *binary_fmt_str; + + sampler_t *sampler; +} reader_init_param_t; + +static inline reader_init_param_t default_reader_init_params(void); /** - * setup the reader struct for reading trace - * @param trace_path + * open a trace for reading; the reader must be released with close_trace() * @param trace_type CSV_TRACE, PLAIN_TXT_TRACE, BIN_TRACE, VSCSI_TRACE, - * TWR_BIN_TRACE - * @param setup_params - * @return a pointer to reader_t struct, the returned reader needs to be - * explicitly closed by calling close_reader or close_trace + * ORACLE_GENERAL_TRACE, TWR_BIN_TRACE, LCS_TRACE, ... */ -reader_t *setup_reader(const char *trace_path, const trace_type_e trace_type, - const reader_init_param_t *const reader_init_param); +reader_t *setup_reader(const char *trace_path, trace_type_e trace_type, + const reader_init_param_t *reader_init_param); -/* this is the same function as setup_reader */ -static inline reader_t * -open_trace(const char *path, const trace_type_e type, - const reader_init_param_t *const reader_init_param) { - return setup_reader(path, type, reader_init_param); -} +/* same function as setup_reader, and the more commonly used name */ +static inline reader_t *open_trace(const char *path, trace_type_e type, + const reader_init_param_t *reader_init_param); +``` -/** - * read one request from reader, and store it in the pre-allocated request_t req - * @param reader - * @param req - */ -uint64_t get_num_of_req(reader_t *const reader); +> [!IMPORTANT] +> `default_reader_init_params()` sets `obj_id_is_num` to **true**, which is the opposite of what `cachesim` does. Set it to `false` yourself if the id field holds strings. The csv reader hashes string ids only on the `false` path; with `true` it runs them through `strtoull()`, which warns and yields `0`, so every object collapses into one and the miss ratio is meaningless. -/** - * as the name suggests - * @param reader - * @return - */ -static inline trace_type_e get_trace_type(const reader_t *const reader) { - return reader->trace_type; -} +```c +reader_init_param_t p = default_reader_init_params(); +p.obj_id_is_num = false; /* string ids: hash them */ +``` -/** - * read one request from reader/trace, stored the info in pre-allocated req - * @param reader - * @param req - * return 0 on success and 1 if reach end of trace - */ -int read_one_req(reader_t *const reader, request_t *const req); +### Iterating over requests -/** - * reset reader, so we can read from the beginning - * @param reader - */ -void reset_reader(reader_t *const reader); +```c +/* read one request into the pre-allocated req; returns 0 on success, + * 1 at end of trace */ +int read_one_req(reader_t *reader, request_t *req); -/** - * close reader and release resources - * @param reader - * @return - */ -int close_reader(reader_t *const reader); +/* number of requests in the trace */ +int64_t get_num_of_req(reader_t *reader); -/** - * clone a reader, mostly used in multithreading - * @param reader - * @return - */ -reader_t *clone_reader(const reader_t *const reader); +static inline trace_type_e get_trace_type(const reader_t *reader); +static inline bool obj_id_is_num(const reader_t *reader); + +/* rewind so the trace can be read again */ +void reset_reader(reader_t *reader); + +/* clone a reader; the usual way to feed one trace to several threads */ +reader_t *clone_reader(const reader_t *reader); +int close_reader(reader_t *reader); +static inline int close_trace(reader_t *reader); ``` -cache and cacheAlgo: +Positioning helpers, used mostly by the analysis tools: -```C -static inline request_t *new_request(); -static inline void copy_request(request_t *req_dest, request_t *req_src); -static inline request_t *clone_request(request_t *req); +```c +void read_first_req(reader_t *reader, request_t *req); +void read_last_req(reader_t *reader, request_t *req); +int skip_n_req(reader_t *reader, int N); +int go_back_one_req(reader_t *reader); +void reader_set_read_pos(reader_t *reader, double pos); /* pos in [0, 1] */ +``` + +--- + +## Requests + +A `request_t` is the container `read_one_req()` fills in. Allocate one up front and reuse it for the whole trace. + +```c +static inline request_t *new_request(void); +static inline void copy_request(request_t *req_dest, const request_t *req_src); +static inline request_t *clone_request(const request_t *req); static inline void free_request(request_t *req); -static inline void print_request(request_t *req); +static inline void print_request(const request_t *req); +``` + +The fields you normally read are `obj_id`, `obj_size`, `clock_time`, `next_access_vtime` (oracle traces only), and `obj_cost`. + +--- + +## Caches + +Every eviction algorithm exposes an `_init` function taking the common parameters plus an optional algorithm-specific parameter string — the same string `cachesim` takes with `-e`. + +```c +typedef struct { + uint64_t cache_size; + uint64_t default_ttl; + int32_t hashpower; + bool consider_obj_metadata; +} common_cache_params_t; + +common_cache_params_t default_common_cache_params(void); + +cache_t *LRU_init(common_cache_params_t ccache_params, + const char *cache_specific_params); +/* ... and FIFO_init, ARC_init, S3FIFO_init, Sieve_init, and the rest; + * see libCacheSim/include/libCacheSim/evictionAlgo.h */ +``` + +A `cache_t` is used through its function pointers: + +```c +/* the whole interface: lookup plus on-demand insert and evict. + * returns true on a cache hit */ +bool (*get)(cache_t *, const request_t *); + +/* look up without the insert/evict; update_cache controls whether the + * lookup also updates state such as recency */ +cache_obj_t *(*find)(cache_t *, const request_t *, bool update_cache); + +bool (*can_insert)(cache_t *, const request_t *); +cache_obj_t *(*insert)(cache_t *, const request_t *); + +/* which object would be evicted, without evicting it */ +cache_obj_t *(*to_evict)(cache_t *, const request_t *); +void (*evict)(cache_t *, const request_t *); + +/* user-triggered removal; eviction should go through evict instead */ +bool (*remove)(cache_t *, obj_id_t); + +void (*cache_free)(cache_t *); ``` -simulator: -```C -sim_res_t * -simulate_at_multi_sizes(reader_t *const reader, - const cache_t *const cache, - const gint num_of_sizes, - const guint64 *const cache_sizes, - reader_t *const warmup_reader, - const double warmup_perc, - const gint num_of_threads); - - -sim_res_t * -simulate_at_multi_sizes_with_step_size(reader_t *const reader_in, - const cache_t *const cache_in, - const guint64 step_size, - reader_t *const warmup_reader, - const double warmup_perc, - const gint num_of_threads); +Most programs only need `get()`. See [advanced_lib_extend.md](advanced_lib_extend.md) to implement a new algorithm. + +--- + +## Simulator + +Rather than driving the loop yourself, you can hand a trace and a cache to the simulator, which parallelizes across cache sizes or across caches. + +```c +/* one cache, many sizes */ +cache_stat_t *simulate_at_multi_sizes(reader_t *reader, const cache_t *cache, + int num_of_sizes, + const uint64_t *cache_sizes, + reader_t *warmup_reader, + double warmup_frac, int warmup_sec, + int num_of_threads, bool use_random_seed); + +/* one cache, sizes at a fixed step up to the working set size */ +cache_stat_t *simulate_at_multi_sizes_with_step_size( + reader_t *reader_in, const cache_t *cache_in, uint64_t step_size, + reader_t *warmup_reader, double warmup_frac, int warmup_sec, + int num_of_threads, bool use_random_seed); + +/* many caches, each at its own configured size */ +cache_stat_t *simulate_with_multi_caches( + reader_t *reader, cache_t *caches[], int num_of_caches, + reader_t *warmup_reader, double warmup_frac, int warmup_sec, + int num_of_threads, bool free_cache_when_finish, bool use_random_seed); ``` +Each returns an array with one `cache_stat_t` per simulation, which the caller frees: +```c +typedef struct { + int64_t n_warmup_req; + int64_t n_req; + int64_t n_req_byte; + double n_req_cost; + int64_t n_miss; + int64_t n_miss_byte; + double n_miss_cost; + + int64_t n_obj; + int64_t occupied_byte; + int64_t cache_size; + float sampler_ratio; + /* ... */ +} cache_stat_t; +``` -profiler: +Object miss ratio is `n_miss / n_req`, and byte miss ratio is `n_miss_byte / n_req_byte`. diff --git a/doc/README.md b/doc/README.md index 0756a0555..64ce86030 100644 --- a/doc/README.md +++ b/doc/README.md @@ -18,6 +18,12 @@ ## Developer Documentation - [Debugging Guide](debug.md) - [Install & Build](install.md) +- [Contributing](/CONTRIBUTING.md) ## Project - [Adoption Census (who outside the project uses libCacheSim, with sources)](adoption.md) + +## Help +- [FAQ](/FAQ.md) +- [Issue tracker](https://github.com/1a1a11a/libCacheSim/issues) +- [Discussions](https://github.com/1a1a11a/libCacheSim/discussions) diff --git a/doc/adoption.md b/doc/adoption.md index 097a19696..83f868ebe 100644 --- a/doc/adoption.md +++ b/doc/adoption.md @@ -271,6 +271,8 @@ To cite the edition you read, pin it to a commit: open the file on GitHub and pr y, or run `git log -1 --format=%H -- doc/adoption.md` in a clone. Each edition's permalink is the commit that bumped its version in the changelog below. +Replace `` below with the permalink of the edition you read. + ```bibtex @techreport{libcachesim-adoption-census-2026, title = {libCacheSim Adoption Census}, @@ -279,7 +281,6 @@ permalink is the commit that bumped its version in the changelog below. number = {census v1.2.0}, year = {2026}, month = aug, - % replace with the permalink of the edition you read url = {https://github.com/1a1a11a/libCacheSim/blob//doc/adoption.md}, note = {Census date 2026-08-13} } diff --git a/doc/advanced_lib.md b/doc/advanced_lib.md index 335f9da29..8efa4aeaf 100644 --- a/doc/advanced_lib.md +++ b/doc/advanced_lib.md @@ -107,22 +107,36 @@ cache->to_evict(cache, req); There are mostly three APIs related to readers, `open_trace`, `close_trace`, `read_one_req`, let's take a look how they work. -##### Setup a txt reader (trace can only contain request id) +#### Setup a txt reader (trace can only contain request id) ```c open_trace(data_path, PLAIN_TXT_TRACE, NULL); ``` -##### Setup a csv reader +#### Setup a csv reader +The fields are 1-indexed and must match the trace. The sample `data/cloudPhysicsIO.csv` has the header `version,time,op,size,lbn`, so time is field 2, size is field 4, and the object id is field 5. + +`obj_id_is_num` says whether the id column holds numbers. Note that `default_reader_init_params()` sets it to **true**, unlike `cachesim`, so a trace with string ids needs it set to `false` explicitly — otherwise the reader parses them with `strtoull()` and every id becomes `0` rather than being hashed. + +Start from `default_reader_init_params()` rather than a bare designated initializer: the defaults for `cap_at_n_req`, `block_size` and `ignore_size_zero_req` are not zero, and a struct literal would silently set them to zero. `has_header` and `obj_id_is_num` are each paired with a `_set` flag; the reader auto-detects unless you raise the flag, so assigning the value alone has no effect. + ```c -reader_init_param_t init_params_csv = - {.delimiter=',', .time_field=2, .obj_id_field=6, .obj_size_field=4, .has_header=FALSE}; -reader_t *reader_csv_c = open_trace("data/trace.csv", CSV_TRACE, &init_params_csv); +reader_init_param_t init_params_csv = default_reader_init_params(); +init_params_csv.delimiter = ','; +init_params_csv.time_field = 2; +init_params_csv.obj_size_field = 4; +init_params_csv.obj_id_field = 5; +init_params_csv.obj_id_is_num = true; +init_params_csv.obj_id_is_num_set = true; +init_params_csv.has_header = true; +init_params_csv.has_header_set = true; +reader_t *reader_csv = open_trace("data/cloudPhysicsIO.csv", CSV_TRACE, &init_params_csv); ``` -##### Setup a binary reader +#### Setup a binary reader ```c -reader_init_param_t init_params_bin = {.binary_fmt="<3I2H2Q", .obj_size_field=2, .obj_id_field=6, }; -reader_t *reader_bin_l = setup_reader("data/trace.vscsi", BIN_TRACE, &init_params_bin); +reader_init_param_t init_params_bin = { + .binary_fmt_str = "cache_size // it runs cache->cache_size/step_size simulations -sim_res_t * -simulate_at_multi_sizes_with_step_size(reader_t *reader, - cache_t *cache, - uint64_t step_size, - reader_t *warmup_reader, - double warmup_perc, - int num_of_threads); +cache_stat_t *simulate_at_multi_sizes_with_step_size(reader_t *reader_in, + const cache_t *cache_in, + uint64_t step_size, + reader_t *warmup_reader, + double warmup_frac, + int warmup_sec, + int num_of_threads, + bool use_random_seed); // simulate with multiple caches, which can have different eviction algorithms or sizes cache_stat_t *simulate_with_multi_caches(reader_t *reader, @@ -163,7 +180,9 @@ cache_stat_t *simulate_with_multi_caches(reader_t *reader, reader_t *warmup_reader, double warmup_frac, int warmup_sec, - int num_of_threads) + int num_of_threads, + bool free_cache_when_finish, + bool use_random_seed); ``` `simulate_at_multi_sizes` allows you to pass in an array of `cache_sizes` to simulate; @@ -171,18 +190,25 @@ cache_stat_t *simulate_with_multi_caches(reader_t *reader, cache sizes `step_size, step_size*2, step_size*3 .. cache->cache_size`. `simulate_with_multi_caches` allows you to pass in an array of `cache_t` to simulate, which can have different eviction algorithms or sizes. -The return result is an array of simulation results, the users are responsible for free the array. +The return result is an array of simulation results, one per simulation, and the caller is responsible for freeing the array. ```c typedef struct { - uint64_t req_cnt; - uint64_t req_bytes; - uint64_t miss_cnt; - uint64_t miss_bytes; - uint64_t cache_size; - cache_stat_t cache_state; - void *other_data; /* not used */ -} sim_res_t; + int64_t n_warmup_req; + int64_t n_req; + int64_t n_req_byte; + double n_req_cost; + int64_t n_miss; + int64_t n_miss_byte; + double n_miss_cost; + + int64_t n_obj; + int64_t occupied_byte; + int64_t cache_size; + float sampler_ratio; + /* ... see libCacheSim/include/libCacheSim/simulator.h */ +} cache_stat_t; ``` +Object miss ratio is `n_miss / n_req` and byte miss ratio is `n_miss_byte / n_req_byte`. ### Trace utils @@ -206,10 +232,10 @@ int32_t *get_access_dist(reader_t *reader, ``` ## Examples -#### C example +### C example -#### C++ example +### C++ example ### Build a cache hierarchy with multiple layers @@ -219,16 +245,12 @@ int32_t *get_access_dist(reader_t *reader, ## FAQ -#### Linking with libCacheSim +### Linking with libCacheSim linking can be done in cmake or use pkg-config Such as in the `_build` directory: ``` export PKG_CONFIG_PATH=$PWD ``` -#### Possible problems +### Possible problems * if you get `error while loading shared libraries`, run `sudo ldconfig` - - - ---- diff --git a/doc/advanced_lib_extend.md b/doc/advanced_lib_extend.md index d91ac92c2..e69f371ba 100644 --- a/doc/advanced_lib_extend.md +++ b/doc/advanced_lib_extend.md @@ -35,9 +35,9 @@ Specifically, you can following the steps: 2. If your cache eviction algorithm needs extra metadata, add a new object metadata struct in [include/libCacheSim/cacheObj.h](/libCacheSim/include/libCacheSim/cacheObj.h). 3. Add `myCache_init()` function to [include/libCacheSim/evictionAlgo.h](/libCacheSim/include/libCacheSim/evictionAlgo.h). -4. Add mycache.c to [CMakeLists.txt](/libCacheSim/cache/eviction/CMakeLists.txt) so that it can be compiled. +4. Add mycache.c to [CMakeLists.txt](/libCacheSim/cache/CMakeLists.txt) so that it can be compiled. 5. Add command line option in [bin/cachesim/cache_init.h](/libCacheSim/bin/cachesim/cache_init.h) so that you can use `cachesim` binary. You may also want to take a look at [bin/cachesim/cli_parser.c](/libCacheSim/bin/cachesim/cli_parser.c). -6. Remember to add a test in [test/test_evictionAlgo.c](/test/test_evictionAlgo.c) and add the algorithm to this [README](README.md). +6. Remember to add a test in [test/test_evictionAlgo.c](/test/test_evictionAlgo.c) and add the algorithm to the [README](/README.md#supported-algorithms). > [!TIP] > Many eviction algorithms use a doubly linked list to maintain state, libCacheSim provides several functions in [cacheObj.h](/libCacheSim/include/libCacheSim/cacheObj.h) to manipulate list. @@ -66,7 +66,7 @@ There are two steps you can follow, libCacheSim supports [txt](/libCacheSim/traceReader/generalReader/txt.c), [csv](/libCacheSim/traceReader/generalReader/csv.c), and binary traces. We prefer binary traces because it allows libCacheSim to run faster, and the traces are more compact. For binary traces, libCacheSim also supports zstd compressed traces without decompression. -But if you ever need to implement a new trace type, please see [here](/libCacheSim/traceReader/customizedReader/akamaiBin.h) for an example reader. +But if you ever need to implement a new trace type, see [twrBin.h](/libCacheSim/traceReader/customizedReader/twrBin.h) for a compact example reader, or [vscsi.h](/libCacheSim/traceReader/customizedReader/vscsi.h) and [oracleGeneralBin.h](/libCacheSim/traceReader/customizedReader/oracle/oracleGeneralBin.h) for the formats used by the sample traces in [data/](/data/). To implement a reader, you need to implement two functions: ```c diff --git a/doc/conf.py b/doc/conf.py new file mode 100644 index 000000000..c0ca37e82 --- /dev/null +++ b/doc/conf.py @@ -0,0 +1,106 @@ +"""Sphinx configuration for the libCacheSim documentation. + +The docs are the Markdown files in this directory, rendered with MyST so the +same sources stay readable on GitHub. Build locally with: + + pip install -r doc/requirements.txt + sphinx-build -b html doc doc/_build/html +""" + +import os + +# -- Project information ----------------------------------------------------- + +project = "libCacheSim" +author = "Juncheng Yang" +copyright = "2024, libCacheSim authors" # noqa: A001 + +_version_file = os.path.join(os.path.dirname(__file__), os.pardir, "version.txt") +with open(_version_file, encoding="utf-8") as f: + release = f.read().strip() +version = release + +# -- General configuration --------------------------------------------------- + +extensions = ["myst_parser", "sphinxcontrib.mermaid"] + +source_suffix = {".md": "markdown", ".rst": "restructuredtext"} + +exclude_patterns = [ + "_build", + # Index for browsing the docs on GitHub; index.md is the Sphinx entry point. + "README.md", + # Image directories, not pages. They are copied verbatim via + # html_extra_path below so the tags in the guides resolve. + "plot", + "assets", +] + +# quickstart_traceAnalyzer.md embeds the plots with raw tags, so the +# files have to exist in the output. Copied rather than referenced, so the +# rendered docs do not depend on the repository being reachable. +html_extra_path = ["plot", "assets"] + +# Generate anchors for headings so cross-file "#section" links resolve. +myst_heading_anchors = 3 + +myst_enable_extensions = [ + "colon_fence", + "deflist", +] + +# Render ```mermaid fences as diagrams rather than trying to syntax-highlight +# them, which GitHub does natively. +myst_fence_as_directive = ["mermaid"] + +# -- HTML output ------------------------------------------------------------- + +html_theme = "sphinx_rtd_theme" +html_title = f"libCacheSim {release}" +html_static_path = [] + +# -- Link rewriting ---------------------------------------------------------- +# +# The Markdown sources are written to be read on GitHub, so links into the +# repository are root-absolute ("/libCacheSim/cache/eviction/LRU.c") or relative +# to the repository root ("../README.md"). Those resolve on github.com but not +# in a rendered docs site, so point them back at the repository. This runs on +# `source-read`, before MyST resolves links, otherwise MyST reports each one as +# a missing cross-reference. + +import re # noqa: E402 + +_REPO_BLOB_URL = "https://github.com/1a1a11a/libCacheSim/blob/develop" + +# Markdown inline links whose target leaves this directory. +_LINK_RE = re.compile(r"\]\((/[^)\s]*|\.\./[^)\s]*)\)") + +# The guides embed the trace-analysis plots with raw tags rather than +# Markdown, so those are not covered by _LINK_RE. html_extra_path copies the +# contents of doc/plot and doc/assets to the output root, so the site-root +# prefix has to come off for the images to resolve. +_IMG_RE = re.compile(r'(src=")/doc/(?:plot|assets)/([^"]+)"') + + +def _rewrite_target(match): + target = match.group(1) + + # Pages in this build: keep them as local cross-references so the sidebar, + # search, and PDF output link them properly. + if target.startswith("/doc/"): + return "](%s)" % target[len("/doc/") :] + + if target.startswith("../"): + target = "/" + target[len("../") :] + + return "](%s%s)" % (_REPO_BLOB_URL, target) + + +def _rewrite_repo_links(app, docname, source): + text = _IMG_RE.sub(r'\1\2"', source[0]) + source[0] = _LINK_RE.sub(_rewrite_target, text) + + +def setup(app): + app.connect("source-read", _rewrite_repo_links) + return {"parallel_read_safe": True, "parallel_write_safe": True} diff --git a/doc/index.md b/doc/index.md new file mode 100644 index 000000000..0fa5e050a --- /dev/null +++ b/doc/index.md @@ -0,0 +1,58 @@ +# libCacheSim + +A high-performance library for building and running cache simulations. + +libCacheSim ships three things: + +* **cachesim**, a high-performance cache simulator for running cache simulations. +* **traceAnalyzer**, a high-performance and versatile analyzer for cache traces. +* **libCacheSim**, a library for building your own cache simulators. + +New here? Start with [Install & Build](install.md), then [the cachesim guide](quickstart_cachesim.md). + +Commands that invoke a built binary — `./bin/cachesim`, `./bin/traceAnalyzer`, `./bin/mrcProfiler` — are run from the build directory (`_build/` if you followed the [README](https://github.com/1a1a11a/libCacheSim#build-and-install-libcachesim)), which is why the sample traces in `data/` appear as `../data/`. The helper scripts under `scripts/`, such as the plotting scripts and `debug.sh`, are run from the repository root instead, and those pages write the paths accordingly. + +```{toctree} +:maxdepth: 2 +:caption: Getting started + +install +quickstart_cachesim +quickstart_traceAnalyzer +quickstart_traceUtils +quickstart_mrcProfiler +quickstart_plugin +``` + +```{toctree} +:maxdepth: 2 +:caption: Using libCacheSim as a library + +advanced_lib +advanced_lib_extend +API +``` + +```{toctree} +:maxdepth: 2 +:caption: Performance and debugging + +performance +memory_usage_profiling +debug +``` + +```{toctree} +:maxdepth: 1 +:caption: About + +adoption +``` + +## Other resources + +* [Python binding](https://github.com/cacheMon/libCacheSim-python) — easier API access, `pip install libcachesim` +* [FAQ](https://github.com/1a1a11a/libCacheSim/blob/develop/FAQ.md) +* [Contributing](https://github.com/1a1a11a/libCacheSim/blob/develop/CONTRIBUTING.md) +* [Open-source cache datasets](https://github.com/cacheMon/cache_dataset) +* [Issue tracker](https://github.com/1a1a11a/libCacheSim/issues) and [Discussions](https://github.com/1a1a11a/libCacheSim/discussions) diff --git a/doc/install.md b/doc/install.md index ec91f79b5..f0270c4e6 100644 --- a/doc/install.md +++ b/doc/install.md @@ -1,19 +1,19 @@ -## Install dependency +# Install dependency libCacheSim uses [cmake](https://cmake.org/) build system with [Ninja](https://ninja-build.org/) generator and has a few dependencies: [glib](https://developer.gnome.org/glib/) [tcmalloc](https://github.com/google/tcmalloc), [zstd](https://github.com/facebook/zstd). -### Install dependency on Ubuntu +## Install dependency on Ubuntu -#### Install glib, tcmalloc, cmake and ninja +### Install glib, tcmalloc, cmake and ninja ```bash sudo apt install libglib2.0-dev libgoogle-perftools-dev cmake ninja-build ``` -#### Install zstd +### Install zstd zstd must be installed from source @@ -27,7 +27,7 @@ sudo ninja install popd ``` -#### Install XGBoost [Optional] +### Install XGBoost [Optional] ```bash git clone --recursive https://github.com/dmlc/xgboost @@ -38,7 +38,7 @@ sudo ninja install popd ``` -#### Install LightGBM [Optional] +### Install LightGBM [Optional] ```bash git clone --recursive https://github.com/microsoft/LightGBM @@ -49,7 +49,7 @@ sudo ninja install popd ``` -### Install dependency on Mac +## Install dependency on Mac using [homebrew](https://brew.sh/) as an example. While the first line is necessary, the following two lines needs to be run if you encounter errors including: @@ -62,7 +62,7 @@ brew install argp-standalone brew install pkg-config ``` -#### Install zstd +### Install zstd Use the below command to install ```bash brew install zstd diff --git a/doc/memory_usage_profiling.md b/doc/memory_usage_profiling.md index 024e5c0ef..695fa22d3 100644 --- a/doc/memory_usage_profiling.md +++ b/doc/memory_usage_profiling.md @@ -36,7 +36,7 @@ ms_print ./massif.out > massif.result The generated report primarily includes a bar chart of memory usage (with instructions executed as the x-axis) and several heap profile snapshots. Some snapshots display detailed function call relationships showing how memory was allocated. Below is an example of such a report: -```sh +```text MB 519.4^ : |#:::::::::::::@::::::::::::::::@::@@::::::::::::::::::::::::@::@::@::@:: diff --git a/doc/performance.md b/doc/performance.md index 9ae42cc40..d38983487 100644 --- a/doc/performance.md +++ b/doc/performance.md @@ -1,15 +1,66 @@ +# Performance tuning +libCacheSim is built for high-throughput trace replay. This page collects the knobs that matter and how to measure their effect on your own workload — the numbers depend heavily on the trace, the algorithm, and the machine, so measure rather than assume. -## Performance +## Measuring throughput +`cachesim` reports throughput (in millions of requests per second) on every run: +```bash +cd _build +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lru 1gb +``` -## Memory efficiency +For a systematic comparison across algorithms and working set sizes, [`scripts/benchmark_throughput.py`](/scripts/benchmark_throughput.py) generates Zipfian traces and sweeps them: +```bash +cd scripts +python3 benchmark_throughput.py --help +``` +The sample traces in [`data/`](/data/) are far too small to benchmark with — they fit in cache and are dominated by startup cost. Use a trace of at least a few million requests. -### Other -#### Performance Optimizations -* hugepage - to turn on hugepage support, please do `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled` +## Build configuration +Build in release mode. A debug build is several times slower, and [`scripts/debug.sh`](/scripts/debug.sh) additionally skips tcmalloc to stay debugger-friendly: +```bash +cmake -G Ninja -B _build -DCMAKE_BUILD_TYPE=Release +``` + +**tcmalloc** is linked automatically when CMake finds it, and matters because the hot path allocates per-object metadata. `cmake` prints `!!! cannot find tcmalloc` when it is missing; install it (`libgoogle-perftools-dev` on Debian/Ubuntu, `gperftools` via Homebrew) and reconfigure. + +**Transparent hugepages** are enabled at compile time by default (`USE_HUGEPAGE=ON`), which reduces TLB misses on the hash table. They also need to be enabled on the host: + +```bash +echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled +``` + +Turn the compile-time option off with `-DUSE_HUGEPAGE=OFF` if your environment does not support them. + +## Trace format + +Binary formats are several times faster than csv, because csv parsing dominates replay for fast algorithms. Convert once with `traceConv` and reuse: + +```bash +./bin/traceConv ../data/cloudPhysicsIO.csv csv \ + -t "time-col=2,obj-id-col=5,obj-size-col=4,obj-id-is-num=1" \ + --output-format=oracleGeneral +``` + +See [quickstart_traceUtils.md](quickstart_traceUtils.md). zstd-compressed binary traces are read without decompressing first, so compression costs little replay time while saving substantial disk. + +When the id column holds numbers, pass `obj-id-is-num=true` so the reader skips hashing. + +## Runtime options + +* `--num-thread=N` — simulations across algorithms and cache sizes are run in parallel, so a sweep costs little more than its slowest single run. +* `--ignore-obj-size 1` — treats every object as size one. Faster, and the right choice when you want object miss ratio rather than byte miss ratio. +* `--consider-obj-metadata=false` — skips accounting for per-algorithm metadata overhead in the cache size. +* `--num-req=N` — caps how much of the trace is read, useful when iterating. + +## Memory + +Memory is dominated by the hash table and per-object metadata, so it scales with the number of *objects* rather than the number of requests. `--ignore-obj-size 1` with a small cache size keeps the object count down. + +To profile actual usage, see [memory_usage_profiling.md](memory_usage_profiling.md). diff --git a/doc/quickstart_cachesim.md b/doc/quickstart_cachesim.md index 5fee2a253..0fb6735b6 100644 --- a/doc/quickstart_cachesim.md +++ b/doc/quickstart_cachesim.md @@ -11,16 +11,18 @@ Meanwhile, cachesim has high-performance with low resource usages. --- ## Installation -First, [build libCacheSim](/doc/install.md). After building libCacheSim, `cachesim` should be in the build directory. +First, [build libCacheSim](/doc/install.md). After building libCacheSim, `cachesim` is in the `bin/` subdirectory of your build directory. + +All commands on this page are run from the build directory (`_build/` if you followed the [README](/README.md)), so the sample traces in [data/](/data/) are at `../data/`. --- ## Basic Usage ``` -./cachesim trace_path trace_type eviction_algo cache_size [OPTION...] +./bin/cachesim trace_path trace_type eviction_algo cache_size [OPTION...] ``` -use `./cachesim --help` to get more information. +use `./bin/cachesim --help` to get more information. ### Run a single cache simulation @@ -29,28 +31,28 @@ Note that vscsi is a trace format, we also support csv traces. ```bash # Note that no space between the cache size and the unit, unit is not case sensitive -./cachesim ../data/trace.vscsi vscsi lru 1gb +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lru 1gb ``` ### Run multiple cache simulations ```bash # Note that there is no space between the cache sizes -./cachesim ../data/trace.vscsi vscsi lru 1mb,16mb,256mb,8gb +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lru 1mb,16mb,256mb,8gb # Or you can quote the cache sizes -./cachesim ../data/trace.vscsi vscsi lru "1mb, 16mb, 256mb, 8gb" +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lru "1mb, 16mb, 256mb, 8gb" # besides absolute cache size, you can also use fraction of working set size -./cachesim ../data/trace.vscsi vscsi lru 0.001,0.01,0.1,0.2 +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lru 0.001,0.01,0.1,0.2 # besides using byte as the unit, you can also treat all objects having the same size, and the size is the number of objects -./cachesim ../data/trace.vscsi vscsi lru 1000,16000 --ignore-obj-size 1 +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lru 1000,16000 --ignore-obj-size 1 # new feature: you can run a few algorithms in parallel by concatenating the algorithms -./cachesim ../data/trace.vscsi vscsi fifo,lru,arc,qdlp 0.01 --ignore-obj-size 1 +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi fifo,lru,arc,qdlp 0.01 --ignore-obj-size 1 # run 4*4 simulations in parallel (no more than n_thread at the same time) -./cachesim ../data/trace.vscsi vscsi fifo,lru,arc,qdlp 0.01,0.05,0.1,0.2 --ignore-obj-size 1 +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi fifo,lru,arc,qdlp 0.01,0.05,0.1,0.2 --ignore-obj-size 1 ``` @@ -59,7 +61,7 @@ cachesim can detect the working set of the trace and automatically generate cach You can enable this feature by setting cache size to 0 or auto. ```bash -./cachesim ../data/trace.vscsi vscsi lru auto +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lru auto ``` ### Use different eviction algorithms @@ -70,27 +72,30 @@ cachesim supports the following algorithms: * [LFU](/libCacheSim/cache/eviction/LFU.c) * [ARC](/libCacheSim/cache/eviction/ARC.c) * [SLRU](/libCacheSim/cache/eviction/SLRU.c) -* [GDSF](/libCacheSim/cache/eviction/GDSF.c) +* [GDSF](/libCacheSim/cache/eviction/cpp/GDSF.cpp) * [WTinyLFU](/libCacheSim/cache/eviction/WTinyLFU.c) * [LeCaR](/libCacheSim/cache/eviction/LeCaR.c) * [Cacheus](/libCacheSim/cache/eviction/Cacheus.c) * [Hyperbolic](/libCacheSim/cache/eviction/Hyperbolic.c) -* [LHD](/libCacheSim/cache/eviction/LHD/LHDInterface.cpp) -* [GLCache](/libCacheSim/cache/eviction/GLCache/GLCache.c) +* [LHD](/libCacheSim/cache/eviction/LHD/LHD_Interface.cpp) * [Belady](/libCacheSim/cache/eviction/Belady.c) * [BeladySize](/libCacheSim/cache/eviction/BeladySize.c) * [QD-LP](/libCacheSim/cache/eviction/QDLP.c) +* [S3-FIFO](/libCacheSim/cache/eviction/S3FIFO.c), [Sieve](/libCacheSim/cache/eviction/Sieve.c) +* [GLCache](/libCacheSim/cache/eviction/GLCache/GLCache.c) — build with `-DENABLE_GLCACHE=ON` + +See the [README](/README.md#supported-algorithms) for the full list, including the algorithms that are behind an optional build flag (GLCache, LRB, 3LCache). Asking for one that was not compiled in fails with `do not support algorithm `. You can just use the algorithm name as the eviction algorithm parameter, for example ```bash -./cachesim ../data/trace.vscsi vscsi lecar auto -./cachesim ../data/trace.vscsi vscsi hyperbolic auto -./cachesim ../data/trace.vscsi vscsi lhd auto -./cachesim ../data/trace.vscsi vscsi glcache auto +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lecar auto +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi hyperbolic auto +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi lhd auto +./bin/cachesim ../data/cloudPhysicsIO.vscsi vscsi s3fifo auto # belady and beladySize require oracle trace -./cachesim ../data/trace.oracleGeneral oracleGeneral beladySize auto +./bin/cachesim ../data/cloudPhysicsIO.oracleGeneral.bin oracleGeneral beladySize auto ``` @@ -103,28 +108,29 @@ Besides the column information, a csv reader also requires the delimiter and whe cachesim builds in a simple delimiter and header detector, if the detected result is not correct, you can provide the correct information using `delimiter=,`, `has-header=true`. +Object ids are hashed unless you tell the reader they are already numeric, so add `obj-id-is-num=true` when the id column holds numbers — `cachesim` stops with an error if you leave it out on such a trace. The sample `cloudPhysicsIO.csv` has a numeric id column, so every example below sets it. + ```bash # note that the parameters are separated by comma and quoted -./cachesim ../data/trace.csv csv lru 1gb -t "time-col=2, obj-id-col=5, obj-size-col=4" - -# if object id is numeric, then we can pass obj-id-is-num=true to speed up -./cachesim ../data/trace.csv csv lru 1gb -t "time-col=2, obj-id-col=5, obj-size-col=4, obj-id-is-num=true" +./bin/cachesim ../data/cloudPhysicsIO.csv csv lru 1gb -t "time-col=2, obj-id-col=5, obj-size-col=4, obj-id-is-num=true" +# omitting obj-id-is-num on a numeric id column fails with +# [ERROR] csv.c: detect obj_id is numeric, please specify -t 'obj-id-is-num=1' # note that csv trace does not support UTF-8 encoding, only ASCII encoding is supported -./cachesim ../data/trace.csv csv lru 1gb -t "time-col=2, obj-id-col=5, obj-size-col=4, delimiter=,, has-header=true" +./bin/cachesim ../data/cloudPhysicsIO.csv csv lru 1gb -t "time-col=2, obj-id-col=5, obj-size-col=4, obj-id-is-num=true, delimiter=,, has-header=true" ``` Besides csv trace, we also support txt trace and binary trace. ```bash # txt trace is a simple format that stores obj-id in each line -./cachesim ../data/trace.txt txt lru 1gb +./bin/cachesim ../data/cloudPhysicsIO.txt txt lru 1gb # binary trace, format is specified using format string similar to Python struct -./cachesim ../data/trace.vscsi binary lru 1gb -t "format= [!NOTE] +> Sampling-based algorithms such as `RandomLRU` and `hyperbolic` draw eviction candidates from the hash table, so their miss ratios shift slightly with `--hashpower`. Keep it fixed when comparing results. diff --git a/doc/quickstart_traceAnalyzer.md b/doc/quickstart_traceAnalyzer.md index 2953c954b..cdcf57d3b 100644 --- a/doc/quickstart_traceAnalyzer.md +++ b/doc/quickstart_traceAnalyzer.md @@ -1,17 +1,17 @@ -## Trace analysis tool +# Trace analysis tool libCacheSim provides a set of tools to help you analyze traces. After building the project, you can find a binary called `traceAnalyzer`. This doc shows how to use the tool. If you are interested, the source code is located in the [bin/traceAnalyzer/](/libCacheSim/bin/traceAnalyzer) and [traceAnalyzer](/libCacheSim/traceAnalyzer) directory. -### Obtain trace statistics -#### Usage: +## Obtain trace statistics +### Usage: ``` # ./bin/traceAnalyzer --help for a list of tasks and options ./bin/traceAnalyzer PATH_TO_TRACE traceType [--task1] [--task2] ``` -#### A list of tasks: +### A list of tasks: * `--common`: run all common tasks, including `--stat`, `--traceStat`, `--reqRate`, `--size`, `--reuse`, `--popularity` * `--all`: run all tasks * `--accessPattern`: generate access pattern data for plotting using [scripts/traceAnalysis/access_pattern.py](/scripts/traceAnalysis/access_pattern.py) @@ -21,7 +21,7 @@ If you are interested, the source code is located in the [bin/traceAnalyzer/](/l * `--popularity`: generate popularity data for plotting using [scripts/traceAnalysis/popularity.py](/scripts/traceAnalysis/popularity.py) * `--popularityDecay`: generate popularity data for plotting using [scripts/traceAnalysis/popularity_decay.py](/scripts/traceAnalysis/popularity_decay.py) -#### Example: +### Example: ```bash # run all common tasks ./bin/traceAnalyzer PATH_TO_TRACE traceType --common @@ -69,11 +69,11 @@ The trace analyzer will generate statistics of the trace and save them to `stat` ---- -### Plot trace statistics and visualize the trace +## Plot trace statistics and visualize the trace We provide plot scripts in [scripts/traceAnalysis/](/scripts/traceAnalysis/) to help you plot the trace statistics. After generating plot data, we can plot access pattern, request rate, size, reuse, and popularity using the following commands: -#### Access pattern +### Access pattern ```bash # plot the access pattern using wall clock (real) time python3 scripts/traceAnalysis/access_pattern.py ${dataname}.accessRtime @@ -105,7 +105,7 @@ The first 10m requests of the Twitter cluster52 trace, this is a Zipf workload. -#### Request rate +### Request rate ```bash # this is only supported for traces that have (wall clock) time field python3 scripts/traceAnalysis/req_rate.py ${dataname}.reqRate_w300 @@ -124,7 +124,7 @@ The block workload has a daily request spike, while the Twitter workload is too
-#### Size distribution +### Size distribution ```bash # this is only supported for traces that have object size python3 scripts/traceAnalysis/size.py ${dataname}.size @@ -144,7 +144,7 @@ The Request curve is weighted by request count, and the Object curve is weighted
-#### Reuse distribution +### Reuse distribution This is the time since the last access of the object. ```bash @@ -173,7 +173,7 @@ The first 10m requests of the Twitter cluster52 trace. The left column shows wal
-#### Popularity +### Popularity ```bash # the popularity skewness ($\alpha$) is in the output of traceAnalyzer # this plots the request count/freq over object rank @@ -195,7 +195,7 @@ The first 10m requests of the Twitter cluster52 trace.
-#### Size distribution heatmap +### Size distribution heatmap This and the following plots are more expensive plots that require more CPU cycles and DRAM usage to generate. This plot requires wall clock time and object size in the trace. This is a heatmap of the size distribution of the trace. The x-axis is the clock time, and the y-axis is the size. The color represents the number of requests having a certain size range at that time. The darker the color, the more requests of the certain size at that time. @@ -217,7 +217,7 @@ Left: a block cache workload (w92), right: the first 10m requests of the Twitter
-#### Reuse distribution heatmap +### Reuse distribution heatmap This is a heatmap of the reuse distribution of the trace. The x-axis is the wall clock time, and the y-axis is the reuse time (in seconds) or reuse distance (the number of requests since last access of the object). The color represents the number of requests having the reuse time or reuse distance. The heatmap is generated using the following command: @@ -237,7 +237,7 @@ Left: a block cache workload (w92), right: the first 10m requests of the Twitter
-#### popularity decay +### popularity decay There are two versions of the plots, one is line plot, and the other is a heatmap. ```bash @@ -246,25 +246,24 @@ There are two versions of the plots, one is line plot, and the other is a heatma python3 scripts/traceAnalysis/popularity_decay.py ${dataname}.popularityDecay_w300_obj ``` - +
-### Advanced features +## Advanced features ```bash # cap the number of requests read from the trace -./traceAnalyzer --num-req=1000000 ../data/trace.vscsi vscsi +./bin/traceAnalyzer --num-req=1000000 ../data/cloudPhysicsIO.vscsi vscsi # change output -./traceAnalyzer -o my-output ../data/trace.vscsi vscsi +./bin/traceAnalyzer -o my-output ../data/cloudPhysicsIO.vscsi vscsi # use part of the trace to warm up the cache -./traceAnalyzer --warmup-sec=86400 ../data/trace.vscsi vscsi +./bin/traceAnalyzer --warmup-sec=86400 ../data/cloudPhysicsIO.vscsi vscsi ``` diff --git a/doc/quickstart_traceUtils.md b/doc/quickstart_traceUtils.md index 47b59e5ea..cd31fba04 100644 --- a/doc/quickstart_traceUtils.md +++ b/doc/quickstart_traceUtils.md @@ -1,7 +1,7 @@ -## Other trace utilities +# Other trace utilities We also provide some trace utilities to help you use the traces and debug applications. -### tracePrint +## tracePrint Print requests from a trace. ```bash @@ -9,7 +9,7 @@ Print requests from a trace. ./bin/tracePrint ../data/cloudPhysicsIO.vscsi vscsi -n 10 ``` -### traceConv +## traceConv Convert a trace to oracleGeneral format so you can run it faster (10x speedup) using less memory. Meanwhile, the generated trace has a smaller size, contains next request time. ```bash # the first parameter is the input trace, the second parameter is trace type, the output is in the same directory with suffic oracleGeneral @@ -28,11 +28,11 @@ We can also sample a trace to reduce its size. ./bin/traceConv ../data/cloudPhysicsIO.vscsi vscsi -s 0.01 --output-format=oracleGeneral ``` -### traceFilter +## traceFilter traceFilter simulates a multi-layer cache hierarchy. It filters the trace based on the cache hit/miss information and generates a trace for the second layer. The generated trace is in oracleGeneral format. ```bash # filter trace using a cache with a size 0.01 of the working set size and the FIFO eviction policy -./bin/traceFilter ../data/trace.vscsi vscsi --filter-type fifo --filter-size 0.01 --ignore-obj-size 1 +./bin/traceFilter ../data/cloudPhysicsIO.vscsi vscsi --filter-type fifo --filter-size 0.01 --ignore-obj-size 1 ``` diff --git a/doc/requirements.txt b/doc/requirements.txt new file mode 100644 index 000000000..025754ded --- /dev/null +++ b/doc/requirements.txt @@ -0,0 +1,6 @@ +# Documentation build only. The plotting/analysis scripts use the root +# requirements.txt instead. +sphinx>=7.0 +myst-parser>=2.0 +sphinx-rtd-theme>=2.0 +sphinxcontrib-mermaid>=0.9