[disk] Replace custom cached I/O with standard buffered readers and writers

[Wei Wu (wuw92)]

• Does this PR have a descriptive title that could go in our release notes?
• Does this PR add any new dependencies?
• Does this PR modify any existing APIs?
• Is the change to the API backwards compatible?
• Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

N/A.

What does this implement/fix? Briefly explain your changes.

Replace CachedReader and CachedWriter with std::io::BufReader and BufWriter in disk index construction, partitioning, and graph merging, removing the hand-maintained buffering implementation without adding dependencies.

• Preserve storage-provider abstraction and existing buffer capacities. A shared open_buf_reader helper caps reader capacity at file length, with a focused test for that policy. Buffer-size values use usize.
• Use read_exact and write_all for complete transfers. Reuse the disk-layout writer to rewind and fill its header instead of reopening the file; explicitly flush completed production outputs to propagate write errors.

Any other comments?

This removes the publicly re-exported CachedReader/CachedWriter types and changes READ_WRITE_BLOCK_SIZE from u64 to usize, so downstream source compatibility is not preserved. Truncated reads now follow standard I/O error handling rather than the custom cached-reader EOF diagnostic; initial reads are lazy rather than performed during construction.

[Copilot]

🔵 Needs a closer look

The partition test must flush its BufWriter before the dataset is read.

Pull request overview

This PR replaces custom cached I/O with standard buffered readers and writers across disk index construction, partitioning, and graph merging.

Changes:

• Adds bounded BufReader creation and updates buffer sizes to usize.
• Uses read_exact, write_all, seeking, and explicit flushing.
• Removes the obsolete CachedReader and CachedWriter implementations.

File summaries

File	Description
diskann-providers/src/utils/sampling.rs	Updates buffer-capacity usage.
diskann-providers/src/utils/mod.rs	Changes the shared block-size constant to usize.
diskann-disk/src/utils/partition.rs	Migrates partition I/O to standard buffering.
diskann-disk/src/storage/mod.rs	Adds a bounded buffered-reader helper and test.
diskann-disk/src/storage/disk_index_writer.rs	Migrates disk-layout I/O and header rewriting.
diskann-disk/src/storage/cached_writer.rs	Removes the obsolete cached writer.
diskann-disk/src/storage/cached_reader.rs	Removes the obsolete cached reader.
diskann-disk/src/build/builder/core.rs	Migrates shard and graph-merge I/O.

Review details

Suppressed comments (1)

diskann-disk/src/utils/partition.rs:480

• This test now wraps the dataset writer in BufWriter, but it drops the writer at the end of the scope without flushing it. BufWriter discards buffered bytes on drop, so the dataset is empty and the subsequent call to shard_data_into_clusters_only_ids fails while reading its header; flush the writer after the nested loops.

                    dataset_writer.write_all(&val.to_le_bytes()).unwrap();

• Files reviewed: 8/8 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 98.98990% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.54%. Comparing base (cc473fc) to head (ec63f38).

Files with missing lines	Patch %	Lines
diskann-disk/src/storage/disk_index_writer.rs	96.42%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1396      +/-   ##
==========================================
- Coverage   92.65%   91.54%   -1.12%     
==========================================
  Files         527      526       -1     
  Lines      103254   103057     -197     
==========================================
- Hits        95674    94339    -1335     
- Misses       7580     8718    +1138     

Flag	Coverage Δ
miri	91.54% <98.98%> (-1.12%)	⬇️
unittests	91.28% <98.98%> (-1.32%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-disk/src/build/builder/core.rs	97.56% <100.00%> (+0.27%)	⬆️
diskann-disk/src/storage/mod.rs	100.00% <100.00%> (ø)
diskann-disk/src/utils/partition.rs	93.92% <100.00%> (+0.20%)	⬆️
diskann-providers/src/utils/sampling.rs	97.98% <ø> (-0.02%)	⬇️
diskann-disk/src/storage/disk_index_writer.rs	96.47% <96.42%> (+0.44%)	⬆️

... and 48 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.