feat(bench): run the random access benchmark against S3 - #9412
Draft
joseph-isaacs wants to merge 4 commits into
Draft
feat(bench): run the random access benchmark against S3#9412joseph-isaacs wants to merge 4 commits into
joseph-isaacs wants to merge 4 commits into
Conversation
Adds an S3 variant of the random-access benchmark. It is the same benchmark and the same code paths, only the data is read from an object store instead of local NVMe: - `--remote-data-dir s3://bucket/prefix/` opens the Vortex, Parquet and Lance files from S3, mirroring the local data directory layout; - `--prepare-data` materializes the local files (and prints their paths) so CI can upload them before the run; - remote measurements are suffixed `-tokio-s3` and reported with `s3` storage so they form a separate series from the local-disk numbers. CI runs it from a new `pr-bench-random-access-s3.yml` workflow (label `action/bench-random-access-s3`, also covered by `action/bench-all`) and from a new `Random Access (S3)` entry in the develop benchmark matrix. The shared PR benchmark runner gained `remote_data_dir` and `variant_id` inputs; it uploads the data to a per-run S3 prefix, runs the benchmark against it, and deletes the prefix afterwards. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will not alter performance
|
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Random Access (S3) 📖Vortex (geomean): no vortex data No baseline is available for this benchmark yet; PR measurements are shown without comparison. random-access / vortex-file-compressed / ns (no group data, 0↑ 0↓)
random-access / parquet / ns (no group data, 0↑ 0↓)
random-access / lance / ns (no group data, 0↑ 0↓)
|
`lance` is pinned with `default-features = false`, which drops the `aws`
feature it enables by default. Without it Lance has no `s3://` object store
provider registered and opening a remote dataset fails with:
Invalid user input: No object store provider found for scheme: 's3'
Parquet and Vortex already read from S3 fine; only Lance was affected.
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
force-pushed
the
claude/random-access-s3-benchmark-f9zi6a
branch
from
August 14, 2026 17:29
0fdd447 to
552c49a
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… files Every synthetic random-access dataset holds 1,000,000 rows and was written with default writer properties, whose `max_row_group_size` is 1Mi rows. A million rows never crosses that threshold, so each file is a single row group. Readers select row groups before rows, so a point lookup fetches and decodes the entire file: masked by page cache locally, ruinous over an object store, where one `take` on feature-vectors spends ~87s moving ~4GB. The tell is in the measurements: nested-lists and nested-structs take the same time under both access patterns to within 0.04%, because the indices never change what is read. Size row groups from the row width instead, targeting 128MiB and clamping to 8-64 batches of 1024 rows. The clamp is what fixes narrow rows: a pure byte budget would still leave nested-lists in one group. Sizes stay whole multiples of the 1024-row Arrow batches that `parquet_to_vortex_chunks` streams, so row group boundaries never split a batch and the derived Vortex files are unchanged -- only Parquet layout moves. Data pages also drop from 20k to 1024 rows, giving the page index resolution worth having here. Scan-oriented generators (TPC-H, SpatialBench, PolarSignals, ...) keep large row groups, which is right for full scans. This shifts Parquet random-access baselines once, most visibly on the correlated pattern. Uniform lookups still touch most row groups; only page-level row selection in the reader addresses those. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
`set_max_row_group_size` is deprecated in favour of `set_max_row_group_row_count`, which takes an `Option<usize>` where `None` means unlimited. The deprecation warning fails the lint job. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add random access object store benchmarks