Run partial Flat random access with batched io_uring

[joseph-isaacs]

Summary

• add partial Flat-layout random access for fixed-width and ALPRD feature-vector data
• submit complete small-range batches through io_uring before waiting for completions
• bound batched ring admission to 512 requests / 16 MiB and retain blocking fallback
• configure the action/bench-random-access workflow to use the measured batched-ring settings
• add reconstruction and ring queue-depth regression coverage

CI behavior

Applying action/bench-random-access runs with:

VORTEX_IO_URING=1
VORTEX_IO_URING_RINGS=4
VORTEX_IO_URING_QUEUE_DEPTH=128
VORTEX_IO_URING_MIN_READ_SIZE=0
VORTEX_IO_URING_MAX_IN_FLIGHT=512

Measured results

On 8 pinned cores:

• cold feature-vectors/uniform: 9.59 ms Vortex batched ring versus 17.01 ms Lance
• cold feature-vectors/correlated: 3.88 ms Vortex versus 6.76 ms Lance
• hot feature-vectors/uniform: 2.26 ms Vortex versus 1.83 ms Lance
• hot feature-vectors/correlated: 0.34 ms Vortex versus 0.54 ms Lance

The ring path strongly improves cold random access but has additional synchronization cost for page-cache-resident uniform reads. The separate blocking-control PR allows CI comparison under the same benchmark label.

Validation

• cargo test -p vortex-array
• cargo test -p vortex-layout
• cargo test -p vortex-io --lib (109 tests)
• batched request regression with a batch larger than ring depth
• cargo clippy -p vortex-io --lib --all-features -- -D warnings
• cargo test --doc -p vortex-io
• workflow YAML parse
• hot and targeted-cold 8-core Vortex/Lance benchmarks

[codspeed-hq]

Merging this PR will degrade performance by 9.61%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 2 improved benchmarks
❌ 7 regressed benchmarks
✅ 1991 untouched benchmarks
🆕 8 new benchmarks
⏩ 89 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	sparse_is_constant	559.8 µs	891.2 µs	-37.18%
❌	Simulation	warm[(64, 256)]	10.8 ms	12.3 ms	-11.64%
❌	Simulation	compress_fsst[(1000, 64, 8)]	1 ms	1.2 ms	-11.54%
❌	Simulation	compact_sliced[(4096, 90)]	1.6 µs	1.8 µs	-10.89%
❌	Simulation	compress_fsst[(10000, 64, 4)]	8.6 ms	9.7 ms	-10.54%
❌	Simulation	compact[(4096, 90)]	1.8 µs	2 µs	-10.16%
❌	Simulation	compact_sliced[(16384, 90)]	1.8 µs	2 µs	-10.12%
⚡	Simulation	nullable_polygons_x_nullable_points	664.9 µs	587.2 µs	+13.24%
⚡	Simulation	nullable_polygons_90pct_x_constant_point	548.3 µs	487.3 µs	+12.52%
🆕	Simulation	arrow_export_100_predecoded_arrays	N/A	1.5 ms	N/A
🆕	Simulation	concat_100_canonical_one_row_arrays	N/A	131.6 µs	N/A
🆕	Simulation	container_concat_100_alprd_one_row_arrays	N/A	134.5 µs	N/A
🆕	Simulation	extract_100_prebuilt_arrays_to_arrow	N/A	5.6 ms	N/A
🆕	Simulation	rebuild_100_flat_arrays_from_resident_buffers	N/A	4.1 ms	N/A
🆕	Simulation	rebuild_100_flat_arrays_with_eager_leaf_decode	N/A	8.9 ms	N/A
🆕	Simulation	rebuild_and_extract_100_arrays_to_arrow	N/A	9.8 ms	N/A
🆕	Simulation	rebuild_with_eager_extract_100_arrays_to_arrow	N/A	10.2 ms	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/partial-flat-random-access-batched-ring (6fe5a3a) with develop (b363fb7)}

Footnotes

1. 89 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	6fe5a3a	random-access-bench	1	Explore Profiling Data

Previous Runs (2)

Status	Commit	Job	Attempt	Link
🟢 Done	a0cb6d7	random-access-bench	1	Explore Profiling Data
🟢 Done	91d9a93	random-access-bench	1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[github-actions]

Benchmarks: Random Access 📖

Verdict: Likely regression (medium confidence)
Attributed Vortex impact: +15.1%
Engines: random-access Likely regression (+15.1%, medium confidence)
Vortex (geomean): 1.152x ❌
Parquet (geomean): 1.001x ➖
Shifts: Parquet (control) +0.1% · Median polish +3.7%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

random-access / vortex-file-compressed / ns (1.152x ❌, 1↑ 9↓)

name	PR 6fe5a3a (ns)	base b363fb7 (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/vortex-tokio-local-disk	309161	325491	0.95
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer	593671	616771	0.96
random-access/feature-vectors/uniform/vortex-tokio-local-disk 🚀	1962159	2563711	0.77
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer	3004946	3013585	1.00
random-access/nested-lists/correlated/vortex-tokio-local-disk	301032	293916	1.02
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer 🚨	388035	340181	1.14
random-access/nested-lists/uniform/vortex-tokio-local-disk 🚨	2458950	1473034	1.67
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer 🚨	2758519	1588092	1.74
random-access/nested-structs/correlated/vortex-tokio-local-disk 🚨	451671	325318	1.39
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer 🚨	618491	450157	1.37
random-access/nested-structs/uniform/vortex-tokio-local-disk 🚨	1238959	960723	1.29
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer 🚨	1606803	1216015	1.32
random-access/taxi/correlated/vortex-tokio-local-disk	861125	888603	0.97
random-access/taxi/correlated/vortex-tokio-local-disk-footer	1244940	1333316	0.93
random-access/taxi/uniform/vortex-tokio-local-disk 🚨	5235227	3967200	1.32
random-access/taxi/uniform/vortex-tokio-local-disk-footer 🚨	7448191	4684375	1.59
random-access/vortex-tokio-local-disk	489174	537780	0.91
random-access/vortex-tokio-local-disk-footer	866519	904694	0.96

random-access / parquet / ns (1.001x ➖, 0↑ 0↓)

name	PR 6fe5a3a (ns)	base b363fb7 (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/parquet-tokio-local-disk	8844326071	8914077594	0.99
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer	8773220989	8781373858	1.00
random-access/feature-vectors/uniform/parquet-tokio-local-disk	8868645082	8796571130	1.01
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer	8915089689	8908534745	1.00
random-access/nested-lists/correlated/parquet-tokio-local-disk	138489797	137652053	1.01
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer	138072883	140569020	0.98
random-access/nested-lists/uniform/parquet-tokio-local-disk	140057676	139050533	1.01
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer	138029853	137892135	1.00
random-access/nested-structs/correlated/parquet-tokio-local-disk	20450925	20597244	0.99
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer	20630833	20833053	0.99
random-access/nested-structs/uniform/parquet-tokio-local-disk	20402751	20334362	1.00
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer	20856796	20609364	1.01
random-access/parquet-tokio-local-disk	183144904	182684728	1.00
random-access/parquet-tokio-local-disk-footer	183252045	181313582	1.01
random-access/taxi/correlated/parquet-tokio-local-disk	275035386	274106803	1.00
random-access/taxi/correlated/parquet-tokio-local-disk-footer	274502891	271756511	1.01
random-access/taxi/uniform/parquet-tokio-local-disk	289140416	290760652	0.99
random-access/taxi/uniform/parquet-tokio-local-disk-footer	290410080	290813261	1.00

random-access / lance / ns (0.995x ➖, 0↑ 0↓)

name	PR 6fe5a3a (ns)	base b363fb7 (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/lance-tokio-local-disk	354235	355553	1.00
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer	1093179	1117265	0.98
random-access/feature-vectors/uniform/lance-tokio-local-disk	1111365	1103367	1.01
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer	1819458	1903521	0.96
random-access/lance-tokio-local-disk	617780	641508	0.96
random-access/lance-tokio-local-disk-footer	1345343	1346907	1.00
random-access/nested-lists/correlated/lance-tokio-local-disk	185781	185918	1.00
random-access/nested-lists/correlated/lance-tokio-local-disk-footer	598399	576459	1.04
random-access/nested-lists/uniform/lance-tokio-local-disk	903151	903831	1.00
random-access/nested-lists/uniform/lance-tokio-local-disk-footer	1343082	1330825	1.01
random-access/nested-structs/correlated/lance-tokio-local-disk	312470	308832	1.01
random-access/nested-structs/correlated/lance-tokio-local-disk-footer	457303	455628	1.00
random-access/nested-structs/uniform/lance-tokio-local-disk	2446378	2468432	0.99
random-access/nested-structs/uniform/lance-tokio-local-disk-footer	2562082	2600968	0.99
random-access/taxi/correlated/lance-tokio-local-disk	963999	963154	1.00
random-access/taxi/correlated/lance-tokio-local-disk-footer	1879862	1879591	1.00
random-access/taxi/uniform/lance-tokio-local-disk	10045455	10042761	1.00
random-access/taxi/uniform/lance-tokio-local-disk-footer	10547006	10776913	0.98