[python] Add paired ACT storage benchmark - #9466
Conversation
622ceae to
5d8890e
Compare
fb77b0e to
7b2b5a4
Compare
42be5b2 to
705683a
Compare
705683a to
9e0340a
Compare
|
Thanks for putting this together. This PR currently combines two independently valuable changes with different review and risk profiles: the generic ContiguousWindowDataset public API, and the RoboMIND ACT benchmark harness and workflow. Given the 4.4k-line scope, could we split this into two PRs?
This would make the public API easier to review on its own, while allowing the benchmark methodology and reproducibility contract to be evaluated independently. It would also keep follow-up fixes in either area from blocking the other. |
Rebuild the benchmark-only change on top of the merged contiguous-window dataset. Compare equivalent HDF5 and Paimon ACT inputs and use frame_index consistently in the public sample contract. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 2395/2395 AI-Contributed/UT: 895/895
9e0340a to
78207fc
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
Three actionable issues found in the updated benchmark-only scope.
| stored_mean = np.asarray(row["action_mean"], dtype=np.float64) | ||
| stored_std = np.asarray(row["action_std"], dtype=np.float64) | ||
| if not ( | ||
| np.allclose(stored_mean, action_mean, rtol=1e-10, atol=1e-10) |
There was a problem hiding this comment.
[P1] Compute both action statistics in the same numeric domain
The HDF5 moments are computed from the raw float64 actions, while the Paimon statistics were computed after the canonical actions were cast to float32. With this 1e-10 tolerance, ordinary decimal values can reject an otherwise equivalent dataset (for example, the mean of 0.1, 0.2, and 0.3 differs by about 5.5e-9 after a float32 round trip). The current integer-based test data masks this. Please cast the HDF5 actions to float32 before accumulating the moments, or compare the float32 normalization actually consumed, and add a decimal-valued regression case.
| Python 3.10 or newer is required. | ||
|
|
||
| ```shell | ||
| pip install 'pypaimon[act,hdf5]' |
There was a problem hiding this comment.
[P1] Install Vortex for the documented workflow
The standard RoboMIND ingestion configuration uses vector.file.format=vortex, but this command installs only the act and hdf5 extras. Vortex is a separate extra, and its pinned package is available only on Python 3.11+, so a clean environment following these instructions cannot read the default warehouse; the stated Python 3.10 baseline cannot run that path at all. Please require Python 3.11+ and include vortex here (or make it part of the act extra), and cover the default Vortex format with a smoke test.
|
|
||
| started_at = _utc_now() | ||
| started = time.monotonic() | ||
| fingerprint = _tensor_fingerprint(dataset_factory(), plan) |
There was a problem hiding this comment.
[P2] Move tensor fingerprinting out of the pre-timing path
_tensor_fingerprint constructs the dataset and reads and decodes every distinct planned sample before any round starts. The subsequent first-batch and throughput measurements access those same windows, so the measured working set is deterministically warmed outside warmup_batches, and first_batch_s cannot represent a cold first access. The page-cache disclaimer does not cover this benchmark-induced pre-read. Please compute the fingerprint after all timed rounds or in an isolated validation process.
Summary
Add a reproducible paired ACT benchmark for equivalent RoboMIND data stored in HDF5 and Paimon.
This PR now contains only the benchmark. It builds on the snapshot-pinned
ContiguousWindowDatasetmerged in #9580.Changes
prepare,run, andcomparecommands for independent HDF5 and Paimon runs using one resolved experiment.frame_indexconsistently as the ACT sample position across both storage backends.Testing
master(7d7a67281).python -m pytest pypaimon/tests/contiguous_window_dataset_test.py pypaimon/tests/act_benchmark_test.py pypaimon/tests/act_runner_test.py -q(51 passed).python -m pytest pypaimon/acceptance -q(3 passed).python -m flake8 --config=dev/cfg.ini pypaimon/benchmark/act pypaimon/tests/act_benchmark_test.py pypaimon/tests/act_runner_test.py setup.py.python dev/check_license_header.py(739 files have valid license headers).python -m compileall -q pypaimon/benchmark/act.git diff --check.Notes