Skip to content

[python] Add paired ACT storage benchmark - #9466

Open
YannByron wants to merge 1 commit into
apache:masterfrom
YannByron:m0/pr3-act-benchmark
Open

[python] Add paired ACT storage benchmark#9466
YannByron wants to merge 1 commit into
apache:masterfrom
YannByron:m0/pr3-act-benchmark

Conversation

@YannByron

@YannByron YannByron commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a reproducible paired ACT benchmark for equivalent RoboMIND data stored in HDF5 and Paimon.

This PR now contains only the benchmark. It builds on the snapshot-pinned ContiguousWindowDataset merged in #9580.

Changes

  • Add prepare, run, and compare commands for independent HDF5 and Paimon runs using one resolved experiment.
  • Verify source identity, logical window order, tensors, and loss traces before comparing performance metrics.
  • Use frame_index consistently as the ACT sample position across both storage backends.
  • Package the default experiment and optional ACT benchmark dependencies.
  • Document the workflow, reported metrics, and measurement boundaries.

Testing

  • Rebased onto the latest master (7d7a67281).
  • python -m pytest pypaimon/tests/contiguous_window_dataset_test.py pypaimon/tests/act_benchmark_test.py pypaimon/tests/act_runner_test.py -q (51 passed).
  • python -m pytest pypaimon/acceptance -q (3 passed).
  • python -m flake8 --config=dev/cfg.ini pypaimon/benchmark/act pypaimon/tests/act_benchmark_test.py pypaimon/tests/act_runner_test.py setup.py.
  • python dev/check_license_header.py (739 files have valid license headers).
  • python -m compileall -q pypaimon/benchmark/act.
  • git diff --check.

Notes

  • Ingestion and canonical-action backfill are outside timed benchmark runs.
  • The benchmark does not control the OS page cache.
  • GPU, multi-worker loading, distributed training, recovery, and policy quality are outside this local CPU benchmark.

@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch from 622ceae to 5d8890e Compare August 29, 2026 14:53
@YannByron
YannByron marked this pull request as draft August 29, 2026 14:54
@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch 2 times, most recently from fb77b0e to 7b2b5a4 Compare September 1, 2026 03:24
@YannByron
YannByron marked this pull request as ready for review September 1, 2026 03:51
@YannByron
YannByron marked this pull request as draft September 1, 2026 03:52
@YannByron
YannByron marked this pull request as ready for review September 2, 2026 01:14
Comment thread paimon-python/pypaimon/multimodal/window_dataset.py Outdated
Comment thread paimon-python/pypaimon/benchmark/act/runner.py Outdated
Comment thread docs/docs/pypaimon/robomind-act-benchmark.md
Comment thread paimon-python/pypaimon/benchmark/act/harness.py Outdated
Comment thread paimon-python/pypaimon/multimodal/window_dataset.py Outdated
Comment thread paimon-python/pypaimon/multimodal/window_dataset.py
@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch from 42be5b2 to 705683a Compare September 2, 2026 09:16
@YannByron YannByron changed the title [python] Add paired HDF5/Paimon ACT benchmark [python] Add ACT storage benchmark and contiguous windows Sep 2, 2026
@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch from 705683a to 9e0340a Compare September 3, 2026 02:48
@JingsongLi

Copy link
Copy Markdown
Contributor

Thanks for putting this together. This PR currently combines two independently valuable changes with different review and risk profiles: the generic ContiguousWindowDataset public API, and the RoboMIND ACT benchmark harness and workflow.

Given the 4.4k-line scope, could we split this into two PRs?

  1. Contiguous-window Dataset: the public API, row-ID and snapshot semantics, BLOB handling, batching behavior, tests, and API documentation.
  2. ACT benchmark: the prepare/run/compare workflow, backend adapters, packaging, benchmark documentation, and ideally a real paired Vortex result artifact with variance and physical-layout identity.

This would make the public API easier to review on its own, while allowing the benchmark methodology and reproducibility contract to be evaluated independently. It would also keep follow-up fixes in either area from blocking the other.

Rebuild the benchmark-only change on top of the merged contiguous-window dataset. Compare equivalent HDF5 and Paimon ACT inputs and use frame_index consistently in the public sample contract.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-5
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 2395/2395
AI-Contributed/UT: 895/895
@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch from 9e0340a to 78207fc Compare September 8, 2026 14:21
@YannByron YannByron changed the title [python] Add ACT storage benchmark and contiguous windows [python] Add paired ACT storage benchmark Sep 8, 2026

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three actionable issues found in the updated benchmark-only scope.

stored_mean = np.asarray(row["action_mean"], dtype=np.float64)
stored_std = np.asarray(row["action_std"], dtype=np.float64)
if not (
np.allclose(stored_mean, action_mean, rtol=1e-10, atol=1e-10)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Compute both action statistics in the same numeric domain

The HDF5 moments are computed from the raw float64 actions, while the Paimon statistics were computed after the canonical actions were cast to float32. With this 1e-10 tolerance, ordinary decimal values can reject an otherwise equivalent dataset (for example, the mean of 0.1, 0.2, and 0.3 differs by about 5.5e-9 after a float32 round trip). The current integer-based test data masks this. Please cast the HDF5 actions to float32 before accumulating the moments, or compare the float32 normalization actually consumed, and add a decimal-valued regression case.

Python 3.10 or newer is required.

```shell
pip install 'pypaimon[act,hdf5]'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Install Vortex for the documented workflow

The standard RoboMIND ingestion configuration uses vector.file.format=vortex, but this command installs only the act and hdf5 extras. Vortex is a separate extra, and its pinned package is available only on Python 3.11+, so a clean environment following these instructions cannot read the default warehouse; the stated Python 3.10 baseline cannot run that path at all. Please require Python 3.11+ and include vortex here (or make it part of the act extra), and cover the default Vortex format with a smoke test.


started_at = _utc_now()
started = time.monotonic()
fingerprint = _tensor_fingerprint(dataset_factory(), plan)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Move tensor fingerprinting out of the pre-timing path

_tensor_fingerprint constructs the dataset and reads and decodes every distinct planned sample before any round starts. The subsequent first-batch and throughput measurements access those same windows, so the measured working set is deterministically warmed outside warmup_batches, and first_batch_s cannot represent a cold first access. The page-cache disclaimer does not cover this benchmark-induced pre-read. Please compute the fingerprint after all timed rounds or in an isolated validation process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants