Skip to content

[Python] Fix tree-model reads and indexing in TsFileDataFrame - #1011

Open
Young-Leo wants to merge 1 commit into
apache:developfrom
Young-Leo:ly/tree-index
Open

Young-Leo wants to merge 1 commit into
apache:developfrom
Young-Leo:ly/tree-index

Conversation

@Young-Leo

Copy link
Copy Markdown
Contributor

TsFileDataFrame could reject valid tree-model datasets when different devices used the same measurement name with different physical types. For example, root.a.b.temperature (DOUBLE) and root.c.d.temperature (INT32) failed index construction with incompatible physical type for root.temperature. Without the index, scanning both devices through a shared tree-table column could also decode values using the wrong type.

This change makes numeric tree sequences usable through the existing sequence, statistics, and training-window APIs.

Changes

  • Validate tree-model physical types per full logical series path. Allow independent devices to use different types for the same measurement name, while retaining same-series shard validation and table-model schema checks.
  • Represent heterogeneous synthetic fields with a float64 logical type and a mixed physical-type marker. Preserve the Dataset Index v1 record layout; actual decoding uses each locator's TimeseriesMetadata.
  • Read unindexed tree sequences through exact metadata locators instead of scanning all devices through a shared column type. Preserve timestamps, aligned null positions, and float64 output.
  • Filter column lookup by FIELD role so measurements such as _col_1 do not resolve to synthetic path tags.
  • Initialize aligned time decoders from the chunk-header encoding, with INT64 timestamp values, in both serial and parallel paths. This fixes reads of Java-generated aligned tree files using PLAIN time encoding.
  • Add 29 tree-model regression cases, Java-generated V4 fixtures for PLAIN and TS_2DIFF time encoding, fixture generation source, and English/Chinese usage documentation.

Validation

  • 325 Python tests passed across test_tree_dataset_training.py, test_tsfile_dataset.py, test_dataset_index.py, and test_dataset_read_options.py.
  • 149 C++ reader, aligned-data, prepared-series, and Dataset Index tests passed.
  • The configured Python/C++ Dataset Index interoperability test passed separately with an actual heterogeneous tree index, bringing the C++ total to 150 passing tests.
  • Black checks, C++ Spotless formatting, and git diff --check passed.

Coverage includes single-file and multi-file datasets, independent measurement types, full-path reads, statistics, positional and time-range windows, index reload without metadata rescanning, name collisions, and aligned nulls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant