Repository navigation
Conversation
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Add type annotation stubs for array, table, tensor, builder, memory, device, config, and types modules. Includes type-ignore annotations in related tests.
Drop the TypeVar/Generic parameters from the DataType subclasses, Field, Array, ChunkedArray and the tabular classes. Where a property can only return one of a few types (dictionary index type, run end type, time units) it now returns a union or Literal instead. Most of these parameters weren't buying much yet: scalar.pyi and lib.pyi are still placeholders, so e.g. Array[Scalar[...]] collapsed to Any anyway, and several annotations were being flagged by mypy and pyright (TypeVars used only once, type arguments on subclasses that weren't actually generic). Plain classes are a lot easier to read and review, and we can bring parameters back later where they clearly pay off. This also removes the stub-only _BasicDataType and _HasPrecisionScale helpers, which have no runtime counterpart, plus the IntoArray and SingleOrList aliases.
Add pyarrow-stubs/tests with pandas-stubs style checks:
check(assert_type(lib.list_(lib.int32()), lib.ListType), lib.ListType)
assert_type is verified by mypy, pyright and ty, which already include
pyarrow-stubs/ in their runs, and check() verifies the runtime class when
the tests run under pytest, so the stubs can't silently drift from what
pyarrow returns. Usage that should be rejected goes under
`if TYPE_CHECKING_INVALID_USAGE:` with a `# type: ignore[...]`; unused
ignores are errors in these tests, so those fail if the usage starts
type checking.
To have something to test, lib.pyi now re-exports the stub modules that
are annotated so far, still falling back to Any for everything else.
The CI type annotation script runs the tests with pytest after the type
checkers.
|
|
table.pyi imports RecordBatchReader from ._ipc, which had no stub, so pyright and ty failed to resolve the import. Add a placeholder in the same style as io.pyi and scalar.pyi until _ipc gets fully annotated.
compute and interchange will be annotated in later PRs, so treat imports of them as Any instead of type checking their untyped sources (mypy) or failing to resolve them (ty). Recent ty versions also no longer honor `# type: ignore[<mypy code>]` comments, so disable the rules those comments suppress: method overrides in array.pyi, and invalid arguments in the tests, where mypy and pyright still check that the invalid usage is rejected.
- TypesEnum: list all members, with the runtime values. - is_nested() also matches run-end encoded types. - Arrays and chunked arrays have no as_py() method. - ChunkedArray.unique() returns an Array. - MapArray is a ListArray, so it has offsets and values. - value_lengths() of large list (view) arrays returns an Int64Array. - Array.from_pandas() arguments can be passed positionally. Add annotation tests for these.
Converting timedelta.max to a millisecond or second duration truncates the sub-unit part, so it doesn't round trip. The comparisons used to be missing their assert, which hid this.
An empty placeholder class rejects valid usage such as table.to_reader().read_all(), as it has no methods.
Decimal types subclass FixedSizeBinaryType, but is_fixed_size_binary() returns False for them. With TypeIs, type checkers would wrongly narrow decimal types away when the check fails.
Run ruff, mypy, pyright and ty on the stubs with pinned versions: pre-commit run --hook-stage manual python-stubs --all-files The hooks only run on request, so they don't affect the existing hooks or CI jobs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
This is the next step in adding type annotations to PyArrow (#32609). It merges #49103 (internal type system stubs) and #49193 (core data structures) because the two depend on each other and can't be type checked separately.
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: