Rewrite Arrow.jl internals for 3.0 - #609
Draft
quinnj wants to merge 301 commits into
Draft
Conversation
Reject unsupported hosts before old generated FlatBuffers code can interpret little-endian metadata with native-endian loads. Co-Authored-By: Codex <codex@openai.com>
Document the fresh IPC endianness finding, its disposition, the withdrawn candidates, and validation evidence. Add round five to the README file index. Co-Authored-By: Codex <codex@openai.com>
Require element alignment only when a vector has an element to read. Add an end-to-end zero-row Null batch regression that matches the empty struct-vector encoding found in official integration streams. Co-Authored-By: Codex <codex@openai.com>
Reject KeyValue entries whose value field is absent instead of silently converting absence to an empty string. Cover both the malformed case and a valid explicit empty value. Co-Authored-By: Codex <codex@openai.com>
State that the C stream importer and facade are future users of the shared pull protocol. Only the IPC reader implements that protocol in this prove-out. Co-Authored-By: Codex <codex@openai.com>
Reject an absent Schema.fields vector instead of treating it as an empty schema. Keep a present zero-length vector valid and cover both encodings. Co-Authored-By: Codex <codex@openai.com>
Avoid claiming that unimplemented C stream and partition adapters already exchange RecordBatch values. Distinguish the implemented IPC path from the target design. Co-Authored-By: Codex <codex@openai.com>
Require every vector length word to be four-byte aligned, including empty vectors. This prevents malformed empty table vectors from reaching an old generated unsafe_wrap getter with a misaligned pointer while retaining valid relaxed element alignment for empty struct vectors. Co-Authored-By: Codex <codex@openai.com>
Document the fresh IPC verifier and documentation findings, their dispositions, scope decisions, and validation evidence. Add round six to the README file inventory. Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Return the exact lifecycle state to open when the winning closer is interrupted while waiting for active guards. This keeps the region usable and allows a later close to release it exactly once. Co-Authored-By: Codex <codex@openai.com>
Clarify that same-size mutation can invalidate cached semantic validation for borrowed Julia buffers, while resizing can also invalidate their pointers. Co-Authored-By: Codex <codex@openai.com>
Detect Arrow 0.17's Message-level experimental compression marker before record or dictionary body decoding. This prevents length-prefixed compressed bytes from being exposed as ordinary values. Co-Authored-By: Codex <codex@openai.com>
State that mapped and foreign backing bytes must stay alive and unchanged while Core uses them or their cached validation certificates. External mutation cannot be detected by the prove-out. Co-Authored-By: Codex <codex@openai.com>
Reject unknown flag bits and flags attached to layouts where their semantics do not apply. Failed imports still release both moved lifetimes exactly once. Co-Authored-By: Codex <codex@openai.com>
Document five adversarial findings, their dispositions, scope decisions, and final validation. Update the README review index for round eight. Co-Authored-By: Codex <codex@openai.com>
Construct imported owners without a finalizer, move the source under an interruption-safe handoff, and arm the copied owner only after ownership transfers. Failed post-move setup releases the copied producer callback exactly once. Co-Authored-By: Codex <codex@openai.com>
Keep the claim, recursive release, and completion transaction interrupt-safe. Roll failed claims back to LIVE and retry idempotent descendant progress before the void C callback returns. Co-Authored-By: Codex <codex@openai.com>
Keep each root registered while one reaper owns its cleanup. Record native frees and source-pin releases as progress so interrupted cleanup can resume without leaks or double frees. Co-Authored-By: Codex <codex@openai.com>
Install release and cleanup rollback handlers before their state claims. Preallocated claim slots let task-delivered exceptions restore LIVE or clear the cleanup claim at the exact post-mutation boundary. Co-Authored-By: Codex <codex@openai.com>
Use the source ArrowArray release field as the authoritative move marker. Task cancellation immediately after the move store now releases only the copied owner and cannot strand the producer callback. Co-Authored-By: Codex <codex@openai.com>
Document the three C Data interruption and ownership findings, their dispositions, scope decisions, and final validation evidence. Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Codex <codex@openai.com>
Retry interrupted mapping release at both constructor and finalizer ownership boundaries. Check munmap failures before publishing the mapping as released. Co-Authored-By: Codex <codex@openai.com>
…er route prose - ci: pass `project: matrix.pkg.dir` to setup-julia so each package's `min` cell resolves its own Julia compat (ArrowStrings 1.10) - manual: composite layouts are dynamic-path with schema-derived types (union and undeclared-null exceptions named); dictionary and run-end wrappers are transparent to their value child's route and type Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With setup-julia resolving each package's own floor, ArrowTypes' declared `julia = "1.0"` selects 1.0.5, whose Pkg predates test/Project.toml and which has no macOS ARM64 binary; the cells cannot run its suite on either OS. Its `lts` cells remain its oldest lane. The floor itself is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Arrow.jl `julia = "1.10"` (was 1.12), ArrowTypes.jl `1.10` (was 1.0); ArrowStrings.jl already 1.10. JuliaC `--trim` needs 1.12, so the trim gate stays 1.12-only (test/trim_compile_tests.jl already guards on it). - ArrowCore: the mmap release cell targets `_mmaproot(arr)` — the backing `Memory` from Julia 1.11, the array itself before (where Mmap registers its unmap finalizer); `mmapregion`'s docstring now attaches to it - ArrowCore: buffer-role sequences are shared constants; `layoutspec` no longer builds a vector per buffer-by-role lookup (Julia 1.11+ elided it, 1.10 did not: typed struct materialization now 1.6 MB/100k rows on 1.12 and 4.8 MB on 1.10, from 8.0 MB / 17.6 MB) - ci: ArrowTypes back in the `min` cells; local ArrowTypes+ArrowStrings are developed BEFORE the build step (Julia 1.10's Pkg does not read `[sources]`, and ArrowStrings is unregistered) - manual: state the 1.10 floor and the 1.12 trim requirement Verified on Julia 1.10.11 and 1.12.6: Arrow 420/321/4 + acceptance, ArrowTypes 133, ArrowStrings 2495; trim 6/6 (1.12); docs; docker conformance corpus/oracle/cdata. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The release audit (dev/release/run_rat.sh) flagged 66 files without the header: the review records and design/research notes under docs/dev, the conformance, bench and trim environment Project files, and one test file with a truncated header. ArrowTypes.jl 2.4.0 carries its raised Julia floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…omments
Tests
- batteries.jl runs the statistics and byte-range scan suites again
(`_stats_main`, `_scan_main` → `_ranged_main`); they had not been called
since the rewrite replaced src/, so ~720 lines of pins were dormant
- child processes: the typed-allocation and threaded-stress children are
`@test success(...)` with the suite's project; the "concurrent IPC pulls"
launch (an env-var guard nothing read) and two orphan pass lines removed
- shared helpers: `battery_prelude.jl` (the alias loop), MIXED_EXPECTED +
`_mixed_two_partitions_bytes`, `_viewentry`/`_viewlong` reused; the fetch
accounting (`FetchLog`/`countingsource`/`_fetched`) moves from src/scan.jl
to the test helpers; `_rejects` is a `function`; `@test true` placeholders
and value dumps dropped; history narration in comments rewritten
- test/Project.toml declares Pkg (trim_compile_tests.jl imports it)
Dead code
- ArrowCore: `_validate_temporal_values`, the `_validate_advisory_values`
fallback, `arrowtype_for(::Type{String})`, an unreachable Struct branch,
redundant structural re-checks in `_validate_ree_values`, `batch`'s
duplicate validation loop, two unused default arguments
- IPC/C data: `_blockmessagebodylength`, `_containsdictionary`, the 5-arg
`IPCStream`, `_vu16` alias, `formatstring(::DictionaryType)`; facade:
the 4-arg `_writecolumn`, 3-arg `_scanbatch`, `Arrow`-scope `using EnumX`
- vendored FlatBuffers: `finishwithfileidentifier`, `createsharedstring!`
(+ the `sharedstrings` field), `createbytevector`, `prependstructslot!`,
the unused `sh` parameter, `getvalue` (called an undefined function),
`getoffsetslot`, `getslot`, `setindex!`, the Builder-based table ctor;
`src/metadata/Flatbuf.jl` and its generator branch (Arrow.jl's `Meta`
module is the include target)
Behaviour
- `fromjulia` on a `Vector{Union{Missing,Vector}}` declares the list field
nullable from the ELEMENT TYPE, like every other builder, not from the
observed null count
- `DictEncode{T,V} <: AbstractVector{T}` (was `AbstractVector{Any}` with an
`eltype` override that disagreed)
- `_validate_descriptor(::TimestampType)` is a `function` (banned form)
- error text uses `descriptorname` where the file mandates it; the
`RecordBatch` arity check precedes per-column validation
Docs and comments (present state only; wrong claims corrected)
- docstrings attached to the right definitions (`mmapregion`,
`ForeignOwner`, `_boundschema`, `_blockmessage`); new one-liners for the
descriptor structs, `Schema`, `BufferRole`, `layoutspec`,
`close!(::ForeignOwner)`, `release!(::StreamOwner)`, typed
`materialize`, `statsfile`, `ArrowTypes.ToArrow`, `Arrow.ValidationError`
- `Arrow.Table`/`Arrow.Stream` docstrings: source kinds, `mmap`, decode set;
`Arrow.jl` module docstring corrected; `getvalue`'s dispatch claim,
`fromjulia`'s Bool claim, `decodefield`'s signature/variadic claim,
`framemessages`' and other stale cross-file references
- generator header/comments and the research note describe the present
tree; the DESIGN note's counts, statistics keys and extension shape;
manual: Tables `jq/scan` prerequisite, file-format-only mmap, `close!`
target, `columnnames` output, LargeListView, native-endian decimals,
ArrowCore currency in the C-data section; README: public surface,
ArrowStrings registration status, layout entries, docker network note;
reference: `ImportedStream`, `close!(::ForeignOwner)`, `ValidationError`
- ArrowTypes: docstring `@ref` targets and typos; a README
Tooling
- CI: `1.11` replaces `lts` (identical to `min` at a 1.10 floor); Arrow.jl
tests develop only ArrowStrings; codecov waits for the 48 uploads
- conformance: unused EnumX/PooledArrays dropped from the env and image
warm list; the host env pins Harbor 1.1; corpus requires
ARROW_TESTING_DIR; dead `_oracle_child` and stale usage lines removed;
Dockerfile pins explained, `CMD` matches the driver
- bench: one workload-name list, absolute workdir; release: ArrowStrings
registration and RC verification steps; `.gitignore`/rat exclusions
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repository's .JuliaFormatter.toml (the CI Format job) had never been run over the 3.0 sources; origin/main is formatter-clean, so the branch must be too. Whitespace/wrapping only — the formatter is idempotent on the result, and every suite passes on Julia 1.10.11 and 1.12.6 before and after (Arrow 421/321 + batteries, ArrowStrings 2495, ArrowTypes 133, trim 6/6, docs, docker conformance 275/170/143). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e, two dead helpers - bench/run.jl: the multi-line backtick command literals are single-line (interpolated locals for the long paths), so a formatter pass cannot change the raw string a `@cmd` macro receives - docs/dev/research-flatbuffers-cdata.md describes the present tree: the writer surface and FlatBuffers gap list without the deleted builder helpers, no line counts, the two malformed reference remnants removed; the C-data comparison states the semantic-tier import validation, the shared-cell `close!` revocation, the 1 MiB C-string bound, imported and exported schema metadata, the existing pyarrow C-data oracle, and the public `release!`/`close!` names - conformance/arrowjson.jl `_u64` and tools/fbsgen.jl `_ident` (both uncalled) removed; the four generated files still regenerate byte-identically after JuliaFormatter - The conformance image's warm list keeps EnumX on purpose: it is a direct Arrow dependency, which is the list's stated rule (690e1cc's message overstated the removal) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… corrected The FlatBuffers/C-data research note now states the vendored builder's shared low-level ancestry with JuliaData/FlatBuffers.jl, the complete runtime surface the rewrite uses, the staged verify order (root start → inline version gate → root rest), the builder head underflow behaviour, how each Base-name collision is resolved, the schema release timing, the eager null count on import, and that schema metadata import trusts the producer's declared extents (no bounds claim). The cdata.jl header citation that does not exist is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tension The byte-range read surface is a source interface plus `Tables.Scan` pushdown on `Arrow.Table`, not a special handle: - `Arrow.AbstractArrowSource` (src/source.jl): an implementation defines `sourcelength(src)` and `readrange(src, offset, len)`, and may override `readranges(src, ranges)` to issue a round's planned ranges concurrently - `Arrow.Table(src::AbstractArrowSource; scan=…)`: the footer comes from one tail read (cached on the handle; its trailing magic also decides file vs stream format — a stream object is read whole), statistics and the scan window prune batches, only the surviving batches' metadata and the selected + filter-referenced columns' buffers are requested, coalesced — three request rounds. The leading magic is no longer fetched: the footer is the sole authority. `Arrow.Stream(src)` reads the object whole - ext/ArrowCloudStoreExt.jl (`[weakdeps] CloudStore`): a `CloudStore.Object` is a source (its known size, one HTTP `Range` GET per range, one task per planned range) and `Arrow.Table(obj; …)` / `Arrow.Stream(obj; …)` accept it directly - REMOVED from the public surface: `RangedSource`, `RangedFile`, `fetchranges`. The planner (`SourceFile`, internal) is the same code re- skinned: `_fetchexact`/`_fetchspans` read through the interface Tests: the ranged battery runs over `BytesSource`/`CountingSource` (exact ranges pinned as before); test/cloudstore_tests.jl drives the extension end to end against CloudBase's Minio server (ranged reads, projected/windowed and statistics-pruned scans, whole reads of file- and stream-format objects, `Stream`); CloudStore/CloudBase are test deps. Docs: manual remote-reads section, reference (`AbstractArrowSource`, `sourcelength`, `readrange`, `readranges`), DESIGN §2, core-README, README. Verified on Julia 1.10.11 and 1.12.6: Arrow 421/321 + all batteries + extension 12/12; trim 6/6; docs; JuliaFormatter no-op; rat. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g, one body round - (H) `readranges` is gone from the source interface: Arrow reads a round's planned spans itself, through `readrange`, with a worker pool of `concurrentreads(src)` tasks (default 1) pulling requests off one counter and storing every result by request index — a transport's completion order can never permute payloads, and the reads in flight are bounded whatever the span count. `readrange` results are type- and length-checked; the docstring states the trusted-source boundary (Arrow cannot authenticate the bytes a source returns for a range) - (M) CloudStore extension: every range GET carries `If-Match` with the object's ETag, so a handle is pinned to one object version — an overwritten key fails the next read instead of mixing versions across request rounds (pinned against Minio); `concurrentreads` is 16 - (M) the planner fetches the selected dictionary bodies and the selected record buffers in ONE round (dictionaries decode first from the shared spans); the dead `blockwants` map is gone - (L) a Footer larger than the tail window is read once and cached on the handle; ranges the cached tail already covers are served from it without a request (a file no larger than the window costs exactly one request) - (L) `sourcelength` is validated as an Integer before conversion; the coalescer's negative-range guard is a `ValidationError` - (L) `Arrow.Table(src)` without a scan, with an unpushable scan, over a zero-field file, or over a stream-format object reads the object whole (the cached tail plus its prefix: two requests) — what the manual and docstring say; the DESIGN request-count model matches - tests: facade pins for request rounds through the public path, tail-once, tail-cache serving, bounded concurrency with out-of-order completion, and every contract violation (short/long/wrong-type payloads, invalid lengths) as ValidationError; the ext test pins ETag rejection; the battery's coalescing check plans outside a small tail Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt worker pool, no GCS claim - `SourceFile` requires `sourcelength` to return an `Integer` (a Float, String, or `nothing` is a ValidationError, pinned through `Arrow.Table`) - the range-read worker pool follows the repository guidelines: the shared request counter is an `@atomic` field on a mutable `_SpanQueue` (no `Threads.Atomic`), each worker is `errormonitor(Threads.@Spawn …)`, and the per-worker loop is its own function - the CloudStore extension claims S3 and Azure Blob Storage only: the registered CloudStore releases (1.6–1.8) have no GCS `Object` path Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e extension) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Arrow.jl test cell, the monorepo job, the docs job and the release verification failed at precompile because registered Tables.jl has no `Tables.Scan`. The root Project's `[sources]` now pins Tables to the `jq/scan` branch (read by Pkg from Julia 1.11, including the release verification's Julia 1.12); the 1.10 test cells, the monorepo env and the docs env add it explicitly alongside the local ArrowStrings develop. The docs job installs Julia through setup-julia like the other jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Complete the final correctness, API, documentation, test, CI, conformance, and release-administration pass for the Arrow 3 rewrite. Track the exact reviewed Tables.Scan prerequisite and add focused regressions for its current API and semantics.
Keep ArrowTypes minimum-version coverage on Ubuntu while excluding only the macOS Apple Silicon cells, for which Julia 1.0 has no binary.
Process root and subpackage coverage from their own source directories so the ArrowTypes Julia 1.0 lane never parses Arrow 3 source syntax.
Restore ArrowTypes as a facade dependency and compatibility boundary. Lower and lift custom values recursively, preserve extension metadata, synthesize dense unions, and keep extension filters in the public value domain. Add cross-version, downstream, nested-layout, partition, scan, and malformed-input coverage. Update the migration, release, CI, and conformance guidance for ArrowTypes 2.4 and ArrowStrings 1.0.
… release files - bench/: the 2.x leg (bench_2x.jl, env2x/) is gone; the driver times this package and PyArrow - docs/dev/research-flatbuffers-cdata.md removed - dev/release/README.md and verify_rc.sh are back at main's version Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- `release!` is the one release verb across the API: `release!(::Table)`, `release!(::Stream)`, `release!(::OwnerRegion)`, and the C-data owners. Arrow exports it (and imports ArrowCore's generic so cdata.jl extends it) - ForeignOwner has one verb and one contract: `release!(owner)` revokes every region over the import through the shared cell (later access is an InvalidStateException), then the cell's action runs the producer release exactly once — the raw post-release undefined behaviour is gone - `ArrowStrings.ArrowStringVector` is `ArrowStrings.StringVector` Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`_applyscan` now consumes a `Tables.Scan` exactly through `_ScanSink` (src/scan.jl), on both the in-memory and the byte-range paths: - the filter is evaluated per decoded batch by the generic evaluator (`Tables.filtermask` over the decode set), so Arrow and the executor share one three-valued semantics by construction - `offset`/`limit` compose over the qualifying rows with saturating arithmetic; without a filter the metadata window still skips whole batches, with one decoding stops the moment the window is full - selection and renames are applied at column construction, in selection order; the residual is empty except for type overrides - the battery pins the consumed residual, filter+limit in one pass, and that a filled window stops decoding before a corrupt later batch Tables.jl's `jq/scan` branch moved: `Tables.bind` is `Tables.resolve` (the `BoundScan` carries the name-normalized filter, so Arrow's own resolver is gone), `All()` is the projection identity, and `coleq`/`colne`/`in_` are `colcmp(==, …)`/`colcmp(!=, …)`/`colin` — adapted in src, tests, and manual. Docs: the DESIGN note describes the implemented pushdown (no stages); manual, core-README and README follow; the core-README "Interruption contract" section is removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unify scan planning and column construction around bounded, single-pass seams. Restore ArrowTypes lowering and lifting, remove JSON3 integration, deepen conformance and deterministic fuzz coverage, update documentation, and prepare ArrowStrings 1.0. Preserve empty-scan schema metadata and charge every package-owned allocation.
Tables.jl 1.14.0 ships Tables.Scan, so the temporary source override on the reviewed scan-branch commit is gone: the package, CI workflows, fuzz workflow, conformance image, and docs all resolve Tables from the registry, and the compat floor is 1.14. The released Scan represents "keep every column" as Tables.All() instead of nothing, so the resolver work model drops its now-impossible nothing branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tables.Scanonce, keep storage and public-domain plans separate, preserve stable empty-result schemas, and support sparse byte-range reads with footer statisticsDependency
JuliaData/Tables.jl#380 is merged and released as Tables.jl 1.14.0. Commit dd8c272 removes the temporary source and CI pins, raises the Tables.jl compatibility floor to 1.14, and resolves Tables from the General registry everywhere (package, CI workflows, conformance image, docs).
Breaking changes
This is the Arrow.jl 3.0 rewrite. Important changes include:
Arrow.TableandArrow.Streamreturn materialized Julia vectors instead of lazy Arrow vector wrappers.Arrow.write(io, table)writes file format by default; usefile=falsefor stream format.ArrowTypesis no longer exported by Arrow.jl. Packages that define mappings should depend on andimport ArrowTypesdirectly.Arrow.ArrowTypesremains as a qualified compatibility binding.Arrow.Fresh declared heterogeneous Julia Union columns are written as canonical dense Arrow Unions. A retained Union rewrite fails closed when materialization has lost its original route. Nested retained dictionary pools follow the same fail-closed rule.
See
CHANGELOG.mdanddocs/src/migration.mdfor the complete list and migration guidance.Validation
67e17cfpasses 115/115 GitHub checks across the push, pull-request, and Conformance workflows, including Windows Julia latestdd8c272, current head) passes the full root suites on Julia 1.10.11 and 1.12.6 against registered Tables 1.14.0, the JuliaC trim gate 6/6, Documenter, JuliaFormatter 2.12.4, and all three docker conformance suites (corpus, oracle, cdata) in the rebuilt imagegit diff --checkpassCo-authored by Codex