GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types by rustyconover · Pull Request #53994 · apache/arrow

rustyconover · 2026-10-16T16:33:12Z

Note

Depends on #54959, which holds the format change, split out at review request. This PR is stacked on it: please review the C++ implementation commits after the format commit. Once #54959 merges, this will be rebased onto main.

Rationale for this change

In IPC a dictionary-encoded column is a single Field whose type is the value type. The only metadata slot was Field.custom_metadata, so dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialized identically. Arrow C++ reads both as the latter, so pyarrow cannot read back its own stream of pa.dictionary(pa.int32(), pa.uuid()): Invalid storage type for UuidType: dictionary<...>.

#54959 adds an optional custom_metadata to DictionaryEncoding for the dictionary value type (format version 1.6). This PR is the C++ reference implementation. Companion PRs:

Go, the second reference implementation: feat(arrow/ipc): store dictionary value type metadata in DictionaryEncoding arrow-go#1381
Rust, which reads the new member and maps it into Field metadata until Field metadata for dictionary-encoded extension types is lost arrow-rs#7982 is fixed: Add custom_metadata to IPC DictionaryEncoding arrow-rs#11622
nanoarrow, which maps the member losslessly to ArrowSchema.dictionary->metadata: feat(ipc): Read and write DictionaryEncoding.custom_metadata for dictionary values arrow-nanoarrow#972

What changes are included in this PR?

Generated code: Schema_generated.h regenerated with flatc 24.3.6, limited to the new member.
C++: the IPC writer and reader, and the integration JSON reader and writer ("dictionary": {..., "metadata": [...]}).
- The writer puts a dictionary value extension type's ARROW:extension:* keys in DictionaryEncoding.custom_metadata.
- The reader applies them to the value type before wrapping it in the dictionary type, then applies Field.custom_metadata as before.
- DictionaryType::value_metadata() retains unconsumed metadata, including unregistered extension annotations and arbitrary keys, separately from the enclosing field. IPC and integration JSON write it back to the dictionary encoding. No reader option is needed.
- C Data import/export preserves the same metadata on ArrowSchema.dictionary->metadata.
- If the values are a registered extension type, its annotations are authoritative; conflicting or duplicate reserved keys in supplied value metadata are removed without mutating the caller.
- Metadata-aware type/schema equality and fingerprints include value metadata. Dictionary type promotion retains the first type's metadata. Existing constructor and factory overloads remain available.
- Dictionary traversal looks through extension value types to find nested dictionaries. The dictionary fingerprint fallback also now compares actual types when an extension type has no fingerprint.
Archery: a new dictionary_extension case, run for C++ only for now.
- Go is skipped until the companion arrow-go PR lands, because integration CI builds arrow-go from main. With that PR, C++ ⇄ Go passes over IPC.
- Rust is skipped because arrow-rs cannot represent a dictionary of an extension type (Field metadata for dictionary-encoded extension types is lost arrow-rs#7982). It reads the case correctly but cannot produce it, and archery has no producer-only skip.
Not included: the Ruby flatbuffers bindings (ruby/red-arrow-format/.../dictionary_encoding.rb) were not regenerated, because no Ruby toolchain was available. They need rake flat_buffers:generate.
Follow-ups:
- A Parquet ARROW:schema round-trip test.

Are these changes tested?

Yes.

C++ IPC tests:
- schema and record batch round trips, including nested types and the file footer;
- the flatbuffer layout;
- delta and replacement dictionaries of an extension value type;
- unknown-extension read/write/read round trips, including restoration after registration, extra metadata, nulls, ordered dictionaries and nested dictionaries inside extension values;
- legacy field-level metadata, which behaves as before.
Also: JSON integration tests, C Data metadata round trips, metadata-aware equality/promotion and reserved-key conflict tests. Metadata containing NUL bytes is covered in IPC, JSON and C Data. The original implementation also includes archery C++ IPC/C Data coverage and a pyarrow regression test.
Follow-up validation: 826 C++ tests passed across the full type, IPC read/write, C Data bridge and JSON integration suites. clang-format 18.1.8 and cpplint 1.6.1 passed.
Cross-implementation, run manually: C++ ⇄ Go passes with the companion arrow-go PR, and arrow-rs reads the C++ and Go output in every format. The follow-up was additionally checked against arrow-go commit e3ee6e33604767c2d3122b1c2ea4336c3ae96f06: repeated C++ → Go → C++ and Go → C++ → Go IPC file/JSON conversions preserve unknown extension annotations, ordinary metadata (including duplicate keys and NUL bytes), separate parent metadata, nested fields and dictionary data.

Are there any user-facing changes?

Yes:

C++ and pyarrow now round-trip a dictionary whose value type is an extension type (e.g. pa.dictionary(pa.int32(), pa.uuid())) through IPC.
The C++/pyarrow IPC output for such columns changes: the extension keys move from Field.custom_metadata to DictionaryEncoding.custom_metadata.
- Readers that predate this change, including older C++/pyarrow, read such columns as a plain dictionary of the storage type and lose the extension type.
- That includes types like arrow.opaque, which older readers used to keep, though with the wrong nesting (as an extension type with dictionary storage).
Extension types whose storage type is a dictionary are serialized as before.
There is deliberately no fallback for data written by existing writers. A field-level arrow.uuid on a dictionary still fails to read, as it does today.

Was AI used for this PR?

PR code and description written by:

Human
AI

Latest follow-up reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [C++][Format] Can't roundtrip Dictionary of ExtensionType over IPC #49704

Initial implementation generated with Claude Code; metadata-preservation follow-up prepared with Codex.

github-actions · 2026-10-16T16:34:10Z

⚠️ GitHub issue #49704 has been automatically assigned in GitHub to PR creator.

…ionary values Implements the draft Arrow format 1.6 change from apache/arrow#49704 (apache/arrow#53994): an optional custom_metadata on DictionaryEncoding that holds the metadata of the dictionary value type. It is the IPC counterpart of ArrowSchema.dictionary->metadata, and Field.custom_metadata corresponds to the parent ArrowSchema's metadata. - flatcc_generated.h is regenerated with dev/update_fbs.sh (flatcc e3e44836) from apache/arrow main plus the draft Schema.fbs change. - The encoder writes schema->dictionary->metadata (all keys) to DictionaryEncoding.custom_metadata, omitted when it is NULL or empty, and keeps writing schema->metadata to Field.custom_metadata. Before, the value metadata was dropped. - The decoder sets schema->dictionary->metadata from DictionaryEncoding.custom_metadata and keeps Field.custom_metadata on the parent schema, as before. This covers nested dictionaries and footer decoding. Footer encoding carries the member when the caller fills footer.dictionaries. - The integration JSON reader and writer support the optional "metadata" of a field's "dictionary" object, and the JSON comparison compares the dictionary value metadata, including nested dictionaries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

raulcd · 2026-10-19T09:12:43Z

+    if (encoding->custom_metadata() != nullptr) {
+      ARROW_ASSIGN_OR_RAISE(auto value_metadata,
+                            internal::GetKeyValueMetadata(encoding->custom_metadata()));
+      RETURN_NOT_OK(MaybeDeserializeExtensionType(value_metadata.get(), &type));


I am wondering whether there would be a way of retaining this value_metadata even if this is a not known extension type. So the reader could retain it and a new writer would write it. I am thinking on the use case of data passing through:
app knows extension type -> app does not know extension type -> app knows extension type

Maybe not entirely related and just an implementation detail that can be done as a follow up.

Hi @raulcd,

Do you think we need to make a different IPC reader options flag for that? Where we can do extension type pass through?

Rusty

pitrou · 2026-10-22T15:22:43Z

@rustyconover Can you split the Format changes into a separate PR to make them easier to review? Thanks!

rustyconover · 2026-10-22T16:10:11Z

@rustyconover Can you split the Format changes into a separate PR to make them easier to review? Thanks!

Sure I can and I will! thanks @pitrou

… value type In IPC, a dictionary-encoded field is a single Field whose type is the dictionary value type, and the only metadata slot was Field.custom_metadata. As a result, dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> were serialized identically. Add an optional custom_metadata member at the end of the DictionaryEncoding table. It holds the custom metadata of the dictionary value type, with the same semantics as Field.custom_metadata (including the reserved ARROW:extension:* keys), and is the IPC counterpart of ArrowSchema.dictionary->metadata in the C Data Interface. Document it in the columnar format (format version 1.6), the versioning history, the C Data Interface, and the integration JSON format, which gains a matching optional "metadata" member in the "dictionary" object. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ta for dictionary value types Implement the DictionaryEncoding.custom_metadata format addition in C++. The IPC writer now puts the extension keys of a dictionary value type there, and the reader deserializes the value extension type from it before wrapping it in the dictionary type and applying Field.custom_metadata as before. Regenerate Schema_generated.h, support the "metadata" member of the "dictionary" object in the integration JSON reader and writer, and add a "dictionary_extension" archery case, which C++ and Go produce and consume (Go skips the C schema format, Rust is skipped as it can read but not write the case). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

rustyconover · 2026-10-22T17:12:36Z

@pitrou Done: the format change is now in #54959 (issue #54958). This PR is stacked on it and contains only the C++ implementation, so please review just the second commit.

Note that #54959 was auto-closed by the 3-open-PR limit for contributors without repository access; I've asked on it for it to be reopened.

… trips Retain unregistered extension annotations and arbitrary value metadata on DictionaryType through IPC, integration JSON and the C Data Interface. Preserve metadata-aware equality and schema promotion, avoid conflicting reserved extension keys, and traverse nested dictionaries inside extension values. Validate unknown-extension pass-through and restoration after registration, nested dictionaries, binary metadata and unchanged field metadata. All 826 affected C++ tests and C++/Go IPC interoperability checks pass.

…ionary values Implements the draft Arrow format 1.6 change from apache/arrow#49704 (apache/arrow#53994): an optional custom_metadata on DictionaryEncoding that holds the metadata of the dictionary value type. It is the IPC counterpart of ArrowSchema.dictionary->metadata, and Field.custom_metadata corresponds to the parent ArrowSchema's metadata. - flatcc_generated.h is regenerated with dev/update_fbs.sh (flatcc e3e44836) from apache/arrow main plus the draft Schema.fbs change. - The encoder writes schema->dictionary->metadata (all keys) to DictionaryEncoding.custom_metadata, omitted when it is NULL or empty, and keeps writing schema->metadata to Field.custom_metadata. Before, the value metadata was dropped. - The decoder sets schema->dictionary->metadata from DictionaryEncoding.custom_metadata and keeps Field.custom_metadata on the parent schema, as before. This covers nested dictionaries and footer decoding. Footer encoding carries the member when the caller fills footer.dictionaries. - The integration JSON reader and writer support the optional "metadata" of a field's "dictionary" object, and the JSON comparison compares the dictionary value metadata, including nested dictionaries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This was referenced Oct 16, 2026

feat(arrow/ipc): store dictionary value type metadata in DictionaryEncoding apache/arrow-go#1381

Open

Add custom_metadata to IPC DictionaryEncoding apache/arrow-rs#11622

Open

github-actions Bot added the awaiting review Awaiting review label Oct 16, 2026

rustyconover mentioned this pull request Oct 16, 2026

[C++][Format] Can't roundtrip Dictionary of ExtensionType over IPC #49704

Open

github-actions Bot added Component: C++ Component: Python Component: Documentation and removed awaiting review Awaiting review labels Oct 16, 2026

rustyconover mentioned this pull request Oct 16, 2026

feat(ipc): Read and write DictionaryEncoding.custom_metadata for dictionary values apache/arrow-nanoarrow#972

Draft

rustyconover force-pushed the gh-49704-dictionary-value-metadata branch from 401634e to 0cd8838 Compare October 16, 2026 17:28

github-actions Bot added the awaiting review Awaiting review label Oct 16, 2026

This was referenced Oct 19, 2026

Support the proposed DictionaryEncoding.custom_metadata (format 1.6) for dictionaries of extension types in IPC apache/arrow-dotnet#475

Open

Support DictionaryEncoding.custom_metadata (format 1.6) for dictionaries of extension types apache/arrow-dotnet#476

Draft

raulcd reviewed Oct 19, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Oct 19, 2026

rustyconover and others added 2 commits October 22, 2026 12:50

This was referenced Oct 22, 2026

[Format] Add custom_metadata to DictionaryEncoding for the dictionary value type #54958

Open

GH-54958: [Format] Add custom_metadata to DictionaryEncoding for the dictionary value type #54959

Open

rustyconover changed the title ~~GH-49704: [Format][C++] Add custom_metadata to DictionaryEncoding for the dictionary value type~~ GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types Oct 22, 2026

rustyconover force-pushed the gh-49704-dictionary-value-metadata branch from 0cd8838 to b267de8 Compare October 22, 2026 16:56

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 22, 2026

Navigation Menu

Repository navigation

GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types - #53994

GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types#53994
rustyconover wants to merge 3 commits into
apache:mainfrom
rustyconover:gh-49704-dictionary-value-metadata

rustyconover commented Oct 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

raulcd Oct 19, 2026

Uh oh!

rustyconover Oct 21, 2026

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

rustyconover commented Oct 22, 2026

Uh oh!

rustyconover commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

rustyconover commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Was AI used for this PR?

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

raulcd Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

rustyconover Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

rustyconover commented Oct 22, 2026

Uh oh!

rustyconover commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rustyconover commented Oct 16, 2026 •

edited

Loading