GH-54958: [Format] Add custom_metadata to DictionaryEncoding for the dictionary value type

docs/source/format/CDataInterface.rst

@@ -537,6 +537,14 @@ For dictionary-encoded arrays, the :c:member:`ArrowSchema.format` string
 encodes the *index* type.  The dictionary *value* type can be read
 from the :c:member:`ArrowSchema.dictionary` structure.
 
+The :c:member:`ArrowSchema.metadata` of the :c:member:`ArrowSchema.dictionary`
+structure holds the metadata of the dictionary value type, such as the
+extension type of the dictionary values, while the parent structure's
+:c:member:`ArrowSchema.metadata` applies to the dictionary-encoded type as a
+whole.  In the IPC format, these correspond to the ``custom_metadata`` of the
+``DictionaryEncoding`` and of the ``Field``, respectively (see
+:ref:`format_metadata_extension_types_dictionary`).
+
 The same holds for :c:member:`ArrowArray` structure: while the parent
 structure points to the index data, the :c:member:`ArrowArray.dictionary`
 points to the dictionary values array.

docs/source/format/Columnar.rst

@@ -21,7 +21,7 @@
 Arrow Columnar Format
 *********************
 
-*Version: 1.5*
+*Version: 1.6*
 
 .. seealso:: :ref:`Additions to the Arrow columnar format since version 1.0.0
    <post-1-0-0-format-versions>`
@@ -1289,7 +1289,10 @@ array. This includes:
 * A ``dictionary`` property indicating whether the field is
   dictionary-encoded or not. If it is, a dictionary "id" is assigned
   to allow matching a subsequent dictionary IPC message with the
-  appropriate field.
+  appropriate field. In this case, the field's data type is the
+  dictionary value type, and the ``dictionary`` property may carry the
+  ``custom_metadata`` of the dictionary value type (see
+  :ref:`format_metadata_extension_types_dictionary`).
 
 We additionally provide both schema-level and field-level
 ``custom_metadata`` attributes allowing for systems to insert their
@@ -1677,7 +1680,10 @@ Custom Application Metadata
 We provide a ``custom_metadata`` field at three levels to provide a
 mechanism for developers to pass application-specific metadata in
 Arrow protocol messages. This includes ``Field``, ``Schema``, and
-``Message``.
+``Message``. The ``DictionaryEncoding`` of a dictionary-encoded ``Field``
+additionally has a ``custom_metadata`` field, with the same semantics as
+the ``Field``'s ``custom_metadata`` but applied to the dictionary value
+type (see :ref:`format_metadata_extension_types_dictionary`).
 
 The colon symbol ``:`` is to be used as a namespace separator. It can
 be used multiple times in a key.
@@ -1732,6 +1738,55 @@ extension types:
   metadata indicating the market trading calendar the data corresponds
   to
 
+.. _format_metadata_extension_types_dictionary:
+
+Dictionary-encoded extension types
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A dictionary-encoded field is serialized as a single ``Field`` whose data
+type is the dictionary value type and whose ``dictionary`` property is a
+``DictionaryEncoding`` giving the index type. The ``Field``'s
+``custom_metadata`` describes the field's own type, so an extension type
+whose storage type is a dictionary type stores its extension keys in the
+``Field``'s ``custom_metadata``, as any other extension type does.
+
+The ``custom_metadata`` of the ``DictionaryEncoding`` is the
+``custom_metadata`` of the dictionary value type. It has the same semantics
+as a ``Field``'s ``custom_metadata``, including the reserved
+``'ARROW:extension:*'`` keys, but applies to the dictionary values rather
+than to the field. It is the IPC counterpart of the metadata of the
+dictionary value ``ArrowSchema`` in the :ref:`C Data Interface
+<c-data-interface>` (``ArrowSchema.dictionary->metadata``), while the
+``Field``'s ``custom_metadata`` corresponds to the metadata of the parent
+``ArrowSchema`` (``ArrowSchema.metadata``).
+
+If the dictionary *value* type is an extension type, for example a
+dictionary of ``arrow.uuid`` values, writers should store the
+``'ARROW:extension:name'`` and ``'ARROW:extension:metadata'`` keys of the
+value type in the ``custom_metadata`` of the ``DictionaryEncoding`` rather
+than in the ``Field``'s ``custom_metadata``. Both may be present, such as
+for an extension type whose storage type is a dictionary of extension
+values. Writers should omit the ``DictionaryEncoding``'s ``custom_metadata``
+when the dictionary value type has no metadata.
+
+When reading a dictionary-encoded field, the dictionary value type is
+reconstructed from the ``Field``'s data type and the ``DictionaryEncoding``'s
+``custom_metadata``. The result is wrapped in the dictionary type given by
+the ``DictionaryEncoding``, and then the ``Field``'s ``custom_metadata`` is
+applied to that dictionary type. As for other extension types, a reader
+that does not recognize the extension name in the ``DictionaryEncoding``'s
+``custom_metadata`` uses the dictionary value's storage type.
+
+Readers that predate this ``custom_metadata`` field (added in format version
+1.6) ignore it and read a dictionary of the storage type. The field is
+additive: the ``MetadataVersion`` remains ``V5``, and the IPC file footer
+and Arrow Flight, which embed the same ``Schema``, need no other change.
+
+.. note::
+   Before format version 1.6, a dictionary whose value type is an extension
+   type could not be distinguished from an extension type whose storage type
+   is a dictionary, as only the ``Field``'s ``custom_metadata`` was available.
+
 .. seealso::
    :ref:`format_canonical_extensions`
 

docs/source/format/Integration.rst

@@ -198,14 +198,19 @@ present if there are dictionary type fields in the schema.
       "dictionary": {
         "id": /* integer */,
         "indexType": /* Type */,
-        "isOrdered": /* boolean */
+        "isOrdered": /* boolean */,
+        "metadata" : /* Metadata */
       },
       "metadata" : /* Metadata */
     }
 
 The ``dictionary`` attribute is present if and only if the ``Field`` corresponds to a
 dictionary type, and its ``id`` maps onto a column in the ``DictionaryBatch``. In this
-case the ``type`` attribute describes the value type of the dictionary.
+case the ``type`` attribute describes the value type of the dictionary, and the
+optional ``metadata`` attribute of ``dictionary`` holds the custom metadata of the
+dictionary value type, as the ``custom_metadata`` of ``DictionaryEncoding`` does in
+the IPC format (see :ref:`format_metadata_extension_types_dictionary`). It may be
+omitted when the dictionary value type has no metadata.
 
 For primitive types, ``children`` is an empty array.
 
@@ -424,6 +429,34 @@ would be represented::
       ]
     }
 
+A dictionary whose value type is an extension type carries the extension
+metadata in the ``metadata`` attribute of its ``dictionary`` object instead of
+the field's ``metadata``. For example, a dictionary of ``arrow.uuid`` values
+with ``int8`` indices would be represented as::
+
+    {
+      "name" : "name_of_the_field",
+      "nullable" : /* boolean */,
+      "type" : {
+        "name" : "fixedsizebinary",
+        "byteWidth" : 16
+      },
+      "children" : [],
+      "dictionary": {
+        "id": 0,
+        "indexType": {
+          "name": "int",
+          "bitWidth": 8,
+          "isSigned": true
+        },
+        "isOrdered": false,
+        "metadata" : [
+          {"key": "ARROW:extension:name", "value": "arrow.uuid"},
+          {"key": "ARROW:extension:metadata", "value": ""}
+        ]
+      }
+    }
+
 **RecordBatch**::
 
     {

docs/source/format/Versioning.rst

@@ -76,7 +76,7 @@ that new clients can read serialized data produced by library version
 Post-1.0.0 Format Versions
 ==========================
 
-Since version 1.0.0, there have been five new minor versions and zero new
+Since version 1.0.0, there have been six new minor versions and zero new
 major versions of the Arrow format. Each new minor version added new features.
 When these new features are not used, the new minor format versions are
 compatible with format version 1.0.0. The new features added in each minor
@@ -110,3 +110,10 @@ Version 1.5
 -----------
 
 * Expanded Decimal type bit widths to allow 32-bit and 64-bit types.
+
+Version 1.6
+-----------
+
+* Added ``custom_metadata`` to ``DictionaryEncoding``, carrying the custom
+  metadata of the dictionary value type, such as an extension type (see
+  :ref:`format_metadata_extension_types_dictionary`).

format/Schema.fbs

@@ -25,6 +25,7 @@
 /// Version 1.4 - Add BinaryView, Utf8View, variadicBufferCounts, ListView, and
 /// LargeListView.
 /// Version 1.5 - Add 32-bit and 64-bit as allowed bit widths for Decimal
+/// Version 1.6 - Add custom_metadata to DictionaryEncoding.
 
 namespace org.apache.arrow.flatbuf;
 
@@ -511,6 +512,9 @@ table DictionaryEncoding {
   isOrdered: bool;
 
   dictionaryKind: DictionaryKind;
+
+  /// Metadata for the dictionary value type, e.g. the extension type of the values
+  custom_metadata: [ KeyValue ];
 }
 
 /// ----------------------------------------------------------------------