Core: Read and write geometry and geography values in Avro#17119
Open
huan233usc wants to merge 1 commit into
Open
Core: Read and write geometry and geography values in Avro#17119huan233usc wants to merge 1 commit into
huan233usc wants to merge 1 commit into
Conversation
wombatu-kun
reviewed
Jul 7, 2026
| } | ||
|
|
||
| @Override | ||
| protected boolean supportsGeospatial() { |
Contributor
There was a problem hiding this comment.
TestAvroEncoderUtil extends DataTestBase and round-trips generic records through this same TypeToSchema + RandomAvroData path via AvroEncoderUtil.encode/decode, and on main it mirrors this test's type flags exactly (supportsUnknown, supportsTimestampNanos, supportsVariant). Enabling supportsGeospatial() there as well covers geo through the encoder entry point for free and keeps the two generic-model Avro tests in parity. szehon-ho raised the same coverage-parity point on the sibling Parquet PR #16982.
500ea8d to
7b2cc6d
Compare
TypeToSchema mapped Iceberg types to Avro schema for every primitive except geometry and geography, so converting a schema with a geo column threw UnsupportedOperationException before any value could be read or written. Map both to an Avro bytes field carrying WKB, per the Avro type mapping in the spec (binary, geometry, and geography all use bytes). The value read/write paths dispatch on the Avro physical type, so once the schema is bytes the existing byteBuffers reader/writer handle geo unchanged in both the generic and internal object models (geo values are WKB ByteBuffers, like binary). Enable the shared DataTest geospatial coverage for the Avro object models (TestGenericAvro, TestInternalAvro, generic data TestGenericData) and add the GEOMETRY/GEOGRAPHY -> ByteBuffer handling to RandomAvroData and AvroTestHelpers that the generic Avro path needs, mirroring RandomInternalData/InternalTestHelpers.
7b2cc6d to
67619a7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TypeToSchemamapped every Iceberg primitive to an Avro schema exceptgeometryandgeography,so converting a schema with a geo column threw
UnsupportedOperationException: Unsupported type ID: GEOMETRYbefore any value could be read or written through the Avro object model. This wires up thevalue path.
Per the Avro type mapping in the spec,
geometryandgeographyare stored as an Avrobytesfieldcarrying WKB — the same representation as
binary:binarybytesgeometrybytes(WKB)geographybytes(WKB)So
TypeToSchema.primitive()now maps both geo types to the existingBINARY_SCHEMA. The valueread/write paths dispatch on the Avro physical type, so once the schema is
bytesthe existingbyteBuffersreader/writer handle geo unchanged in both the generic and internal object models (geovalues are WKB
ByteBuffers, exactly likebinary) — no value-path code changes are needed.This is a follow-up split out of #16982 (which added the Parquet WKB value path and deliberately left
the Avro object model as a follow-up).
Note on the reverse direction
SchemaToTypeintentionally still maps Avrobytes → binary. Plain Avrobytescarries no marker todistinguish geometry/geography from binary, and Iceberg resolves the real column type from the
expected/table schema, never by reverse-inferring from the file's Avro schema — the same way
binaryand other
bytes-backed types already behave.Test plan
DataTestgeospatial coverage (supportsGeospatial()) for the Avro object models:TestGenericAvro,TestInternalAvro, and the generic-dataTestGenericData. This round-tripsgeometry and geography across multiple CRS and edge algorithms with randomly generated WKB values.
GEOMETRY/GEOGRAPHY→ByteBufferhandling that the generic Avro path needs inRandomAvroDataandAvroTestHelpers, mirroring the existingRandomInternalData/InternalTestHelpershandling../gradlew :iceberg-core:test --tests org.apache.iceberg.avro.TestGenericAvro --tests org.apache.iceberg.avro.TestInternalAvro./gradlew :iceberg-data:test --tests org.apache.iceberg.data.avro.TestGenericData