Skip to content

Core: Read and write geometry and geography values in Avro#17119

Open
huan233usc wants to merge 1 commit into
apache:mainfrom
huan233usc:geo-avro-schema
Open

Core: Read and write geometry and geography values in Avro#17119
huan233usc wants to merge 1 commit into
apache:mainfrom
huan233usc:geo-avro-schema

Conversation

@huan233usc

Copy link
Copy Markdown
Contributor

Summary

TypeToSchema mapped every Iceberg primitive to an Avro schema except geometry and geography,
so converting a schema with a geo column threw UnsupportedOperationException: Unsupported type ID: GEOMETRY before any value could be read or written through the Avro object model. This wires up the
value path.

Per the Avro type mapping in the spec, geometry and geography are stored as an Avro bytes field
carrying WKB — the same representation as binary:

Iceberg type Avro type
binary bytes
geometry bytes (WKB)
geography bytes (WKB)

So TypeToSchema.primitive() now maps both geo types to the existing BINARY_SCHEMA. The value
read/write paths dispatch on the Avro physical type, so once the schema is bytes the existing
byteBuffers reader/writer handle geo unchanged in both the generic and internal object models (geo
values are WKB ByteBuffers, exactly like binary) — no value-path code changes are needed.

This is a follow-up split out of #16982 (which added the Parquet WKB value path and deliberately left
the Avro object model as a follow-up).

Note on the reverse direction

SchemaToType intentionally still maps Avro bytes → binary. Plain Avro bytes carries no marker to
distinguish geometry/geography from binary, and Iceberg resolves the real column type from the
expected/table schema, never by reverse-inferring from the file's Avro schema — the same way binary
and other bytes-backed types already behave.

Test plan

  • Enable the shared DataTest geospatial coverage (supportsGeospatial()) for the Avro object models:
    TestGenericAvro, TestInternalAvro, and the generic-data TestGenericData. This round-trips
    geometry and geography across multiple CRS and edge algorithms with randomly generated WKB values.
  • Add the GEOMETRY/GEOGRAPHYByteBuffer handling that the generic Avro path needs in
    RandomAvroData and AvroTestHelpers, mirroring the existing RandomInternalData /
    InternalTestHelpers handling.
  • ./gradlew :iceberg-core:test --tests org.apache.iceberg.avro.TestGenericAvro --tests org.apache.iceberg.avro.TestInternalAvro
  • ./gradlew :iceberg-data:test --tests org.apache.iceberg.data.avro.TestGenericData

}

@Override
protected boolean supportsGeospatial() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestAvroEncoderUtil extends DataTestBase and round-trips generic records through this same TypeToSchema + RandomAvroData path via AvroEncoderUtil.encode/decode, and on main it mirrors this test's type flags exactly (supportsUnknown, supportsTimestampNanos, supportsVariant). Enabling supportsGeospatial() there as well covers geo through the encoder entry point for free and keeps the two generic-model Avro tests in parity. szehon-ho raised the same coverage-parity point on the sibling Parquet PR #16982.

TypeToSchema mapped Iceberg types to Avro schema for every primitive except
geometry and geography, so converting a schema with a geo column threw
UnsupportedOperationException before any value could be read or written. Map
both to an Avro bytes field carrying WKB, per the Avro type mapping in the
spec (binary, geometry, and geography all use bytes).

The value read/write paths dispatch on the Avro physical type, so once the
schema is bytes the existing byteBuffers reader/writer handle geo unchanged in
both the generic and internal object models (geo values are WKB ByteBuffers,
like binary).

Enable the shared DataTest geospatial coverage for the Avro object models
(TestGenericAvro, TestInternalAvro, generic data TestGenericData) and add the
GEOMETRY/GEOGRAPHY -> ByteBuffer handling to RandomAvroData and AvroTestHelpers
that the generic Avro path needs, mirroring RandomInternalData/InternalTestHelpers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants