[GH-3053] Add Flink 2.2 support by joshmanis618-stack · Pull Request #3211 · apache/sedona

joshmanis618-stack · 2026-07-29T17:10:26Z

Summary

Fixes the two API breaks that block Sedona's flink module from compiling/running against Flink 2.x, and adds an opt-in sedona-flink-2.2 profile so both lines are tested in CI.

Kryo registration: ExecutionConfig.registerTypeWithKryoSerializer was removed in Flink 2.x; registration moved to SerializerConfigImpl via getSerializerConfig().
TypeSerializerSnapshot signature change: resolveSchemaCompatibility now takes a TypeSerializerSnapshot<T> instead of a TypeSerializer<T>. Fixed across all six type serializers (Geometry, Geometry[], Box2D, Box3D, Geography, and a new GeometryDoublePairTypeSerializer).
ST_MinimumBoundingRadius: its bare @DataTypeHint(value = "RAW") relied on Flink auto-deriving a Kryo serializer for Pair<Geometry, Double>, which throws under Flink 2.2.1. Added an explicit GeometryDoublePairTypeSerializer, matching how every other RAW hint in Functions.java already avoids auto-derivation.
Added a sedona-flink-2.2 Maven profile (-Dflink=2.2 -Dscala=2.12), mirroring the existing sedona-spark-* version-profile convention, pinning flink.version to 2.2.1.
Added a Flink 2.2.1 row to the java.yml CI matrix and updated docs/setup/flink/platform.md.

Why this doesn't need to wait for a breaking/2.0 release

When I filed #3053 I expected this to require migrating off legacy SourceFunction/SinkFunction (removed in Flink 2.0) and moving to unsuffixed Flink 2.x table artifacts. Neither turned out to be necessary in practice:

The flink module doesn't use SourceFunction/SinkFunction anywhere.
flink-table-planner_2.12 is still published for Flink 2.2.1, so the existing Scala-suffixed artifact naming keeps working unchanged.

The actual break was narrower: just the two API changes above. This PR is purely additive — Flink 1.19 remains the default and is unaffected; -Dflink=2.2 -Dflink.version=2.2.1 is opt-in. Since no existing behavior changes, I don't think this needs to wait for a breaking Sedona 2.0 release — happy to target the upcoming 1.9.1 if that works for you.

Testing

Full flink/flink-shaded test suite (331 tests, 0 failures/errors) passes against both:

Default Flink 1.19.0
-Dflink=2.2 -Dscala=2.12 -Dflink.version=2.2.1

mvn spotless:apply run before submitting.

Closes #3053

- Fix ExecutionConfig.registerTypeWithKryoSerializer removal (Flink 2.x moved Kryo registration to SerializerConfigImpl via getSerializerConfig()) - Fix TypeSerializerSnapshot.resolveSchemaCompatibility signature change (now takes a TypeSerializerSnapshot instead of a TypeSerializer) Both fixes compile against Flink 1.19.0 and 2.2.1 with no version branching.

…nk 2.2 - Add sedona-flink-2.2 Maven profile (-Dflink=2.2 -Dscala=2.12), mirroring the existing sedona-spark-* version-profile convention - Fix ST_MinimumBoundingRadius: its bare @DataTypeHint(value = "RAW") return type relied on Flink auto-deriving a Kryo serializer for Pair<Geometry, Double>, which throws a ValidationException under Flink 2.2.1. Added GeometryDoublePairTypeSerializer and wired it in as an explicit rawSerializer, matching how every other RAW hint in this file already avoids Flink's automatic derivation. Verified: full flink/flink-shaded test suite (262 tests) passes against both Flink 1.19.0 (default) and 2.2.1 (-Dflink=2.2 -Dscala=2.12). examples/flink-sql also builds and passes against 2.2.1 unchanged.

- Add a Flink 2.2.1 row to the java.yml CI matrix (spark 3.5.8, scala 2.12.15, jdk 17), so the -Dflink=2.2 path is actually exercised in CI instead of only the untouched 1.19 default - Update docs/setup/flink/platform.md to note Flink 2.2 support and the Java 11/17/21 requirement Flink 2.x itself imposes

joshmanis618-stack · 2026-07-29T22:52:28Z

I did not touch the component that failed in this test, and it built fully on my local machine.

jiayuasu · 2026-07-30T07:24:40Z

Thanks. I will look into the PR today

jiayuasu

Thanks for adding Flink 2.2 support. I think we should target this change for Sedona 2.0.0 and make Flink 1.19 the minimum supported version starting with that release. Flink 1.18 and earlier are no longer supported by the Apache Flink community, so Sedona 2.0.0 is a reasonable boundary for removing compatibility with those versions.

Before merging, please make the following changes:

Remove the sedona-flink-2.2 Maven profile. Unlike the Spark profiles, it only assigns flink.version; there are no version-specific modules or source shims. Builds can use -Dflink.version=2.2.1 directly.
Avoid casting getSerializerConfig() to the internal SerializerConfigImpl. With Flink 1.19 as the minimum, we can consistently use the public pipeline.serialization-config API introduced in Flink 1.19.
Update the platform and Maven-coordinate documentation to state that Sedona 2.0.0 requires Flink 1.19 or later. Please also record this minimum-version change in the Sedona 2.0.0 release notes.
Add skipLibPostalTests: 'true' to the new Flink matrix row. Otherwise it downloads and runs the unrelated 1.3 GB libpostal suite in a second matrix combination.
Document the checkpoint/savepoint compatibility impact of changing ST_MinimumBoundingRadius from a Kryo-backed RAW type to GeometryDoublePairTypeSerializer. Fresh stateless queries work, but stateful Table/SQL jobs containing this result cannot restore state written with the previous serializer. Since this is targeting Sedona 2.0.0, the compatibility break can be documented as part of the major release.

I will handle broader Flink-version coverage and separating the Sedona Spark and Sedona Flink CI workflows in a follow-up.

I verified that the profile-free Flink 2.2.1 build passes all 333 Flink tests with:

mvn -pl flink,flink-shaded -am test \
  -Dscala=2.12 -Dflink.version=2.2.1

…a-flink-2.2 profile Per review feedback: since Flink 1.19 is now the minimum supported version (targeting Sedona 2.0.0), we can consistently use the public pipeline.serialization-config API (FLIP-398, Flink 1.19+) instead of casting getSerializerConfig() to the internal SerializerConfigImpl. Kryo registrations now go through SerializerConfig.configure(ReadableConfig, ClassLoader) on the public interface. SedonaFlinkRegistrator.registerType now delegates to the shared SedonaContext.registerGeometryKryoSerializers instead of duplicating the registration list. Also removes the sedona-flink-2.2 Maven profile: it only ever set flink.version with no version-specific modules or source shims, so -Dflink.version=2.2.1 works directly without it. Adds a note on ST_MinimumBoundingRadius's checkpoint/savepoint compatibility impact from switching off the auto-derived Kryo serializer.

…9+ minimum - Drop the -Dflink=<compat> profile flag from the CI script now that the sedona-flink-2.2 profile is gone; pass -Dflink.version directly. - Add skipLibPostalTests: 'true' to the new Flink 2.2.1 matrix row so it doesn't redundantly download the 1.3 GB libpostal suite. - Update platform.md and maven-coordinates.md to state Flink 1.19 as the minimum supported version (Flink 1.12 - 1.18 no longer supported). - Record both breaking changes (Flink 1.19+ minimum, ST_MinimumBoundingRadius checkpoint/savepoint compatibility) in the Sedona 2.0.0 release notes.

…ter sync)

joshmanis618-stack · 2026-07-31T20:48:33Z

I think I’ve addressed your request

jiayuasu

Please address the inline comment.

jiayuasu · 2026-08-02T06:57:26Z

+    Configuration configuration = new Configuration();
+    configuration.set(PipelineOptions.SERIALIZATION_CONFIG, kryoRegistrations);
+    env.getConfig()
+        .getSerializerConfig()


ExecutionConfig.getSerializerConfig() is annotated @Internal in both Flink 1.19 and 2.2, so this still depends on an internal API even though the SerializerConfigImpl cast is gone. Please apply the configuration through the public environment API instead:

env.configure(configuration);

This also uses the environment’s user classloader rather than the current thread’s context classloader. I tested this exact replacement locally against Flink 1.19 and 2.2.1; ModuleTest passed all 7 tests with both versions.

…erConfig getConfig().getSerializerConfig() is annotated @internal in both Flink 1.19 and 2.2, so calling configure() on it still depended on an internal API even without the SerializerConfigImpl cast. Switch to StreamExecutionEnvironment.configure(ReadableConfig), which is public and uses the environment's own user classloader rather than the calling thread's context classloader.

….1 release + master sync)

ST_Collect_Aggr/ST_Collect_Agg's accumulator used a bare @DataTypeHint(value = "RAW", bridgedTo = Accumulators.AccGeometryCollection.class) with no rawSerializer, relying on Flink's automatic Kryo-serializer derivation for the accumulator state — the same failure mode already fixed elsewhere in this PR for ST_MinimumBoundingRadius, just surfaced by ST_Collect_Agg landing on master after this branch was first rebased. Added AccGeometryCollectionTypeSerializer and wired it in as an explicit rawSerializer in all four affected @FunctionHint annotations. No compatibility-break note needed for this one: ST_Collect_Agg hasn't shipped in any release yet, so there's no prior on-disk state to worry about.

joshmanis618-stack added 3 commits July 29, 2026 12:03

joshmanis618-stack requested a review from jiayuasu as a code owner July 29, 2026 17:10

Merge branch 'master' into flink-2.2-support

70e4f3b

jiayuasu added this to the sedona-2.0.0 milestone Jul 30, 2026

jiayuasu reviewed Jul 30, 2026

View reviewed changes

joshmanis618-stack added 3 commits July 30, 2026 22:35

Merge remote-tracking branch 'myfork/flink-2.2-support' (picks up mas…

f371927

…ter sync)

jiayuasu requested changes Aug 2, 2026

View reviewed changes

jiayuasu and others added 4 commits August 2, 2026 00:08

Merge master and resolve release notes

803ed23

Merge remote-tracking branch 'myfork/flink-2.2-support' (picks up 1.9…

57b29f6

….1 release + master sync)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[GH-3053] Add Flink 2.2 support - #3211

[GH-3053] Add Flink 2.2 support#3211
joshmanis618-stack wants to merge 11 commits into
apache:masterfrom
joshmanis618-stack:flink-2.2-support

joshmanis618-stack commented Jul 29, 2026

Uh oh!

joshmanis618-stack commented Jul 29, 2026

Uh oh!

jiayuasu commented Jul 30, 2026

Uh oh!

jiayuasu left a comment

Uh oh!

joshmanis618-stack commented Jul 31, 2026 •

edited

Loading

Uh oh!

jiayuasu left a comment

Uh oh!

jiayuasu Aug 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joshmanis618-stack commented Jul 29, 2026

Summary

Why this doesn't need to wait for a breaking/2.0 release

Testing

Uh oh!

joshmanis618-stack commented Jul 29, 2026

Uh oh!

jiayuasu commented Jul 30, 2026

Uh oh!

jiayuasu left a comment

Choose a reason for hiding this comment

Uh oh!

joshmanis618-stack commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiayuasu left a comment

Choose a reason for hiding this comment

Uh oh!

jiayuasu Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshmanis618-stack commented Jul 31, 2026 •

edited

Loading