[AMORO-4377][AMS] Decouple the maintenance Iceberg version from mixed-format connectors and move AMS to Iceberg 1.10.2

[j1wonpark]

Why are the changes needed?

Close #4377.

This follows up on the Iceberg upgrade discussion: let the Iceberg version used by the self-optimizing / maintenance path evolve independently from the mixed-format connectors, so that a maintenance-side upgrade no longer forces engine drops on connector users.

It keeps every module and class where it is. AMS, the optimizers and dist select Iceberg 1.10.2; amoro-format-iceberg, amoro-mixed-hive and all engine connectors stay on 1.8.1.

The code is ready for review. I am opening it as a draft because one point needs a community decision (Hadoop 2 support for AMS, see below); once that is settled I will mark it ready.

Brief change log

• Root pom.xml: new maintenance-iceberg.version (1.10.2) and maintenance-parquet.version (1.16.0) properties. iceberg.version stays 1.8.1.
• Root pom.xml: the connector-side Parquet version is now explicit as well. parquet-avro.version becomes parquet.version (still 1.15.2) and the root manages all six Parquet artifacts with it, so the two sides read as iceberg.version / parquet.version vs. maintenance-iceberg.version / maintenance-parquet.version. On the connector side parquet-jackson keeps its separate 1.13.1 pin; on the maintenance side it is managed at 1.16.0 like the rest of the family. Verified with dependency:list before and after: shipped artifacts (connector runtime jars, Trino) resolve exactly as before; the only change is that amoro-format-paimon and amoro-mixed-spark-3-common previously resolved a mix of 1.13.1 / 1.15.0 / 1.15.2 and now resolve 1.15.2 consistently.
• amoro-ams, amoro-optimizer, dist: set iceberg.version to the maintenance version and manage the Iceberg artifacts the root does not pin (iceberg-api, iceberg-bundled-guava, Spark binding and extensions) plus the Parquet family. The Parquet alignment is required: Iceberg 1.10.x needs a newer Parquet than the one resolved through 1.8.1, and this shows up as NoSuchMethodError at runtime, not at build time.
• Two Parquet reader calls that Iceberg removed in 1.9 (ParquetValueReaders.StructReader(types, readers) and ParquetValueReader.setPageSource(pages, rowPosition)) are switched to the form that exists in both 1.8.1 and 1.10.x, in amoro-mixed-hive and the Spark 3.4 / 3.5 readers that AMS loads for its local terminal. The shared modules are still compiled against 1.8.1; the same bytecode runs on both versions.
• New TestMaintenanceIcebergCompatibility in amoro-ams writes a Parquet file and reads it back through the mixed-Hive generic reader (AdaptHiveParquet / AdaptHiveGenericParquetReaders) and the Spark 3.5 reader (SparkParquetReaders) on the AMS classpath, i.e. against Iceberg 1.10.2. Without the reader change above both fail with the NoSuchMethodError; with it, both pass. It needs no metastore, so it does not interact with the shared test HMS of the other AMS tests.
• TestIcebergV3OptimizingCommit: one V2 fixture wrote three records to replace two. Iceberg 1.10 validates that a REPLACE commit does not add more records than it replaces (Invalid REPLACE operation: 3 added records > 2 replaced records), so the fixture now keeps the input row count. Amoro's rewrite outputs never add records, so no production code is affected.
• CI: the Spark 3.3 matrix entry and the Hadoop 2 job now build connectors and optimizers only (-pl '!amoro-ams,!dist'), see open questions below. The Hadoop 2 docker image builds dist with -Phadoop2,spark-3.5.
• dev/deps: dependency lists regenerated with dev/dependencies.sh --replace. Besides Iceberg 1.10.2 and Parquet 1.16.0, the maintenance classpath picks up parquet-variant, jts-core, and transitive bumps of aircompressor (0.27 -> 2.0.3), avro (1.12.0 -> 1.12.1) and httpclient5 / httpcore5 (5.4.1 / 5.3.1 -> 5.5 / 5.3.4). The Hadoop 2 list is now dependencies-hadoop-2-spark-3.5 because the Hadoop 2 dist build uses the Spark 3.5 profile.
• LICENSE-binary: jts-core is a new transitive dependency of Parquet 1.16 (EDL 1.0). The other new or bumped artifacts are already covered.

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

• Same shared bytecode (amoro-common, amoro-format-iceberg, amoro-mixed-hive, compiled with JDK 11 against Iceberg 1.8.1) run with the existing I/O tests on both an Iceberg 1.8.1 and an Iceberg 1.10.2 classpath: 85 passed / 15 assumption-skipped / 0 failed on each side. Class hashes verified identical between the two runs.

• On the AMS classpath (Iceberg 1.10.2, JDK 11): TestKeyedContinuousOptimizing and TestUnKeyedContinuousOptimizing (12/12, native Iceberg, mixed Iceberg and mixed Hive), TestSnapshotExpire* and TestOrphanFileClean* (116 passed / 28 assumption-skipped / 0 failed).

• Local Spark 3.5 with the mixed, Iceberg and Paimon extensions registered together (the terminal's combination): create native Iceberg table, insert, aggregate.

• Packaged artifacts inspected with a small probe on JDK 11: amoro-ams-dependency/lib, Flink and Spark optimizer jars load Iceberg 1.10.2 (no duplicate org/apache/iceberg classes); mixed Flink 1.18 / 1.19 / 1.20 and Spark 3.5 runtime jars load relocated Iceberg 1.8.1.

• Reactor builds: default (Spark 3.5), -Pspark-3.4, -Pspark-3.3 (connectors and optimizers only), -Phadoop2 (connectors and optimizers only).

Not covered: end-to-end run with an external optimizer process, real terminal API / mixed catalog SQL, Trino, full dist tarball run.

Decision needed

Hadoop 2 support for AMS. Parquet 1.16 calls FileSystem.openFile(Path), which does not exist in Hadoop 2.10, so with -Phadoop2 the AMS I/O tests fail with NoSuchMethodError. There is no workaround on our side short of keeping AMS on the old Parquet, which defeats the upgrade.

My proposal: AMS requires Hadoop 3 from the next release; the mixed-format connectors keep their Hadoop 2 build. This PR already implements that on the CI side (the Hadoop 2 job builds connectors and optimizers only). What still needs to follow if we agree: drop the Hadoop 2 dist / docker image (it currently still compiles with -Phadoop2,spark-3.5 but AMS cannot read Parquet in it) and note the change in the release notes. If anyone still runs AMS on Hadoop 2, please speak up.

Notes and limitations

1. Spark 3.3. iceberg-spark-3.3 was last published for Iceberg 1.8.1, so AMS can no longer be built with -Pspark-3.3. The default AMS build is Spark 3.5 and is unaffected; the Spark 3.3 mixed-format connector is still built and published, just not in the same reactor as AMS.
2. AMS and optimizers must share one Iceberg version. Task descriptors cross the AMS / optimizer boundary through Java serialization, and DataFile serialization is not compatible between 1.8.1 and 1.9+ (serialVersionUID). This PR sets both from the same property.
3. Compile boundary. The shared modules are compiled against 1.8.1, so code that needs 1.9+ API (row lineage, MetadataColumns.ROW_ID, ...) cannot live in amoro-format-iceberg as it is today. New V3-specific maintenance code needs a location that compiles against the maintenance version. This PR does not decide that; it only shows that the version split itself does not require moving modules first.
4. Parquet 1.16.0 is the version Iceberg 1.10.2 declares. I have not tried to find the minimum Parquet version that works.

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable
• README build section updated: Hadoop 2 and Spark 3.3 connector build commands.