Skip to content

[AMORO-4379][Iceberg] Fix OS-dependent File.separator in table URI path resolution - #4380

Open
aakashofficial-k01 wants to merge 1 commit into
apache:masterfrom
aakashofficial-k01:AMORO-4379-fix-iceberg-uri-path-separator
Open

aakashofficial-k01 wants to merge 1 commit into
apache:masterfrom
aakashofficial-k01:AMORO-4379-fix-iceberg-uri-path-separator

Conversation

@aakashofficial-k01

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Table locations and file paths in Iceberg are URIs (RFC 3986) where path components are delimited by /. In amoro-format-iceberg, three locations constructed paths using the host OS File.separator:

  1. TableFileUtil.getNewFilePath()
  2. IcebergTableMaintainer.clearInternalTableContentsFiles()
  3. IcebergTableMaintainer.clearInternalTableMetadata()

On Windows environments, File.separator resolves to \. This produces invalid URIs (such as hdfs://nn:8020/tbl\00000-0.parquet or s3://bucket/tbl\metadata), which causes:

  • TableFileUtil.getUriPath() / URI.create() to throw IllegalArgumentException: Illegal character in path during table optimize commit (UnKeyedTableCommit.moveTargetFiles()).
  • Cloud object store prefix queries (pio.listPrefix()) to treat \ as a literal ASCII character, returning 0 files and silently skipping orphan metadata cleanup.
  • Redundant double slashes (//) on Linux when a base table location contains a trailing slash.

Fix #4379.

Brief change log

  • In TableFileUtil.getNewFilePath(), use Iceberg's standard LocationUtil.stripTrailingSlash(newDirectory) and format child file paths with forward slash /.
  • In IcebergTableMaintainer, construct dataLocation and metadataLocation using LocationUtil.stripTrailingSlash(table.location()) and format with / adhering to Iceberg's DefaultLocationProvider convention.
  • Remove unused import java.io.File; in both classes.
  • Add parameterized unit tests in TestFileUtil verifying getNewFilePath with and without trailing slashes across S3 and HDFS URIs.

How was this patch tested?

  • Added unit tests for getNewFilePath in TestFileUtil.java.
    ./mvnw test -pl amoro-common,amoro-format-iceberg -Dtest=TestFileUtil: BUILD SUCCESS (14/14 passed)
  • Validated Spotless and Checkstyle:
    ./mvnw spotless:check checkstyle:check -pl amoro-format-iceberg: BUILD SUCCESS (0 violations)
  • Screenshots are not applicable.

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Fix OS-dependent File.separator in Iceberg table URI path resolution

1 participant