Skip to content

[Bug] Parquet reader fails on BLOB restored from ARROW:schema #296

Closed

Description

@zjw1111

Search before asking

  • I searched in the issues and found nothing similar.

Paimon-cpp version

main at commit ded7cfb8.

Minimal reproduce step

  1. Use pypaimon==2.0.0 with pyarrow==19.0.1 to write an append-only Paimon table containing a BLOB column.
  2. The generated Parquet file stores the physical column as binary, while its ARROW:schema metadata records the original Arrow type as large_binary.
  3. Read the table with Paimon C++. The Arrow Parquet reader restores the column to large_binary before Paimon applies its physical binary-to-BLOB conversion.

A reproducing fixture is included in test/test_data/parquet/append_types_compatibility.db/python_types.

What doesn't meet your expectations?

The read fails because Paimon C++ strictly compares the restored large_binary source type with the expected physical binary type. These representations are compatible for a Paimon BLOB that has already been restored by ARROW:schema.

Paimon C++ should accept the already-restored large_binary BLOB while continuing to reject unrelated type mismatches.

Anything else?

The compatibility fixtures also cover equivalent tables written by Python, Rust, and Java Paimon. The proposed fix is available in #297.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

  1. changed the title [-]Parquet reader fails on BLOB types restored from ARROW:schema[/-] [+]Parquet reader fails on BLOB restored from ARROW:schema[/+] on Sep 9, 2026
  2. changed the title [-]Parquet reader fails on BLOB restored from ARROW:schema[/-] [+][Bug] Parquet reader fails on BLOB restored from ARROW:schema[/+] on Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions