Skip to content

fix: return empty file metadata tables for snapshots without manifests - #4272

Open
YuangGao wants to merge 1 commit into
apache:mainfrom
YuangGao:fix/inspect-files-without-manifests
Open

YuangGao wants to merge 1 commit into
apache:mainfrom
YuangGao:fix/inspect-files-without-manifests

Conversation

@YuangGao

@YuangGao YuangGao commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Closes #4269

Rationale for this change

  • Appending an empty table creates a snapshot with no manifests, and _files / _all_files then call pa.concat_tables([]), which raises ArrowInvalid.
  • Both now return an empty table with the files schema when there are no manifests to read.

Are these changes tested?

  • test_inspect.py — files, data_files, delete_files, all_files, all_data_files and all_delete_files on a snapshot without manifests.

Are there any user-facing changes?

  • These metadata tables return an empty table instead of raising.

@oresttokovenko

Copy link
Copy Markdown
Contributor

@YuangGao What if a table already has data and someone overwrites it with an empty table (using the overwrite() method in pyiceberg/table/__init__.py): files() comes back empty, but all_files() still returns the old data file, is that what you'd expect? I added a diagram to help visualize this:

the situation you're merging a fix for

   flowchart TD
       A["no snapshot"] --> B["append(0 rows)"]
       B --> C["snapshot with 0 manifests"]
       C --> D["inspect.files() / all_files()"]
       D --> E["results = []<br/>unique_manifests = {}"]
       E --> F{"guard?"}
       F -- "before: none" --> G["ArrowInvalid"]
       F -- "after: not empty" --> H["empty_table(), 0 rows"]
Loading

what the new logic can introduce

flowchart TD
    A["append 3 rows"] --> B["snapshot 1: APPEND<br/>1 manifest, 1 data file"]
    B --> C["overwrite with an empty table"]
    C --> D["snapshot 2: DELETE<br/>1 manifest, 0 entries"]
    C --> E["snapshot 3: APPEND<br/>0 manifests"]
    D --> F["current snapshot = 3"]
    E --> F
    F --> G["files(): 0 rows"]
    F --> H["all_files(): manifests from snapshots 1 and 2<br/>unique_manifests = 2"]
    H -->|"guard on all snapshots"| I["1 row, history kept"]
    H -->|"guard on current snapshot only"| J["0 rows, history lost<br/>13 tests pass"]
Loading

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

inspect.files() and the other file metadata tables raise ArrowInvalid when a snapshot has no manifests

2 participants