Repository navigation
[C++] Improve error message when reading Streaming file with File reader and vice versa #33823
Description
Describe the bug, including details regarding any error messages, version, and platform.
I am trying to loan an arrow file
with pa.memory_map('flights-200k.arrow', 'r') as source:
my_arrow = pa.ipc.open_file(source).read_all()but get this error
File "/opt/homebrew/Caskroom/miniforge/base/envs/ramsch/lib/python3.10/site-packages/pyarrow/ipc.py", line 228, in open_file
return RecordBatchFileReader(
File "/opt/homebrew/Caskroom/miniforge/base/envs/ramsch/lib/python3.10/site-packages/pyarrow/ipc.py", line 110, in __init__
self._open(source, footer_offset=footer_offset,
File "pyarrow/ipc.pxi", line 862, in pyarrow.lib._RecordBatchFileReader._open
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not an Arrow file
The arrow file is https://github.com/uwdata/flights-arrow/blob/master/flights-200k.arrow and loads fine in the arrow js library.
Component(s)
Python
Activity
domoritz commentedon Jan 21, 2023
The issue is that I need to use
open_stream. The error message should be better.[-]pyarrow.lib.ArrowInvalid: Not an Arrow file[/-][+][Python] pyarrow.lib.ArrowInvalid: Not an Arrow file[/+]vibhatha commentedon Jan 24, 2023
I am not sure if this is related. I had a similar experience when I have mistakenly written files but haven't closed the file writer. In your case since it is loaded in JS properly, this could be an entirely different scenario. But thought it is worth mentioning here.
jorisvandenbossche commentedon Jan 24, 2023
Small reproducer without having to download a file:
I agree it would be nice we can give a more informative error message and hint the user they are reading a Arrow Streaming format file and not a Arrow File format file.
[-][Python] pyarrow.lib.ArrowInvalid: Not an Arrow file[/-][+][C++] Improve error message when reading Streaming file with File reader and vice versa[/+]jorisvandenbossche commentedon Jan 24, 2023
Similarly, reading a File with a Streaming reader also gives a non-informative error message:
westonpace commentedon Jan 24, 2023
Files written in the file format have a magic number on both sides of the data. The error message "Not an Arrow file" is thrown when that magic number is wrong. So we already detect this situation, we just need to be proactive about suggesting solutions / alternatives (e.g. "Not an Arrow file, perhaps this is in the streaming format?") so this should be very doable.
domoritz commentedon Jan 25, 2023
I am an arrow committer and got totally thrown off by the error message and thought my file was corrupt. So yes, your suggested error message sounds great.
jalajk24 commentedon Jan 27, 2023
@domoritz i would like to contribute in this project can you assign this project to me
domoritz commentedon Jan 27, 2023
I assigned it to you. Please send a pull request soon.
pbaner16 commentedon Dec 8, 2023
Hello @domoritz -- has this issue been fixed? If not, i can contribute!
RobertLD commentedon Apr 15, 2026
This looks to be very easy change and I've been trolling the issue board to find a good first contribution to arrow.
I'd love to post a PR if @jalajk24 is no longer working on this (I assume so it's been some years
kou commentedon Apr 15, 2026
You can work on this!
apache#33823 Improve err msgs when opening files that are the wrong f…
apache#33823 Improve err msgs when opening files that are the wrong f…
RobertLD commentedon Apr 19, 2026
@kou PR is posted for your review!
GH-33823: [C++][IPC] Improve error messages when opening files that a…
pitrou commentedon Oct 29, 2026
Issue resolved by pull request 49771
#49771