Repository navigation
[Python] MapArray pandas round trip is broken #31965
Description
pyarrow.MapArray when converted to pandas cannot be successfully converted back.
The following snipper does not work:
import pyarrow as pa
data = [[('x', 1), ('y', 0)], [('a', 2), ('b', 45)]]
ty = pa.map_(pa.string(), pa.int64())
map_col = pa.array(data, type=ty)
pa.MapArray.from_pandas(map_col.to_pandas())
Uncaught exception: ArrowTypeError: Expected bytes, got a 'int' object (java.lang.RuntimeException)
Reporter: Robbie Gruener / @rgruener
Note: This issue was originally created as ARROW-16611. Please see the migration documentation for further details.
Activity
asfimport commentedon May 18, 2022
Antoine Pitrou / @pitrou:
Hmm, where does the "java.lang.RuntimeException" come from?
asfimport commentedon May 18, 2022
Antoine Pitrou / @pitrou:
Also, did it work with previous versions of PyArrow? I don't think so.
asfimport commentedon May 31, 2022
Joris Van den Bossche / @jorisvandenbossche:
If you specify the type again, then it works:
Also small note that the
from_pandasmethod is a generic Array class method, and not specific toMapArray. So what is actually being done under the hood ispa.array(map_col.to_pandas(), from_pandas=True)(which also illustrates why you need to specify thetypeagain).In general it is difficult to infer a map type (i.e. inferring the type from the data when not specifying an exact type). A list of tuples can for example also be the representation for a struct array.
thisisnic commentedon Oct 26, 2026
@jorisvandenbossche I'm going through old
Type: bugissues. You explained here thatMapArray.from_pandas()is just the genericArray.from_pandas(), so a map type cannot be inferred from a list of tuples and passingtype=pa.map_(pa.string(), pa.int64())makes the round trip work; the reporter never followed up. Are you happy for me to close this as answered?