Pandas v3 issue with new str dtype

[liquidcarbon]

What happens?

Highlights of pandas 3.0

...
Dedicated string data type by default: string columns are now inferred as the new str dtype instead of object, providing better performance and type safety

When you query a pandas dataframe containing new str dtypes via DuckDB, you get

_duckdb.NotImplementedException: Not implemented Error: Data type 'str' not recognized

To Reproduce

import duckdb

duckdb_rel = duckdb.sql("FROM 'https://raw.githubusercontent.com/liquidcarbon/chembiodata/main/isotopes.csv'")
pandas3_df = duckdb_rel.df()

duckdb.sql("FROM duckdb_rel")  # works
duckdb.sql("FROM pandas3_df")  # fails

OS:

any

DuckDB Version:

1.4.3

DuckDB Client:

python

Hardware:

No response

Full Name:

Alex

Affiliation:

self

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

• Yes, I have

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant data sets for reproducing the issue?

Yes

[rustyconover]

I think this is because in pandas version 3 strings are now "large" arrow strings, meaning they have a 64 bit length rather than 32 bit length.

Rusty

[evertlammerts]

Can you test with 1.4.4? This should be fixed.

[liquidcarbon]

Indeed!

uv run pandas3duck144.py

# /// script
# dependencies = [
#   "pandas>=3",
#   "duckdb==1.4.4",
# ]
# ///

import duckdb

duckdb_rel = duckdb.sql(
    "FROM 'https://raw.githubusercontent.com/liquidcarbon/chembiodata/main/isotopes.csv'"
)
pandas3_df = duckdb_rel.df()

duckdb.sql("FROM duckdb_rel")  # works
duckdb.sql("FROM pandas3_df")  # works on 1.4.4