_read_deletes builds its result with a dict comprehension that filters the whole table once per distinct file_path:
{
file.as_py(): table.filter(pc.field("file_path") == file).column("pos")
for file in table.column("file_path").chunks[0].dictionary
}
Cost is therefore O(rows x distinct paths). A delete file with many distinct paths makes this quadratic. The ORC branch below has the same shape via unique().
Issue investigation generated via claude, reviewed by Sung, Kevin, Fokko.
_read_deletesbuilds its result with a dict comprehension that filters the whole table once per distinctfile_path:{ file.as_py(): table.filter(pc.field("file_path") == file).column("pos") for file in table.column("file_path").chunks[0].dictionary }Cost is therefore O(rows x distinct paths). A delete file with many distinct paths makes this quadratic. The ORC branch below has the same shape via
unique().Issue investigation generated via claude, reviewed by Sung, Kevin, Fokko.