[Python] Table.to_pandas() fails for dictionary encoded columns with an is_null partition_expression #32225

Minimal steps to reproduce:

I attached a .zip file containing a python script and a test parquet file. Running this python script reproduces the issue.

The steps taken to reproduce:

1. Create a test parquet file with one column containing only null.

2. Create a parquet fragment from this file adding a partition_expression with an is_null guarantee on this fragment.

3. Create a FileSystemDataset from this fragment setting the schema to be a dictionary column.

4. Call .to_table().to_pandas() on the resulting pyarrow dataset. You will get the following error.

     File "/.../pip-core_pandas/pandas/core/dtypes/dtypes.py", line 492, in validate_categories
       raise ValueError("Categorical categories cannot be null")
   ValueError: Categorical categories cannot be null 

    

   My understanding of why this doesn't work:

5. There are 2 ways of dictionary encoding nulls: mask and encode described in the pyarrow docs. Pyarrow supports both but pandas categoricals only supports mask. Arguably the real issue here is pandas should support encode style categoricals.

6. When you provide an .is_null guarantee on a fragment arrow will not actually read the data. It knows the type from the schema, we've guaranteed the values are all null and it can get the length from the parquet metadata so it has everything it needs.

7. Instead of reading the data it uses the Null ArrayFactory. For dictionary type columns I believe that calls this DictionaryArray constructor which appears to be creating the dictionary in the encode style.

   Would it be possible to make this configurable? It seems like the mask style of dictionary encoding is the default for the rest of PyArrow and it would solve the Pandas compatibility issue. I appreciate this is probably an extremely niche issue but my options for a workaround are looking pretty horrible. 

Environment: Ubuntu 18.04, PyArrow 8.0.0, Pandas 1.4.3
Reporter: Thomas Newton

Original Issue Attachments:

• reproduce_null_dictionary_issue.zip

_{Note: This issue was originally created as ARROW-16905. Please see the migration documentation for further details.}