khiops.sklearn : text variables not used

[DTRGit]

Description

Text variables are not used in khiops.sklearn.
I noticed this issue when looking at the genarated .khj report file where the "text preparation" tab is absent wheras I have string columns with values of more than 100 characters.

Description

With pandas 3.0, string columns using the new default str dtype are
inferred as Categorical instead of Text by khiops.sklearn.

Consequently, text features are not generated and the Text preparation
section is absent from the generated .khj report, even when values are
longer than 100 characters.

Setting KhiopsClassifier(n_text_features=10000) does not change the result,
because the source variable has already been inferred as Categorical.

Minimal reproduction

import pandas as pd
from khiops.sklearn.dataset import PandasTable

texts = ["short", "x" * 101, "y" * 180]

for dtype in ("str", "string"):
    dataframe = pd.DataFrame({
        "message": pd.Series(texts, dtype=dtype),
    })

    table = PandasTable("Table", dataframe)
    variable = table.create_khiops_dictionary().get_variable("message")

    print(
        f"dtype={dtype!r}, "
        f"pandas dtype={dataframe['message'].dtype}, "
        f"max length={dataframe['message'].str.len().max()}, "
        f"Khiops type={variable.type}"
    )

Output:

dtype='str', pandas dtype=str, max length=180, Khiops type=Categorical
dtype='string', pandas dtype=string, max length=180, Khiops type=Text

The behavior was also reproduced with a complete
KhiopsClassifier.fit() call and confirmed in the generated .khj report.

Expected behavior

Both pandas string dtype variants should be inferred as Khiops Text when
the maximum value length is greater than 100 characters:

dtype='str'    -> Text
dtype='string' -> Text

Actual behavior

dtype='str'    -> Categorical
dtype='string' -> Text

Probable cause

In khiops/sklearn/dataset.py, get_khiops_type converts the dtype to a
lowercase string and only recognizes this exact value:

elif lower_numpy_type == "string":

With pandas 3.0, the default string dtype is represented as str, although
it is also a pandas.StringDtype. Its maximum length is computed by
PandasTable, but get_khiops_type then falls through to Categorical.

The test could probably cover both str and string representations, or
the mapper could use the dtype class instead of its string representation.

Workaround

Explicitly converting the affected columns to the pandas nullable string
dtype enables text preparation:

text_columns = [
    "motifs_messaging_1m",
    "motifs_appels_traites_1m",
    "motif_detaille_btq",
]

dataframe[text_columns] = dataframe[text_columns].astype("string")

Environment

• Python 3.13.2
• pandas 3.0.5
• khiops 11.0.1.0
• Khiops engine 11.0.1
• Debian GNU/Linux 12

Context

• Khiops version

Khiops 11.0.1

Environment variables:
KHIOPS_API_MODE true
KHIOPS_DRIVERS_PATH /home/david_trehin_orange_com/atelier_khiops/.venv/lib
Internal environment variables:
None
Temporary file directory: /tmp
System resources:
Host number 1
Physical cores on system 8
Logical processes on system 8
Available memory on system 102.2 GB (Logical 63.9 GB)
Available disk space on system 193.5 GB
hostname MPI ranks logical memory disk cores
vm-ic3-gce 0,1,2,3,4,5,6,7 63.9 GB 193.5 GB 8
System:
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"