Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ public, discoverable protocol.
## Why PreTab?

- **Familiar interface.** A scikit-learn `fit`/`transform`/`fit_transform` API that drops
into existing pipelines and works with both `pandas.DataFrame` and `numpy.ndarray`
inputs.
into existing pipelines and works with `pandas.DataFrame`, `polars.DataFrame` and
`numpy.ndarray` inputs.
- **Automatic feature handling.** Feature-type detection and per-feature strategies let you
describe intent once instead of wiring transformers by hand.
- **Beyond scaling.** Spline bases, neural basis maps, piecewise-linear encoding, and
Expand Down Expand Up @@ -80,8 +80,8 @@ print(X.shape)
# (100, 22)
```

> **Note:** PreTab accepts a `pandas.DataFrame` or a `numpy.ndarray` and infers numerical
> versus categorical columns either way.
> **Note:** PreTab accepts a `pandas.DataFrame`, a `polars.DataFrame` or a `numpy.ndarray` and
> infers numerical versus categorical columns either way.

> **Tip:** Swap the global methods for a `feature_preprocessing` map, for example
> `{"age": "ple", "income": "rbf", "city": "one-hot"}`, and PreTab fits each column with its
Expand Down
27 changes: 26 additions & 1 deletion docs/core_concepts/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ example `{"feature_0": "rbf"}`. Check `numerical_features_` / `categorical_featu
overrides, rather than guessing the column order.
```

At `transform`, array columns are matched by position, so the training and serving
At `transform`, an array or a list of rows is matched by position, so the training and serving
containers may differ, as in scikit-learn. A `Preprocessor` fitted on a string-labelled
`DataFrame` accepts the same data as an array of the same width (for example `df.to_numpy()`
in NumPy-based serving or an explainer such as `shap.KernelExplainer`). It warns that `X`
Expand All @@ -64,6 +64,31 @@ the fitted labels are literally `feature_0`, `feature_1`, .... Input matched by
with a different number of columns raises the usual
`X has 4 features, but Preprocessor is expecting 3 features as input.` error.

## Polars DataFrame input

A `polars.DataFrame` is accepted as input to the `Preprocessor` and `RepresentationSearchCV`
wherever a pandas `DataFrame` is (temporal columns are not supported), so a scikit-learn
`Pipeline` configured with `set_output(transform="polars")` can hand its polars output straight
to the `Preprocessor`. PreTab reads the frame through pandas with the same column names, order
and dtypes, then detects column types as usual: `Float*` columns are numerical, `Int*` /
`UInt*` columns follow the integer cardinality rule (an integer column with nulls becomes
float, as in pandas), and `String` (`Utf8` on older Polars), `Categorical`, `Enum` and `Boolean` columns are
categorical. Polars nulls become missing values, so imputation and `missing_policy` treat them
like `NaN` in a pandas frame. The conversion does not need `pyarrow`.

```python
import polars as pl
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

X = pl.DataFrame({"age": [25.0, None, 47.0, 33.0], "income": [3.1e4, 5.2e4, None, 4.4e4]})
y = [0.0, 1.0, 1.0, 0.0]

pipe = Pipeline([("impute", SimpleImputer()), ("pretab", Preprocessor(numerical_method="minmax"))])
pipe.set_output(transform="polars")
pipe.fit_transform(X, y) # a polars.DataFrame with columns num_age, num_income
```

## Per-feature overrides

Columns rarely want identical treatment. The `feature_preprocessing` dict assigns a strategy
Expand Down
6 changes: 4 additions & 2 deletions docs/getting_started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,12 @@ pip install "pretab[polars]"
```

```{note}
`polars` is only needed for the `set_output(transform="polars")` output path; every other
`polars` is only needed for polars DataFrame input or the `set_output(transform="polars")` output path; every other
output (`output_structure="matrix"`/`"blocks"`, `output_format="dense"`/`"sparse"`,
`set_output(transform="pandas")`) works without it. Requesting `"polars"` output without the
extra installed raises a clear `OptionalDependencyError`.
extra installed raises a clear `OptionalDependencyError`. A `polars.DataFrame` is also accepted
as input to `fit` / `transform` (see
[Polars DataFrame input](../core_concepts/configuration.md#polars-dataframe-input)).
```

Use the convenience `all` extra to install every optional dependency at once:
Expand Down
4 changes: 2 additions & 2 deletions docs/homepage.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ compose standalone representations with scikit-learn transformers and estimators
:::

:::{grid-item-card} 🧠 Smart defaults
Automatic feature-type detection (numerical vs. categorical) with support for both
`pandas.DataFrame` and `numpy.ndarray` inputs.
Automatic feature-type detection (numerical vs. categorical) with support for
`pandas.DataFrame`, `polars.DataFrame` and `numpy.ndarray` inputs.
:::

::::
Expand Down
41 changes: 34 additions & 7 deletions pretab/compose/feature_detection.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Coerce inputs to DataFrames and classify columns as numerical or categorical.
"""Coerce inputs to pandas DataFrames and classify columns as numerical or categorical.

Feature-type detection decides which construction path each column takes. It is
kept here, separate from orchestration, so the Preprocessor's ``fit`` reads as a
Expand All @@ -10,27 +10,54 @@
import numpy as np
import pandas as pd

from ..core.validation import is_polars_frame, polars_to_pandas
from ..exceptions import PretabDataError, invalid_param_error

__all__ = ["bool_columns_as_object", "detect_column_types", "to_dataframe", "with_string_labels"]
__all__ = ["bool_columns_as_object", "detect_column_types", "has_column_labels", "to_dataframe", "with_string_labels"]


def has_column_labels(X) -> bool:
"""Return True if ``X`` carries its own column labels.

DataFrames (pandas or polars) and dicts of columns are matched by label;
anything else (a NumPy array, a list of rows, ...) is matched by position.
"""
return isinstance(X, dict) or hasattr(X, "columns")


def to_dataframe(X, *, copy: bool = False) -> pd.DataFrame:
"""Return ``X`` as a DataFrame, naming array columns ``feature_0``, ``feature_1`` ....
"""Return ``X`` as a pandas DataFrame, naming array columns ``feature_0``, ``feature_1`` ....

Dicts and NumPy arrays are wrapped in a fresh DataFrame; an existing
DataFrame is returned as-is, or copied when ``copy`` is True.
Dicts are wrapped in a fresh DataFrame and a polars DataFrame is converted to
one with the same column names, order and (as far as possible) dtypes. Input
without column labels -- a NumPy array, a list of rows -- is read as a 2D
array. An existing pandas DataFrame is returned as-is, or copied when
``copy`` is True.

Raises
------
PretabDataError
If the resulting columns contain a duplicate label. A
If ``X`` is a pandas Series or array-like input that is not 2D, or if the
resulting columns contain a duplicate label. A
:class:`~sklearn.compose.ColumnTransformer` keys its per-column steps by
name, so a duplicate cannot be routed unambiguously.
"""
if isinstance(X, pd.Series):
raise PretabDataError(
"Expected 2D input (a DataFrame or a 2D array), got a pandas Series.\n"
"Fix: pass X.to_frame() to use the Series as a single column."
)
if isinstance(X, dict):
X = pd.DataFrame(X)
elif isinstance(X, np.ndarray):
elif is_polars_frame(X):
X = polars_to_pandas(X)
elif not has_column_labels(X):
X = np.asarray(X)
if X.ndim != 2:
raise PretabDataError(
f"Expected 2D input (a DataFrame or a 2D array), got an array with {X.ndim} dimension(s).\n"
"Fix: reshape a single feature to one column, e.g. X.reshape(-1, 1)."
)
X = pd.DataFrame(X, columns=pd.Index([f"feature_{i}" for i in range(X.shape[1])]))
else:
X = X.copy() if copy else X
Expand Down
12 changes: 11 additions & 1 deletion pretab/compose/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from sklearn.utils.validation import check_is_fitted

from ..core._typing import PredictorLike
from ..core.validation import is_polars_frame, polars_to_pandas
from ..exceptions import InvalidParamError
from ..preprocessor import Preprocessor

Expand Down Expand Up @@ -80,13 +81,22 @@ def _make_preprocessor(self, method):
return Preprocessor(numerical_method=method, **params)

def fit(self, X, y=None):
"""Search over ``methods`` and refit the best representation on all data."""
"""Search over ``methods`` and refit the best representation on all data.

``X`` may be a pandas or polars DataFrame or an array. A polars frame is
converted to pandas once, before the fold rows are taken, so every fold's
Preprocessor detects the same column types as the final refit.
"""
methods = list(self.methods)
if not methods:
raise InvalidParamError("methods must be a non-empty sequence of numerical methods.")
if y is None:
raise InvalidParamError("RepresentationSearchCV requires y at fit time; got y=None.")
y_arr = np.asarray(y).ravel()
if is_polars_frame(X):
# Converted once, so every fold's rows carry the dtypes of the whole frame,
# as a pandas frame's do (see polars_to_pandas).
X = polars_to_pandas(X)
n_samples = X.shape[0] if hasattr(X, "shape") else len(X)
cv = cast(BaseCrossValidator, check_cv(self.cv, y_arr, classifier=is_classifier(self.estimator)))
# Reuse the same held-out rows for every candidate, including splitters
Expand Down
45 changes: 44 additions & 1 deletion pretab/core/validation.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,58 @@
``transform`` -- and so ``n_features_in_`` is recorded consistently.
"""

import sys
import warnings
from typing import Literal

import numpy as np
import pandas as pd
from sklearn.utils.validation import check_array

from ..exceptions import DataWarning, PretabDataError

__all__ = ["validate_2d_allow_nan"]
__all__ = ["is_polars_frame", "polars_to_pandas", "validate_2d_allow_nan"]


def is_polars_frame(X) -> bool:
"""Return True if ``X`` is a polars DataFrame, without importing polars.

A polars object can only exist once polars has been imported, so when the
module is absent from ``sys.modules`` ``X`` cannot be a polars frame.
"""
polars = sys.modules.get("polars")
return polars is not None and isinstance(X, polars.DataFrame)


def polars_to_pandas(X) -> pd.DataFrame:
"""Convert a polars DataFrame to pandas, keeping column names, order and dtypes.

``polars.DataFrame.to_pandas`` requires pyarrow, which neither polars nor
PreTab depends on, so each column goes through ``Series.to_numpy`` instead:
integer, unsigned, float, boolean and temporal columns keep their NumPy
dtype (an integer column with nulls becomes float with NaN, as in pandas),
while string, categorical and enum columns -- and boolean columns with nulls
-- become ``object`` columns. Their nulls arrive as ``None`` and are mapped
to ``NaN``, the missing marker the imputers and missing indicators recognize.
Categorical and enum columns are first cast to strings so the native NumPy
export also works on older Polars releases.

Convert a frame once and take row subsets of the result: converted on its
own, a subset can get another dtype (an integer column with a null is float
as a whole, but int in a subset without the null).
"""
polars = sys.modules["polars"]
columns = {}
for column in X.get_columns():
if column.dtype in (polars.Categorical, polars.Enum):
column = column.cast(str)
values = column.to_numpy()
if column.dtype == polars.Boolean and column.null_count() == 0:
values = values.astype(bool, copy=False)
if values.dtype == object:
values = np.where(pd.isna(values), np.nan, values)
columns[column.name] = values
return pd.DataFrame(columns)


def validate_2d_allow_nan(X, *, allow_nan: bool = True, reset: bool, estimator):
Expand Down
36 changes: 23 additions & 13 deletions pretab/preprocessor.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,13 @@

from .compose.config import PreprocessorConfig
from .compose.factory import build_column_transformer
from .compose.feature_detection import bool_columns_as_object, detect_column_types, to_dataframe, with_string_labels
from .compose.feature_detection import (
bool_columns_as_object,
detect_column_types,
has_column_labels,
to_dataframe,
with_string_labels,
)
from .compose.inspection import (
block_name,
build_feature_info,
Expand Down Expand Up @@ -128,7 +134,8 @@ class Preprocessor(TransformerMixin, BaseEstimator):
--------
- Supports a wide range of preprocessing methods for numerical and categorical features.
- Automatically detects feature types (numerical vs. categorical).
- Compatible with both pandas DataFrames and NumPy arrays.
- Compatible with pandas and polars DataFrames and NumPy arrays (a polars frame is read
through pandas, so it gets the same column-type detection).
- Handles external embedding arrays for models that require learned representations.
- Returns either a dictionary of transformed feature blocks or a single NumPy array.
- Fully compatible with scikit-learn transformers and pipelines.
Expand Down Expand Up @@ -511,8 +518,11 @@ def fit(self, X, y=None, embeddings=None):

Parameters
----------
X : pandas.DataFrame, numpy.ndarray, or dict
The input features.
X : pandas.DataFrame, polars.DataFrame, numpy.ndarray, or dict
The input features. A polars frame is converted to pandas with the same column
names, order and dtypes (no ``pyarrow`` needed); an array (or a list of rows) has
its columns named ``feature_0``, ``feature_1``, ....
A frame or dict is matched by column name, an array (or a list of rows) by position.
y : array-like, default=None
Target values (used for decision tree-based methods).
embeddings : np.ndarray or list of np.ndarray, optional
Expand Down Expand Up @@ -574,7 +584,7 @@ def fit(self, X, y=None, embeddings=None):
verbose=resolved["verbose"],
)

fitted_on_array = isinstance(X, np.ndarray)
fitted_on_array = not has_column_labels(X)
X = to_dataframe(X)

if self.feature_preprocessing:
Expand Down Expand Up @@ -710,9 +720,9 @@ def transform(self, X, embeddings=None, return_array=None):

Parameters
----------
X : pandas.DataFrame, numpy.ndarray, or dict
X : pandas.DataFrame, polars.DataFrame, numpy.ndarray, or dict
Input features to transform. A DataFrame or dict is matched to the fitted
columns by label, integer labels included. A NumPy array is matched by
columns by label, integer labels included. An array (or a list of rows) is matched by
position: after a fit on string-labelled columns it takes those names
(numeric columns of an object array are
re-inferred), with scikit-learn's usual warning that ``X`` does not have
Expand Down Expand Up @@ -795,7 +805,7 @@ def fit_transform(self, X, y=None, embeddings=None, return_array=None):

Parameters
----------
X : pandas.DataFrame, numpy.ndarray, or dict
X : pandas.DataFrame, polars.DataFrame, numpy.ndarray, or dict
Input features.
y : array-like, optional
Target values.
Expand Down Expand Up @@ -889,7 +899,7 @@ def get_feature_names_out(self, input_features=None):
Input feature names, following scikit-learn's convention. When the
preprocessor was fitted on named (string-labelled) columns they must equal
``feature_names_in_``. When it was fitted without feature names (a NumPy
array or a frame with non-string column labels), any
array, a list of rows or a frame with non-string column labels), any
``n_features_in_`` names are accepted and used in place of the fitted
ones, so names propagate through a ``Pipeline`` whose upstream steps
output arrays. Non-string labels are converted to strings in the output
Expand Down Expand Up @@ -982,7 +992,7 @@ def output_dims_(self) -> dict:
def _align_input(self, X):
"""Return ``X`` as a DataFrame whose columns line up with the fitted columns.

A NumPy array is matched by position. Once the preprocessor was fitted on
An array (or a list of rows) is matched by position. Once the preprocessor was fitted on
named columns, an array of the fitted width takes those column labels
(numeric columns of an object array are re-inferred), and a DataFrame
passed to a preprocessor fitted on an array is matched by position too --
Expand All @@ -1005,7 +1015,7 @@ def _align_input(self, X):
not has_feature_names and fitted_labels is not None and list(fitted_labels) == positional_labels
)

is_array = isinstance(X, np.ndarray)
is_array = not has_column_labels(X)
X = to_dataframe(X, copy=True)
if not (is_array or (fitted_on_array and list(X.columns) != positional_labels)):
return X
Expand Down Expand Up @@ -1070,7 +1080,7 @@ def estimate_output_shape(self, X) -> tuple:

Parameters
----------
X : pandas.DataFrame, numpy.ndarray, or dict
X : pandas.DataFrame, polars.DataFrame, numpy.ndarray, or dict
Input whose row count drives the estimate; not transformed.

Returns
Expand All @@ -1090,7 +1100,7 @@ def estimate_memory(self, X) -> int:

Parameters
----------
X : pandas.DataFrame, numpy.ndarray, or dict
X : pandas.DataFrame, polars.DataFrame, numpy.ndarray, or dict
Input whose row count drives the estimate; not transformed.

Returns
Expand Down
Loading
Loading