Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 59 additions & 16 deletions docs/user_guide/outliers/OutlierTrimmer.rst
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ In the following output, we see the maximum of the variables after removing the
.. code:: python

fare 65.0
age 53.0
age 74.0
dtype: float64

Finally, we can check the boxplot of the transformed variables to corroborate the effect on their distribution.
Expand Down Expand Up @@ -521,7 +521,7 @@ We see the adjusted data size compared to the original size here:

.. code:: python

((916, 8), (736, 76))
((916, 8), (828, 142))

Feature-engine's pipeline can also adjust the target:

Expand All @@ -535,7 +535,7 @@ We see the adjusted data size compared to the original size here:

.. code:: python

((916,), (736,))
((916,), (828,))

To wrap up, let's add a machine learning algorithm to the pipeline. We'll use logistic regression to predict survival:

Expand Down Expand Up @@ -565,7 +565,7 @@ We see the following output:

.. code:: python

array([1, 1, 1, 0, 1, 0, 1, 1, 0, 1], dtype=int64)
array([1, 1, 0, 1, 0, 1, 0, 0, 1, 0])

We can obtain the probability of survival:

Expand All @@ -580,16 +580,16 @@ We see the following output:

.. code:: python

array([[0.13027536, 0.86972464],
[0.14982143, 0.85017857],
[0.2783799 , 0.7216201 ],
[0.86907159, 0.13092841],
[0.31794531, 0.68205469],
[0.86905145, 0.13094855],
[0.1396715 , 0.8603285 ],
[0.48403632, 0.51596368],
[0.6299007 , 0.3700993 ],
[0.49712853, 0.50287147]])
array([[0.23320943, 0.76679057],
[0.22089305, 0.77910695],
[0.85469885, 0.14530115],
[0.28510312, 0.71489688],
[0.85468117, 0.14531883],
[0.0494853 , 0.9505147 ],
[0.58079146, 0.41920854],
[0.536129 , 0.463871 ],
[0.36885157, 0.63114843],
[0.81102131, 0.18897869]])

We can obtain the accuracy of the predictions over the test set:

Expand All @@ -601,7 +601,7 @@ That returns the following accuracy:

.. code:: python

0.7823343848580442
0.804093567251462

We can obtain the names of the features after the transformation:

Expand Down Expand Up @@ -635,7 +635,7 @@ We see the resulting sizes here:

.. code:: python

((393, 8), (317, 76))
((393, 8), (342, 142))


Setting up the stringency (param `fold`)
Expand All @@ -656,6 +656,49 @@ The default values for fold are as follows:
You can manually adjust the fold value to make the outlier detection process more or less
conservative, thus customising the extent of outlier trimming.

With polars
-----------

:class:`OutlierTrimmer()` works in the same way with a polars dataframe:

.. code:: python

import polars as pl
from feature_engine.outliers import OutlierTrimmer

df = pl.DataFrame({
"Age": [20, 21, 19, 18, 95],
"Marks": [0.9, 0.8, 0.7, 0.6, 0.1],
})

transformer = OutlierTrimmer(
capping_method="quantiles",
tail="both",
fold=0.2,
)

print(transformer.fit_transform(df))

Only the rows where both `Age` and `Marks` fall within the 20th-80th
percentile range survive; the other three rows breach the bound on at
least one of the two variables:

.. code:: text

shape: (2, 2)
┌─────┬───────┐
│ Age ┆ Marks │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═══════╡
│ 21 ┆ 0.8 │
│ 19 ┆ 0.7 │
└─────┴───────┘

`transform_x_y()` and `get_feature_names_out()` work identically to the
pandas examples above.


Additional resources
--------------------

Expand Down
21 changes: 20 additions & 1 deletion feature_engine/_base_transformers/mixins.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,26 @@ def transform_x_y(self, X: IntoDataFrame, y: IntoSeries):
else:
row_index_col = "__feature_engine_row_index__"
nw_X = nw.from_native(X, eager_only=True).with_row_index(row_index_col)
X = self.transform(nw_X.to_native())
# Some transform() implementations (e.g. BaseOutlier/BaseImputer)
# validate X's column count/names against feature_names_in_/
# n_features_in_, which would reject row_index_col - widen both
# just for this call, when present, so the marker survives.
has_feature_names_in = hasattr(self, "feature_names_in_")
if has_feature_names_in is True:
original_features_in: List[
Union[str, int]
] = self.feature_names_in_ # type: ignore[has-type]
original_n_features_in: int = (
self.n_features_in_ # type: ignore[has-type]
)
self.feature_names_in_ = original_features_in + [row_index_col]
self.n_features_in_ = original_n_features_in + 1
try:
X = self.transform(nw_X.to_native())
finally:
if has_feature_names_in is True:
self.feature_names_in_ = original_features_in
self.n_features_in_ = original_n_features_in
nw_X = nw.from_native(X, eager_only=True)
row_positions = nw_X.get_column(row_index_col)
X = nw_X.drop(row_index_col).to_native()
Expand Down
Loading