Migrate MeanEncoder.fit() to narwhals, add polars support - #1027
Open
solegalli wants to merge 2 commits into
Open
Migrate MeanEncoder.fit() to narwhals, add polars support#1027solegalli wants to merge 2 commits into
solegalli wants to merge 2 commits into
Conversation
fit() computes, per variable, the mean of y per category (and, with
smoothing="auto", the target variance per category), blended with the
overall target mean via a weight that increases with category count.
transform() and inverse_transform() already came dataframe-agnostic
for free from CategoricalMethodsMixin (base_encoder.py, merged
separately).
Benchmarked a pure-narwhals fit() (group_by/agg for count+mean(+var))
against pandas-native (value_counts + groupby) at 10k-100k rows x
1-10 cols x 5-50 categories: narwhals-on-pandas ran ~1.5x-2.9x slower,
worst at the most common shape (1-2 columns, 50k-100k rows), crossing
the ~1.7x real-loss threshold; narwhals-on-polars was competitive to
faster than pandas-native throughout. Per the benchmark-driven
merge-vs-split rule, and matching what the OrdinalEncoder sibling
migration found for the same y-groupby-by-category shape of fit(),
this splits on `is_pandas = nwd.is_pandas_dataframe(X)`: pandas keeps
a close variant of its original value_counts/groupby code, while
polars (and other narwhals backends) goes through group_by()/agg().
Bug fixed (pre-existing, confirmed against the unmodified file): the
old fit() always called `y.groupby(X[var])`, which raises
AttributeError whenever y is a numpy array rather than a Series -
e.g. list/array-like y input, which sklearn's check_X_y machinery
converts to numpy. This is the exact same bug the OrdinalEncoder
sibling found and fixed in its own fit(). Confirmed failing against
the unmodified file (tests/test_encoding/test_mean_encoder.py::
test_inverse_transform_when_no_unseen, ::test_inverse_transform_when_
ignore_unseen, ::test_inverse_transform_when_encode_unseen, plus
test_check_estimator_encoders.py::test_encoders_when_x_pandas_y_numpy
[encoder1] for MeanEncoder) and now passing. Fixed on the pandas
branch by pairing X[var] with y via `.assign()` when y isn't a Series
(aligns a numpy y positionally, matching how `y.groupby(X[var])`
aligned a Series y by index), and on the narwhals branch via
`nw.new_series` for a numpy y. Unlike OrdinalEncoder, no cross-backend
tie-break fix was needed: MeanEncoder's encoder_dict_ is a
category-to-target-mean mapping (a dict), not a rank-ordered list, so
backend-dependent group order doesn't affect the result - verified
pandas and polars produce identical dicts across smoothing=0.0/100/
"auto" and all three `unseen` settings.
Rewrote every test in test_mean_encoder.py as one
@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) case
per behavior (41 tests, up from 20), using a narwhals-based, NaN-aware
comparison helper; y is passed as a plain list in most tests, which
also exercises the numpy-y bug fix on every parametrized case.
test_variables_cast_as_category stays pandas-only - it exercises
pandas Categorical dtype, which polars has no direct equivalent for.
Verified: tests/test_encoding/test_mean_encoder.py 41 passed (was 20,
3 failing). tests/test_encoding full suite: 345 passed, 13 failed -
same failing test IDs as the unmodified base minus the 4 MeanEncoder-
specific ones fixed here (unmodified base: 17 failed/326 passed);
remaining 13 are pre-existing and unrelated (numpy-X rejection per the
narwhals check_X() contract, affecting every encoder; OrdinalEncoder's
and WoEEncoder's own unmigrated fit() bugs on other in-progress
branches). flake8 and mypy clean. Module imports with pandas blocked
(verified in isolation from unmigrated sibling modules in the
encoding package, which still import pandas on this per-file
migration branch). sphinx -W build clean (only the pre-existing
linkcode_resolve warning). Verified the class docstring example and
every code example in docs/user_guide/encoding/MeanEncoder.rst that
doesn't require the Titanic dataset against real output, and added a
"With polars" section verified the same way; the Titanic-dataset
examples could not be re-run in this sandbox (no network access to
openml.org) but are untouched by this change. Downstream consumers
(feature_engine/_prediction/base_predictor.py and
target_mean_selection.py, which construct MeanEncoder internally)
verified via their test suites: 66 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
force-pushed
the
narwhals-mean-encoder
branch
from
August 30, 2026 22:46
710a8de to
edf4980
Compare
check_X_y now returns a narwhals frame, so bind that to nw_X and keep the original native X for _check_or_select_variables, _check_na, _get_feature_names_in and the nwd.is_pandas_dataframe(X) fast-path check (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). The pandas value_counts/groupby fast path is unchanged - X stays native so no rehydration is needed. The narwhals branch reuses nw_X from check_X_y instead of nw.from_native(X). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
force-pushed
the
narwhals-mean-encoder
branch
from
August 30, 2026 22:51
edf4980 to
d9fbf78
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates
MeanEncoder.fit()to narwhals with polars support.transform()/inverse_transform()already come dataframe-agnostic fromCategoricalMethodsMixin.fit()computes, per variable, the mean ofyper category (and, withsmoothing="auto", the target variance per category), blended with the overall target mean.Merge vs split: benchmarked a pure-narwhals
fit()(group_by/agg) vs pandas-native (10k–100k rows × 1–10 cols × 5–50 categories). narwhals-on-pandas ran ~1.5x–2.9x slower, worst at the most common shape (1–2 cols, 50k–100k rows), crossing the ~1.7x real-loss threshold; narwhals-on-polars competitive to faster. So this splits onis_pandas = nwd.is_pandas_dataframe(X): pandas keeps a close variant of itsvalue_counts/groupbycode, polars/other backends go throughgroup_by()/agg(). No cross-backend tie-break fix needed here —encoder_dict_is a category→mean dict, not a rank-ordered list (verified identical dicts pandas vs polars acrosssmoothing=0.0/100/"auto"and all threeunseensettings).Bug fixed (pre-existing, confirmed against the unmodified file): the old
fit()always calledy.groupby(X[var]), raisingAttributeErrorwhenyis a numpy array (list/array-likey, as sklearn'scheck_X_yproduces) — same bug theOrdinalEncodersibling found. Fixed on the pandas branch by pairingX[var]withyvia.assign()whenyisn't a Series (positional alignment, matching how a Seriesyaligned by index), and on the narwhals branch vianw.new_series. Fixestest_inverse_transform_when_*(3) andtest_encoders_when_x_pandas_y_numpy[encoder1].Tests: every test in
test_mean_encoder.pyrewritten as onemake_df in [pd.DataFrame, pl.DataFrame]case per behaviour (41, up from 20),ypassed as a plain list in most so the numpy-yfix is exercised throughout.test_variables_cast_as_categorystays pandas-only.Verified:
test_mean_encoder.py41 passed; fulltests/test_encoding345 passed / 13 failed — same IDs as the unmodified base minus the 4 fixed here. flake8 / mypy clean, sphinx -W clean. Docstring +MeanEncoder.rstnon-Titanic examples verified against real output, "With polars" section added (Titanic examples untouched — no network in sandbox). Downstream_prediction/base_predictor.pyandtarget_mean_selection.pytest suites: 66 passed.Stacked on #999 (
narwhals-encoding-base). Until that merges this PR's diff also contains the sharedCategoricalMethodsMixincommit; review #999 first.