Migrate RareLabelEncoder to narwhals, add polars support - #1030
Open
solegalli wants to merge 2 commits into
Open
Migrate RareLabelEncoder to narwhals, add polars support#1030solegalli wants to merge 2 commits into
solegalli wants to merge 2 commits into
Conversation
fit() replaces pandas .unique()/.value_counts(normalize=True) with
narwhals Series.n_unique() (for the cardinality check - matches pandas'
plain unique() length, which counts a null as its own category, unlike
pandas' nunique() which drops it) and drop_nulls().value_counts(sort=True,
normalize=True) (same drop_nulls()/sort=True reasoning as
CountEncoder.fit(): narwhals' value_counts() has no dropna param and
narwhals' own value_counts default is unsorted).
transform() doesn't reuse CategoricalMethodsMixin._encode() (that's a
dict-based numeric remap; this encoder keeps frequent categories as-is
and only replaces the rest), so it's rewritten from pandas'
.loc[~isin(...), feature] = replace_with onto
nw.when(<Series>).then(<Series>).otherwise(nw.lit(replace_with)).alias(
feature). Passing Series from get_column() (not nw.col()) into
when/then/otherwise keeps this working for pandas integer column names,
same as base_encoder.py's precedent. A pandas Categorical column still
needs its own add_categories(replace_with) step before assignment - kept
as a small is_pandas-gated block (structural, like base_encoder.py's
existing reorder branches), since narwhals has no cross-backend
equivalent and polars has no matching restriction. Unlike the old
pandas-only code, no manual object-dtype fixup is needed before
assignment for the ignore_format + numeric-variable + string
replace_with case: narwhals resolves the common dtype itself (object in
pandas, cast-to-string in polars).
Benchmarked pandas-native vs narwhals-on-pandas vs narwhals-on-polars at
10k/50k/100k rows x 1/2/10 columns x 5/50 categories, warmed up. First
pass (zip_with(col, new_series_filled_with_replace_with)) averaged
2.41x pandas-native at 50k-100k rows - most of that cost was
constructing a full same-length replacement Series every transform()
call (~2.5ms of a ~4.8ms transform at 100k rows, confirmed by isolating
just the Series construction). Switched to nw.when(keep).then(col)
.otherwise(nw.lit(replace_with)), which lets the backend broadcast the
scalar instead of materialising a parallel array: dropped the average
to 1.60x, converging to 1.12x-1.54x at 100k rows/10 columns, the
"realistic size" range. narwhals-on-polars is faster than pandas-native
throughout (0.7x-1.5x, mostly <1x at 50k+ rows). Merged into a single
narwhals path per the established decision rule - no pandas/polars
performance split - the remaining overhead is fixed per-call cost, not
scaling cost, and stays under a few ms in absolute terms even at the
largest sizes tested.
Rewrote test_rare_label_encoder.py to one parametrized test per
behaviour over @pytest.mark.parametrize("make_df", [pd.DataFrame,
pl.DataFrame]), replacing the shared pandas-only module-level fixtures
(df_enc_big, df_enc_big_na, df_enc_numeric, from tests/conftest.py,
still used by other encoder test files) with local dict constants both
backends can build from, per the CountEncoder precedent. Kept
test_when_varnames_are_numbers and the three category-dtype tests
pandas-only (integer column names and pandas Categorical dtype are
backend-specific per AGENTS.md). Split
test_max_n_categories_with_numeric_var into a pandas-only version (the
existing str()-workaround test, unchanged) plus a new polars-only
version documenting the real, expected behavioural difference: polars
can't hold mixed int/str values in one column the way pandas' object
dtype does, so a numeric variable with a string replace_with casts the
whole column to string instead of leaving frequent numeric categories
as numbers.
Verified: tests/test_encoding/test_rare_label_encoder.py - 39 passed
(up from 29, from parametrizing over both backends); full
tests/test_encoding suite - 336 passed, 17 pre-existing failures with
identical test IDs confirmed against the unmodified base_encoder.py
baseline (numpy-array-input rejection checks plus 3 MeanEncoder
inverse_transform failures from mean_encoding.py's still-unmigrated
fit() - predate this change, reproduced identically on the unmodified
rare_label.py too). flake8 clean on feature_engine and tests. mypy
clean. Module imports with pandas blocked (loaded standalone, same
technique as the base_encoder.py migration, since sibling encoder files
in this package still import pandas at module level). sphinx -W build
clean (only the pre-existing linkcode_resolve warning, confirmed
identical against the unmodified baseline). Verified every doc example
in RareLabelEncoder.rst against actual output; fixed a pre-existing,
unrelated value_counts() Series-name drift ("Name: var_A" ->
"Name: count", a pandas version difference, not caused by this
migration) while touching that page, and added a verified "With
polars" section to both the class docstring and the user guide (the
polars value_counts() example needed an explicit .sort() - unlike
pandas, its groupby-based value_counts() order isn't stable run to
run). The Titanic-dataset section of the user guide could not be
re-verified against live output in this sandboxed environment (SSL
cert verification blocks urllib by default here, though curl succeeds)
and was left untouched; a workaround (unverified SSL context) showed
matching encoder_dict_/transform output, with only an unrelated
.unique() repr-formatting difference from a newer pandas version.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables, _check_na and _check_contains_na (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the redundant nw.from_native(X) round-trips in fit() and transform(). In transform(), detect the pandas Categorical fix-up path via nw_X.implementation .is_pandas() and run it on a copy so the user's dataframe is not mutated. Drop the now-unused narwhals.dependencies import. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
force-pushed
the
narwhals-rare-label-encoder
branch
from
August 30, 2026 22:43
0365e7e to
b6eaf5c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates
RareLabelEncoderto narwhals with polars support.fit(): pandas.unique()/.value_counts(normalize=True)replaced with narwhalsSeries.n_unique()(cardinality check — matches pandas' plainunique()length, which counts null as its own category) anddrop_nulls().value_counts(sort=True, normalize=True)(samedrop_nulls()/sort=Truereasoning asCountEncoder.fit()).transform()doesn't reuseCategoricalMethodsMixin._encode()(that's a dict-based numeric remap; this encoder keeps frequent categories as-is and only replaces the rest). Rewritten from pandas'.loc[~isin(...), feature] = replace_withontonw.when(<Series>).then(<Series>).otherwise(nw.lit(replace_with)). Series fromget_column()(notnw.col()) keeps this working for pandas integer column names. A pandas Categorical column still needsadd_categories(replace_with)first — kept as a smallis_pandas-gated block (structural, like the base's existing reorder branches); polars has no matching restriction.Merge vs split: first pass (
zip_with+ full replacement Series) averaged 2.41x pandas-native at 50k–100k rows, mostly the cost of building a full-length Series everytransform(). Switched tonw.when(keep).then(col).otherwise(nw.lit(replace_with))so the backend broadcasts the scalar — dropped to 1.60x average, 1.12x–1.54x at 100k/10 cols. narwhals-on-polars faster than pandas-native throughout. Single narwhals path, no performance split — remaining overhead is fixed per-call cost, a few ms at most.Tests rewritten to one parametrized test per behaviour over
make_df in [pd.DataFrame, pl.DataFrame](39, up from 29). Integer-column-name and category-dtype tests kept pandas-only.test_max_n_categories_with_numeric_varsplit into a pandas-only version and a new polars-only version documenting the real expected difference (polars can't hold mixed int/str in one column, so a numeric var + stringreplace_withcasts the whole column to string).Verified:
test_rare_label_encoder.py39 passed; fulltests/test_encoding336 passed / 17 pre-existing failures with identical IDs to the baseline. flake8 / mypy clean, sphinx -W clean.RareLabelEncoder.rstexamples verified against actual output (also fixed an unrelated pre-existingvalue_counts()Series-name drift while there); "With polars" section added. Titanic section left untouched (SSL blocks urllib in sandbox; workaround showed matchingencoder_dict_/transform output).Stacked on #999 (
narwhals-encoding-base). Until that merges this PR's diff also contains the sharedCategoricalMethodsMixincommit; review #999 first.