Migrate OneHotEncoder to narwhals, add polars support - #1028
Open
solegalli wants to merge 2 commits into
Open
Conversation
Uses narwhals' to_dummies() for the actual expansion rather than a manual
numpy/dict loop, since it's a real vectorized one-hot op on both backends.
Handles two edge cases to_dummies() doesn't cover directly: a fixed-length
prefix placeholder ("__ohe_tmp__") swapped back out by slicing rather than
by using the real column name, since to_dummies() only prefixes with the
Series name when it's truthy - a falsy real name (e.g. an int column
literally named 0) would otherwise silently drop the prefix; and learned
categories absent from (or present-but-unlearned in) a given transform
batch, filled with an explicit all-0 column so unseen categories are
encoded as 0 across the board, matching the pre-narwhals behavior exactly.
fit()'s value_counts()/unique() calls and transform()'s reassembly are a
single unified narwhals path - no pandas/polars split needed, verified
directly on both backends (identical dummy columns/values for identical
input).
Rewrote tests/test_encoding/test_onehot_encoder.py to the single
cross-backend-parametrized-test convention: local dict fixtures (dropping
the pandas-only global df_enc_big/df_enc_numeric/df_enc_binary fixtures)
parametrized over make_df in [pd.DataFrame, pl.DataFrame], with narwhals-
based column/sum assertions replacing pd.testing.assert_frame_equal.
test_variables_cast_as_category stays pandas-only (pandas category dtype
has no polars equivalent under test there).
Verified: 43/43 own tests, full encoding suite 340 passed/17 pre-existing
failures (matches the narwhals-encoding-base baseline exactly), flake8
and mypy clean, sphinx -W build clean (only the pre-existing unrelated
linkcode_resolve warning), no pandas import in this file itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables, _check_contains_na and _get_feature_names_in (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the now-redundant nw.from_native(X) round-trips in fit() and transform(); they reuse the narwhals frame returned by check_X / _check_transform_input_and_state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
force-pushed
the
narwhals-one-hot-encoder
branch
from
August 30, 2026 22:10
630817c to
54d6c81
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates
OneHotEncoderto narwhals with polars support.Uses narwhals'
to_dummies()for the expansion rather than a manual numpy/dict loop (it's a real vectorized one-hot op on both backends). Two edge casesto_dummies()doesn't cover directly:"__ohe_tmp__") swapped back out by slicing, not by the real column name:to_dummies()only prefixes with the Series name when it's truthy, so a falsy real name (e.g. an int column literally named0) would silently drop the prefix.fit()'svalue_counts()/unique()andtransform()'s reassembly are a single unified narwhals path — no pandas/polars split, verified identical dummy columns/values on both backends for identical input.Tests rewritten to the cross-backend-parametrized convention (local dict fixtures over
make_df in [pd.DataFrame, pl.DataFrame], narwhals-based column/sum assertions).test_variables_cast_as_categorystays pandas-only.Verified: 43/43 own tests; full encoding suite 340 passed / 17 pre-existing failures (matches the
narwhals-encoding-basebaseline exactly); flake8 / mypy clean; sphinx -W clean; no pandas import in this file.Stacked on #999 (
narwhals-encoding-base). Until that merges this PR's diff also contains the sharedCategoricalMethodsMixincommit; review #999 first.