Repository navigation
BUG: Preserve missing categorical observations in dummy conversion - #858
jackiectl2 wants to merge 1 commit into
Conversation
Signed-off-by: Tianlang (Jackie) Chen <305351710+jackiectl2@users.noreply.github.com>
bashtage
left a comment
bashtage
left a comment
There was a problem hiding this comment.
Overall a good idea.
Looks a bit risky in the order that it is in. Likely an easy fix to just drop all 0 columns if s.hasnans. Is hasnans old pandas, or newer? Might have some backward compat problems, but CI will tell.
| if isinstance(s.dtype, pd.CategoricalDtype): | ||
| out = pd.get_dummies(s, drop_first=drop_first) | ||
| if s.hasnans: | ||
| out = out.astype(float).where(s.notna(), axis=0) |
There was a problem hiding this comment.
This looks problematic with the next line. If something is dropped, the columns won't match. How about just checking if any dummies are all 0 and then dropping them?
| if isinstance(s.dtype, pd.CategoricalDtype): | ||
| out = get_dummies(s, drop_first=drop_first) | ||
| if s.hasnans: | ||
| out = out.astype(float).where(s.notna(), axis=0) |
There was a problem hiding this comment.
Same comment. First assign the columns, then do the drop.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #858 +/- ##
=======================================
Coverage 99.68% 99.68%
=======================================
Files 199 199
Lines 37875 37900 +25
Branches 2436 2438 +2
=======================================
+ Hits 37756 37781 +25
Misses 51 51
Partials 68 68
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
Preserve missing categorical observations when converting IV and panel inputs to dummy variables. Missing categories currently become all-zero rows and are treated as observations of the reference category. Keep those dummy rows missing so the existing model missing-data handling can remove them. Nonmissing category coding is unchanged.
Validation
Regression before: 4 failures. After: 212 tests pass in the IV and panel data test modules. Ruff, Black, numpydoc lint on changed production modules, and git diff --check pass. The full model suite was not run.