Repository navigation
BUG: Preserve missing categorical observations in dummy conversion - #858
Conversation
Signed-off-by: Tianlang (Jackie) Chen <305351710+jackiectl2@users.noreply.github.com>
bashtage
left a comment
bashtage
left a comment
There was a problem hiding this comment.
Overall a good idea.
Looks a bit risky in the order that it is in. Likely an easy fix to just drop all 0 columns if s.hasnans. Is hasnans old pandas, or newer? Might have some backward compat problems, but CI will tell.
| if isinstance(s.dtype, pd.CategoricalDtype): | ||
| out = pd.get_dummies(s, drop_first=drop_first) | ||
| if s.hasnans: | ||
| out = out.astype(float).where(s.notna(), axis=0) |
There was a problem hiding this comment.
This looks problematic with the next line. If something is dropped, the columns won't match. How about just checking if any dummies are all 0 and then dropping them?
There was a problem hiding this comment.
Updated in 46645aa. The dummy columns are named first; when the category has missing observations, columns with no observed 1s are then dropped before marking missing rows as NaN. I replaced s.hasnans with s.isna().any() for pandas compatibility. The new tests cover unused categories with and without drop_first; 216 IV/panel data tests, Ruff, and Black pass.
| if isinstance(s.dtype, pd.CategoricalDtype): | ||
| out = get_dummies(s, drop_first=drop_first) | ||
| if s.hasnans: | ||
| out = out.astype(float).where(s.notna(), axis=0) |
There was a problem hiding this comment.
Same comment. First assign the columns, then do the drop.
There was a problem hiding this comment.
Applied the same order in the panel conversion path in 46645aa: name the columns, drop unused dummies, and then mark missing rows. The panel regression cases pass.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #858 +/- ##
==========================================
- Coverage 99.68% 99.68% -0.01%
==========================================
Files 199 199
Lines 37875 37919 +44
Branches 2436 2440 +4
==========================================
+ Hits 37756 37798 +42
Misses 51 51
- Partials 68 70 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Tianlang (Jackie) Chen <305351710+jackiectl2@users.noreply.github.com>
Summary
Preserve missing categorical observations when converting IV and panel inputs to dummy variables. Missing categories currently become all-zero rows and are treated as observations of the reference category. Keep those dummy rows missing so the existing model missing-data handling can remove them. Nonmissing category coding is unchanged.
Validation
Regression before: 4 failures. After: 212 tests pass in the IV and panel data test modules. Ruff, Black, numpydoc lint on changed production modules, and git diff --check pass. The full model suite was not run.