Skip to content

BUG: Do not materialize weighted data when all weights are 1 - #862

Open
DresdenGman wants to merge 2 commits into
bashtage:mainfrom
DresdenGman:fix-607-weights-dtype
Open

DresdenGman wants to merge 2 commits into
bashtage:mainfrom
DresdenGman:fix-607-weights-dtype

Conversation

@DresdenGman

Copy link
Copy Markdown

Closes #607

Problem

In IV2SLS (and the other IV estimators sharing the same base class), the weighted arrays _wy, _wx, and _wz were always constructed by multiplying the input data by the square root of the weights. That multiplication promotes the arrays to float64 and allocates full-size copies, even when every weight is 1 and the weighted data is identical to the input — including the default unweighted case, where the weights are populated with 1.0 internally. For large low-dtype inputs, this causes the memory blow-up reported in #607: full-size float64 copies are materialized unconditionally before estimation begins.

Change

linearmodels/iv/model.py: when all weights are 1, _wy, _wx, and _wz now alias the original arrays instead of allocating multiplied float64 copies, preserving input dtypes and avoiding the extra full-size allocations. (Weights are normalized to have mean 1.0 before these arrays are built, so any constant weight vector reaches this check as all ones.) Any non-constant weight vector takes the existing materializing path unchanged.

Multiplying by a weight of exactly 1 is exact, so for float64 and integer inputs the estimation results are unchanged. One deliberate consequence: float32 inputs now stay in float32 through the weighting step instead of being silently promoted to float64, so a float32 fit agrees with the float64 fit of the same data within float32 precision rather than exactly (the tests below use rtol 1e-6 for parameters and 1e-5 for the covariance).

Scope note: #607 also mentioned PanelOLS. PanelOLS does not unconditionally materialize weighted copies — it already branches on np.any(weights != 1.0) to take a fast path for unweighted data (the same idiom used here) — so this PR is scoped to the IV models, where the unconditional copy occurred.

Tests

9 new test functions in linearmodels/tests/iv/test_model.py:

  • Unit weights alias the original arrays (no copy, dtypes preserved); non-constant weights still materialize.
  • Results with no weights and with constant weights are identical (parameters and covariance).
  • A float32 fit with all-ones float32 weights matches the float64 fit of the same data within the tolerances above; an integer-input fit with constant integer weights matches the float64 results exactly.
  • Equivalence against an independently constructed reference: 2SLS computed by hand from materialized float64 arrays using numpy.linalg.pinv agrees with the new path to 1e-8 (measured deviation ~1e-13).
  • Boundary behavior is unchanged from baseline: a single non-unit weight takes the materializing path, with the stored arrays scaled by the square root of the weights; NaN weights follow the existing missing-value contract (the observation is dropped, verified against a manual row deletion).

Verification

IV test suite (same command before and after): baseline 9521 passed / 2400 skipped / 0 failed → with this change 9530 passed / 2400 skipped / 0 failed. The delta is exactly the 9 new test functions; the affected subset (test_model, test_weight_config, test_data, test_gmm) is 140 passed.

Panel suite re-run: 3680 passed (PanelOLS code untouched).

An A/B comparison of 20 result items between unweighted and all-ones-weighted runs of the new code is bit-identical; equivalence to the previous materialized path is covered by the independent-reference test above.

Peak memory on the reproduction case: 21.2 MiB → 10.7 MiB.

Known limitation (performance only): weights that are constant only up to floating-point rounding (within 1 ulp of a constant after normalization) still take the materializing path. Results are correct; the shortcut is simply not taken.

The IV models always stored copies of the dependent variable, the
regressors and the first-stage regressors scaled by the square root of
the weights, even when all of the weights are 1, which is the case
when weights is None or the weights are constant. Scaling by 1 does
not change the data, so the weighted data is now the data in this
case. This removes three full-size copies, which could require tens
of GiB for the large data sets in GH 607, where constructing the
weighted regressors raised a MemoryError.

Estimation results are unchanged: multiplication by a square root
weight of exactly 1 is exact, and the new tests verify that results
with no weights or constant weights are identical, that non-constant
weights still materialize the scaled data, and that float32 inputs
with float32 weights agree with the float64 estimates.
Add tests requested in review of the GH 607 fix: float32 data with
all-one float32 weights and integer data with constant integer weights
are fitted and agree with the float64 estimates; the unmaterialized
path agrees with a reference computed from explicitly materialized
weighted data; a single non-one weight materializes; and a NaN weight
drops its observation, as before.
@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.68%. Comparing base (5b85e0d) to head (8f81bfe).

Additional details and impacted files
@@           Coverage Diff            @@
##             main     #862    +/-   ##
========================================
  Coverage   99.68%   99.68%            
========================================
  Files         199      199            
  Lines       37875    37998   +123     
  Branches     2436     2437     +1     
========================================
+ Hits        37756    37879   +123     
  Misses         51       51            
  Partials       68       68            
Flag Coverage Δ
adder 99.67% <100.00%> (+<0.01%) ⬆️
subtractor 99.67% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MemoryError with IV2SLS

1 participant