Optionally release memory during solving process

[FabianHofmann]

We could optionally write the model references to a NetCDF file just before the solving process and reassign afterwards. This could reduce the total memory usage by 30%. (Example: for a 112 GB peak memory example solving, 31 GB was held by linopy and 81 GB by gurobi.)

[tgi-climact]

I am testing #281 on larger problems of PyPSA-Eur with multiple planning horizons and scenarios. While waiting for a token, I have a lot of memory allocated for each waiting solve_sector_network_myopic. Could my problem be related to this issue ? Thanks

[MaykThewessen]

Hi @FabianHofmann — I've been hitting this on sequential PyPSA-Eur-style chunked solves (RSS climbing monotonically across weekly chunks until OOM on a 64 GB box). Before I open a PR I'd like to check whether the approach below fits the scope of this issue or should be a sibling issue, since the original framing here was about reducing peak during a single solve via NetCDF round-trip, and I'm tackling a different axis.

Proposed scope

A small opt-in helper that runs gc.collect() then asks the platform allocator to return freed arenas to the kernel after Model.solve() returns. Linux: libc.malloc_trim(0). macOS: libSystem.malloc_zone_pressure_relief(NULL, 0). Everything else: gc only, silent no-op. Wired into Model.solve(release_memory=False) so default behaviour is unchanged.

This is orthogonal to:

• perf: matrix accessor rewrite #630 (CSR freeze=True, merged) — shrinks peak in one solve.
• Persistent solver: in-place model updates via ModelDiff #699 (persistent solver / ModelDiff) — avoids rebuilds between solves.

Neither addresses the glibc/libSystem arena-retention pattern: even after del model + gc.collect(), freed pages stay cached in the allocator and RSS only grows, until the kernel OOM-kills the process.

Benchmark

5 sequential solves of an identical (5k vars × 2k cons, dense block) LP via HiGHS in one Python process, on macOS arm64, linopy master HEAD (0.7.0.post1.dev19+g37af4ba12):

metric	baseline	release_memory=True	delta
RSS post-cleanup, iter 4 (GB)	4.41	2.02	−54 %
ru_maxrss (GB)	5.30	4.32	−19 %
wall-clock total (s)	30.97	32.09	+3.6 % (≤ noise)

Baseline RSS climbs 3.71 → 4.41 GB monotonically across the 5 iterations. With the trim, post-cleanup RSS drops back to 1.5–2.0 GB after each solve. Linux numbers via malloc_trim are typically more dramatic; happy to add a Linux benchmark before opening the PR.

What I'd like to PR

• linopy/_memory.py — single release_allocator_pages() helper (~30 LOC, ctypes via libc.so.6 / libSystem.B.dylib, swallows OSError / AttributeError, debug-logs on miss).
• Model.solve(..., release_memory: bool = False) — kwarg, called once at the end of the existing solve path.
• Tests: platform-mocked (Linux / Darwin / Windows / FreeBSD), kwarg-default backward-compat smoke test, repeated-call stability.
• doc/memory-management.rst — short note on when to enable, with the table above.

Open questions

1. Does this fit inside Optionally release memory during solving process #219 or should I file a sibling issue ("opt-in arena trim after solve") and keep Optionally release memory during solving process #219 reserved for the NetCDF-roundtrip idea?
2. Naming: release_memory=True on Model.solve vs. a top-level linopy.release_allocator_pages() users call themselves vs. both. I'd default to both — kwarg for the common case, public helper for chunked-solve loops where users want to trim mid-pipeline.
3. Once Persistent solver: in-place model updates via ModelDiff #699 lands, the chunked-solve story shifts. I expect the trim hook stays useful (persistent solver reduces rebuild RAM but Gurobi/HiGHS still hold the native model + factor; arena cache still grows on the Python side). Agree?

Happy to defer until #699 settles if you'd prefer to look at the memory plane holistically.

[FBumann]

Not sure if I fully get this, but writing to netcf might not fully work if we do sos reformulations (raises on current master). Solvable, but needs to be checked

[MaykThewessen]

@FBumann thanks — good catch on the SOS reformulation interaction. I just checked master (37af4ba) to make sure I understood the failure mode: to_netcdf itself doesn't raise on a model with an active SOS reformulation — it warns (UserWarning, io.py L866-875) and persists the reformulated MILP form. What does break is the round-trip undo: after read_netcdf the lifecycle token is gone, so undo_sos_reformulation() raises RuntimeError: No SOS reformulation is currently applied. That matches the gap documented at io.py L856-861.

So the NetCDF-roundtrip path (this issue's original framing) has a real unresolved item: persisting (or reconstructing) the SOS lifecycle token across serialization. Worth keeping the focus of this issue.

To keep the two concerns from tangling I opened #720 for the arena-trim proposal sketched in my earlier comment. It's orthogonal: runs once after Model.solve() returns, never touches to_netcdf, purely asks the platform allocator to return freed pages to the kernel. Happy to keep #219 focused on the NetCDF / SOS-undo design question and move the allocator discussion to #720.

[MaykThewessen]

@FabianHofmann to close the loop on the question I asked you in my earlier comment: based on @FBumann's note above plus what I found in the code (to_netcdf warns rather than raises, but the SOS lifecycle token isn't persisted, so post-read_netcdf undo_sos_reformulation() raises), I think the two ideas live at different layers and benefit from separate tracking:

• This issue (Optionally release memory during solving process #219) — your original NetCDF-roundtrip proposal targets the solver-side native model + Python-side linopy model split (the 31 GB / 81 GB example). Real value, but the SOS-undo round-trip gap is a non-trivial design item that should drive the work here.
• Opt-in allocator arena release after Model.solve() to bound RSS across sequential solves #720 (just opened) — an opt-in release_memory=True kwarg on Model.solve() that asks the platform allocator (glibc malloc_trim, libSystem malloc_zone_pressure_relief) to return freed arenas to the kernel. Addresses a different failure mode: sequential chunked solves where Python-side RSS climbs monotonically across iterations even after del model + gc.collect(). Measured −54 % post-cleanup RSS on a 5-iteration HiGHS benchmark, +3.6 % wall time (≤ noise).

Happy to leave this issue focused on the NetCDF-roundtrip approach and move forward with #720 in parallel — they're complementary, not competing. Let me know if you'd rather fold both into one thread, or if there's a different scoping you'd prefer before I push a PR for #720.