[jpn--]

Full-population destination and trip models could exhaust a roughly 52 GB memory budget because large table joins happened before normal utility chunking. This change bounds those joins and uses fixed chunk sizes in the laptop profile, while preserving the full model, output stages, and chooser random streams.

The production benchmark now validates inputs and outputs, records the complete container memory peak, and distinguishes a successful model run from a passing memory test. Both benchmark runners load extensions consistently and use the locked dependencies. The base profile remains available.

Full-population testing reached 47.50–50.23 GB with four workers in about 2 h 18 m, with identical final tables across the two runs. This is a constrained-memory proxy for a 64 GB machine, not a physical-hardware guarantee. See the run guide.

Validation after merging origin/merge-fixes: 148 tests passed; the updated 2,000-household non-Sharrow fixture passed output validation; Ruff, documentation formatting, links, and instruction synchronization checks passed. The post-merge Sharrow comparison and other remaining checks are described in the PR comment. PR creation is intentionally not waiting for long-running tests.

JPN: This PR is authored primarily by Codex using ASTRA-6.0-MEDIUM. I have not rigorously reviewed the changes here yet, but I've opened the PR so they can also be reviewed by others.

[jpn--]

Work completed

• Fixed the benchmark extension loading, configuration selection, locked Docker build, run provenance, and resume/qualification decisions.
• Bounded large destination logsum joins, reduced fixed chunk sizes where full-scale runs exceeded the budget, and avoided unnecessary wide person/tour joins. Batches keep each chooser's alternatives together to preserve random draws.
• Kept all production model steps, shadow pricing, matrices, and final outputs enabled. Added regression tests and a concise run guide.
• Merged origin/merge-fixes into this branch, resolving the overlapping runner/documentation changes and keeping its updated test data and fixtures. That branch is already in upstream main.

Measured results

Full inputs: 3,602,383 households and 8,674,446 persons. These measurements used model implementation 2abb9bf, before the integration merge, on a 128 GiB ARM Mac with a constrained Docker VM.

Full-population run	Peak RAM (decimal GB)	Runtime
Four workers, new compilation-cache lifecycle	47.50	2 h 17 m 38 s
Four workers, fresh warm-cache repeat	50.23	2 h 17 m 34 s
Two workers	47.43	3 h 19 m 24 s

Runtime excludes the separately monitored warm-up (about two to six minutes). The container hard cap was 52.08 GB (48.5 GiB) and the passing target was 50.47 GB (47 GiB). All three runs passed output validation with zero OOM events, container swap, or host swap growth. The repeated four-worker runs produced byte-identical final household, person, tour, trip, land-use, and accessibility tables.

Four workers are the recommended tested setting. The worst run had only 0.24 GB below the passing target, so this is not evidence of unlimited headroom. Raw reports and private production inputs remain local and are not committed.

Testing completed and still needed

Completed after the merge: 148 tests passed, plus the updated 2,000-household non-Sharrow laptop-profile run with output validation. Ruff, Markdown formatting for the resolved documents, documentation links, and instruction synchronization checks passed. Before the merge, the full-data backend fixture comparison passed exact non-logsum equality and the existing logsum tolerance (largest absolute difference about 0.000005).

Still to do:

• Finish GitHub CI and run the Sharrow/non-Sharrow comparison on the newly merged fixture; that post-merge comparison has not been run locally.
• Exercise the larger 10,000-household and single-process comparisons.
• Confirm the selected full-population configuration on the final integrated revision; the full-scale sweep has not been repeated after this merge.
• When hardware is available, test a physical 64 GB machine with normal background applications, checking responsiveness and swap. No such machine was available for this work.

Other operating systems, dependency versions, worker counts, or changed production inputs need their own memory checks. Per the request, these remaining longer tests are not delaying the PR.

[i-am-sijia]

I got an error running the full model with this branch. The error is in calculating mode choice logsum for school location. I haven't investigated, just want to keep you posted.
mp_households_tours_0-activitysim.log