2026-04-09 Engineering Team #88
Description
Agenda
Items with Progress
-
We apparently have a problem with non-deterministic sort order on ProtoPop persons, which has become evident on BOTH of the next two items.
- Sort order problem appears in 1 of 2 tests of SANDAG-Example#41 here
-
Utility not aligned #1022 - possible cause identified and fix implemented in #1027, pending review from PSRC
-
Remove Three Zone Code and Examples #1056 and SANDAG-Example#41
-
Lower default threshold for skipped failed choices to 0.1% #1063 - Awaiting note in changelog then merge
-
Explicit Error Terms #1064
-
PopulationSim intermediate geographies camsys#1
Items without Progress
-
Park and Ride #1001 - review pending
-
Speed Up Nearest Zone Calculation in Disaggregate Accessibilities #1031 - review pending
-
Adding ignore list to skim load for unused skims while running with sharrow #1036 - review pending
-
Implement deletion of temporary variables in summarize.py #1062 - pending docs/testing
-
Model component recompilation when running MP with sharrow.
-
Comprehensive testing plan.
-
A bunch of other contributed or smaller PRs
New Contributed PRs
- Poisson sampling methodology for EET #1065 - Draft proposed new algorithm for sampling
Activity
jpn-- commentedon Apr 16, 2026
Meeting Overview
Agenda
Participants
Duration
Approximately 50 minutes
Executive Summary
Jeff Newman identified and resolved two related CI test reliability issues: a non-deterministic sort order in the proto population persons table, and a NumPy/sparse integer overflow bug causing failures in PSRC's full-scale two-zone model on Windows. The root fix for both involves upgrading NumPy to version 2, which standardizes 64-bit integer defaults across platforms and eliminates a longstanding class of Windows-specific bugs. Jan Zill provided an update on the explicit error term (EET) pull request, which is nearly ready for review and merge, pending addition of a zone-ID-based ordering fix for sampled choices. A PopulationSim config-reading issue remains blocked pending Nick Fournier's return.
Meeting Notes
Non-Deterministic Sort Order in Proto Population
Jeff Newman described a non-deterministic issue discovered in the proto population persons table. The problem surfaced when David Hensle's pull request removing an unused setting from the SANDAG model caused the CI tests to run twice on identical code — one pass, one failure — because the person row ordering differed between runs.
The root cause appears to be that the existing sorting code sorted on a single key (person ID) while implicitly relying on the table already being partially sorted by household. When that implicit assumption fails, the row order is non-deterministic, causing disaggregate accessibility to use different random number seeds (which are keyed to index row number), leading to approximately 1% variance in log sum outputs. While small, this is enough to cause CI regression test failures.
The fix is to make all sorting explicit by adding compound sort keys (e.g., sort the persons table by household ID and person ID, and the tours table by multiple keys), so the sort outcome is fully determined without relying on any prior ordering. Jeff noted this only needs to be done in one place — the proto population generation — since the rest of ActivitySim operates on an exogenously provided population with stable IDs.
Joe Castiglione asked whether the fix introduces any real cost. Jeff confirmed the overhead is negligible and that going forward all sorts should be made explicit rather than relying on stable-sort assumptions.
NumPy Integer Overflow Bug (PSRC Two-Zone Model)
Jeff Newman also described a related bug causing PSRC's full-scale two-zone model to fail in Sharrow mode when running under
uvon Windows. The failure arises from a specific combination of conditions:sparselibrary at certain versionsUnder these conditions, sparse matrix indices overflow, causing values in a portion of the walk table to be misaligned.
The solution Jeff implemented is to upgrade NumPy to version 2, which standardizes on 64-bit integers across all platforms. This required also upgrading Pandera (pinned at v18, which is incompatible with NumPy 2) and a few other dependencies. Jeff noted that existing workaround code (conditionals guarding against int32 overflow) can remain in place harmlessly — it will simply never trigger — and can be cleaned up later.
Sijia Wang clarified that the overflow issue is specific to Sharrow mode (not legacy/non-Sharrow). Jeff confirmed that PSRC's model runs fine with Sharrow disabled; the bug only surfaces when all the trigger conditions are met simultaneously. Stefan Coe is expected to test the fix on his Windows machine when he returns.
Three-Zone Code Removal (David Hensle)
David Hensle (not present) has completed and pushed the pull request removing the three-zone system code from ActivitySim. Jeff Newman noted this needs to be reviewed and merged.
Explicit Error Term (EET) Pull Request — Jan Zill
Jan Zill gave an update on the EET pull request. The PR includes documentation and unit tests for the explicit error term functionality. During pre-submission testing, Jan discovered an edge case in the existing trip scheduling model related to how trips get rescheduled on subsequent iterations in small-sample (20-household) test runs, which could produce divergent paths depending on multi- vs. single-processing. This was addressed with a random seed fix, documented in the PR.
One remaining item is a fix for alternative order sampling: currently, random numbers for sampled destination choices are indexed by sample index rather than zone ID, making them unstable. A fix based on MATS's implementation is in progress and could be added to the current PR or submitted separately. Jan indicated this could be completed within ~12 hours.
The PR has substantial changes and will require agency review. Jeff noted David Hensle's availability is limited in the near term. Jan confirmed the PR is otherwise ready to merge and will complete the ordering fix before tagging reviewers.
Sijia Wang asked whether the probabilistic trip scheduling changes were tested against the logit-based trip scheduling model (DTPS) used by ARC and others. Jan confirmed that the underlying choice mechanisms are unit tested, and that models using DTPS go through the same underlying ActivitySim Core path, which is covered.
Testing Infrastructure Discussion
The group discussed CI testing constraints. GitHub provides free CI runners for open-source repositories, but they are significantly slower than local machines (~6–10× slower per Jan). Current longer-running tests take 60–90 minutes on CI. Jeff noted the GitHub per-job cap may be around 3 hours (exact limit unclear), and that adding more tests should be done by breaking out additional jobs rather than extending existing ones.
Jeff expressed a preference for moving toward smaller, faster unit tests that each run in seconds, reserving large integration/regression tests for pre-release validation rather than every push. A more robust system for running those larger tests on non-GitHub infrastructure was noted as worth exploring, though it has been discussed without resolution multiple times in the past.
PopulationSim Config Issue — Joe Flood
Joe Flood reported that the PopulationSim issue — where the tool reads config files from two different directories, causing setting conflicts and crashes — remains unresolved. Nick Fournier, who owns this work, is out this week. The group agreed to wait for Nick's return before deciding next steps.
Action Items
janzill commentedon Apr 17, 2026
@jpn-- looks like tagging reviewers for the EET PR (ActivitySim/activitysim#1064) is on your and my todo list :) The work is done but I cannot add reviewers, could you do this please?