2026-05-21 Engineering Team

[joecastiglione]

Agenda

• sampling bias in accessibility
• random numbers

(Jeff might join late, don't wait)

[janzill]

Full EET PR, including Poisson sampling, at ActivitySim/activitysim#1064. As mentioned on Tuesday and in that PR, I believe disaggregate accessibilities are biased by log(sample_size) when using the sample correction factor in MTC_extended/SANDAG with MC or EET sampling, whereas it is not for Poisson sampling with the current implementation and the same spec. I will prepare some slides to go over this.

[dhensle]

New "faster" RNG is indeed faster, but still not quite as fast as the non-EET run. Below screenshot shows full sandag disagregate accessibilities model running single process and I think is a solid comparison for a model with a ton of EET draws.

Full profiling for a run with a 10k households running single process without sharrow on an RSG server shown below.
profiling_eet_legacy_10k_single_process.zip
profiling_no_eet_10k_single_process.zip
profiling_faster_10k_single_process.zip

runtime_comp_sandag_10k_sample_sp.xlsx

[jpn--]

Meeting Overview

Agenda

• Disaggregate accessibility sampling bias: analysis, implications, and proposed fix
• EET (explicit error term) random number generation performance investigation

Participants

• Jeff Newman (@jpn--), Driftless
• David Hensle (@dhensle), RSG
• Jan Zill (@janzill), Outer Loop
• Joe Castiglione (@joecastiglione), Zephyr Foundation
• Stefan Coe (@stefancoe), PSRC
• Sijia Wang (@i-am-sijia), WSP
• Bhargava Sana (@bhargavasana), SANDAG

Duration

Approximately 62 minutes

---

Executive Summary

Jan Zill presented an analysis showing that the sample correction factor in ActivitySim's disaggregate accessibility calculation contains a systematic mathematical bias equal to the natural log of the sample size (log k). The bias is inconsequential for location choice models (it cancels out) but materially affects disaggregate accessibility outputs used by agencies such as SANDAG and PSRC. The group reached broad consensus on a fix: correct the sampling code to produce unbiased outputs by default, and provide a configuration switch allowing agencies with existing estimated models to reintroduce the legacy bias until they can recalibrate. A separate investigation into Gumbel random number generation performance showed meaningful but not transformative speed improvements for ET sampling.

---

Meeting Notes

Disaggregate Accessibility Sampling Bias: Analysis

Jan Zill presented histograms comparing disaggregate accessibility distributions under Monte Carlo and Poisson sampling for the MTC model, demonstrating that the two methods produce statistically identical results in terms of noise but show a constant mean shift. For the MTC model the shift was approximately 6.6, and for SANDAG approximately 4.6, matching the natural logarithm of the respective sample sizes. Jan showed analytically that the current sample correction factor in the YAML specification (using pick_count and prob) omits the true McFadden correction term, specifically the log(k) (sample size) component. This omission is harmless for discrete destination choice because the constant drops out of the utility differences, but it directly biases disaggregate accessibility log sums used downstream.

Practical Impact on Model Results

Jan showed that when running the SANDAG model with Poisson-based person sampling (the new default), auto ownership and CDAP results deviate noticeably from the Monte Carlo baseline — consistent with the ~0.5 utility unit bias propagating into auto ownership choice. We discussed whether this was a SANDAG-specific problem or more widespread; Stefan Coe confirmed that PSRC uses disaggregate accessibilities in several estimated models (approximately 3–4). David Hensle noted that the same bias would affect location choice log sums output for downstream use, not just the dedicated disaggregate accessibility step. MWCOG was also identified as a user of disaggregate accessibilities.

Proposed Fix and Strategy

Jeff Newman synthesized the group's discussion into a recommended approach:

1. Fix the code to return the correct, unbiased inclusion probabilities from the sampling step, making Poisson and Monte Carlo interchangeable by default.
2. Remove the sample correction factor from user-facing YAML specs and handle it internally in ActivitySim code, improving user-friendliness and eliminating a common source of error.
3. Add a configuration switch (YAML setting) that, when enabled, reintroduces the legacy bias into disaggregate accessibility outputs. This allows agencies whose models were estimated against the biased values (currently SANDAG and PSRC, potentially MWCOG and others) to maintain consistent results without immediate recalibration.
4. Provide hands-on support to all current consortium member agencies affected by the breaking change, ensuring their agency models are updated correctly.

David Hensle raised the point that because the bias is a constant shift, model coefficients for threshold and linear terms can be adjusted analytically rather than requiring full re-estimation, which Jan and Jeff confirmed is valid for the model structures observed so far (no nonlinear interaction terms seen in the relevant specs). Jeff Newman noted the fix should be brought to the broader group for buy-in before adoption.

Bhargava Sana expressed support for simply fixing the issue, characterizing it as an expected kind of change in a continuously evolving model. Joe Castiglione asked whether the consortium's SANDAG example model and the SANDAG production model need to remain in sync; Jeff Newman argued that the example model should track the correct code going forward, while production SANDAG uses the compatibility switch until recalibration.

Sijia Wang asked how this affects new agencies adopting SANDAG as a donor model; Jeff Newman noted the example model should be updated to use the corrected code so that new adopters get the unbiased version out of the box.

Jan estimated the implementation effort at approximately 2–3 days for a minimum viable version, noting the relevant logic is well-isolated within interaction_sample.

EET Sampling Runtime Performance

David Hensle reported benchmarking results comparing no-EET, legacy EET, and an optimized "faster EET" approach on the disaggregate accessibility module (single process, full model):

Method	Runtime
No ET	~55 min
Original EET	~186 min (237% slower)
Faster EET	~138 min (~26% improvement over legacy, still ~2.5× slower than no ET)

A similar pattern was observed across other destination choice models (non-mandatory tour destination, workplace location). Profiling showed the dominant cost is not entropy/randomness generation but the two numpy.log operations needed to transform a uniform draw into a Gumbel variate. Jeff Newman suggested investigating a Ziggurat-style approximation for the Gumbel transformation, which could yield another 10–15% improvement, though it would not close the gap entirely.

The group agreed that Poisson sampling still has a valuable role to play in EET work given its speed advantage over the legacy EET approach, and that the sampling bias fix is a prerequisite for making Poisson sampling properly available.

David Hensle noted that reading random numbers from a file was also discussed as a possible avenue and may warrant further exploration.

Pull Request / Merge Discussion

Jan Zill noted that a recent pull request covering explicit error terms testing, documentation, and Poisson sampling fixes (with loop restructuring to avoid excess memory materialization) is ready for review. The sheet shared by Joe Castiglione lists RSG and TfNSW as reviewers. David Hensle indicated he would review and likely merge it into the current working branch. Jan will tag reviewers on the PR.

---

Action Items

• Jan Zill: Present the sampling bias fix recommendation to the broader ActivitySim group for buy-in before the engineering team proceeds.
• Jan Zill: Implement the sampling correction factor fix in code (~2–3 days), including the compatibility switch for agencies with legacy-biased estimated models, and ensure the consortium example model uses unbiased outputs.
• Jan Zill: Tag David Hensle (and TfNSW reviewer) on the explicit error terms / Poisson sampling pull request for review.
• David Hensle: Review and merge Jan's open pull request on explicit error terms and Poisson sampling into the current branch.
• David Hensle and Jeff Newman: Investigate whether a Ziggurat-style approximation for the Gumbel transformation can further reduce ET sampling runtime.
• David Hensle: Profile memory usage of the ET approach, which has not yet been done.
• David Hensle (or other): Investigate reading random numbers from a file as a potential performance avenue.
• Engineering team: Audit all model specifications (SANDAG, PSRC, MWCOG, and any other affected agencies) for use of disaggregate accessibility and location log sums to identify every place the constant bias adjustment or coefficient update will need to be applied.
• Jeff Newman: Coordinate with SANDAG, PSRC, and MWCOG (and any other affected member agencies) to ensure their models are correctly updated following the breaking change.

[janzill]

@jpn-- regarding the second action item, I did not commit to fixing things within the current task.

[jpn--]

@jpn-- regarding the second action item, I did not commit to fixing things within the current task.

Thanks for checking me, I deleted that action item. The AI notes are getting good enough that it becomes all too easy to miss details when reviewing them.