[snyaggarwal]

OpenConceptLab/ocl_online#636

When a new release comes out, the previous release is re-flagged without its shared members (exclude_members_of). On a large release this kept the staging database at about 99% CPU for about 37 hours.

Cause. exclude_source_members produced id NOT IN (subquery). Postgres can't turn NOT IN into an anti-join, and once the member list doesn't fit in work_mem it scans the whole list again for every row. OFFSET paging then repeats that work on every page of 500.

Changes

• exclude_source_members uses NOT EXISTS, correlated on the member id. Postgres runs it as an anti-join on the existing (source_id, member_id) index. It leaves out the same rows.
• batch_index_partial_by_ids gets each page after the first with id < last id instead of OFFSET, so every page costs the same. A union can't be filtered, so it keeps paging by OFFSET.
• Two new database tests, one for keyset paging and one for the union fallback. The mock paging test now checks the keyset filter.

Measured on staging (read-only, a release of about 430k concepts excluded from one of about 31k):

	Per page	Whole run
Before	23 s for a single row	about 35 h observed
NOT EXISTS	2.6–106 ms	about 3 s projected
NOT EXISTS + keyset	2.6–16 ms	under 1 s projected

Tests: BaseModelTest, SourceReleaseFlagsTest, core.sources.tests and TaskTest pass locally (361 tests), and pylint scores 10/10.

🤖 Generated with Claude Code

[snyaggarwal]

Review verdict: no P1–P5 findings and no nits.

Verified:

• BaseModelTest and SourceReleaseFlagsTest: 40 tests passed.
• Pylint on core/common/models.py, core/common/tasks.py, and core/common/tests.py: 10.00/10.
• The NOT EXISTS rewrite remains equivalent for both concept and mapping through tables, and the keyset path preserves the union fallback.

(Codex / GPT-5)