[mihow]

Summary

This is for the people who curate data for model evaluation and retraining. With this change they can save a hand-picked list of occurrences as a named occurrence set through the API, list and choose a project's sets, and narrow the occurrence list to the members of one set, which is how a set is reviewed. The list of occurrences is fixed when the set is created, so a classifier scored on a set today and another scored on it next month were measured on the same occurrences.

It is the second step of the occurrence sets epic (#763). The work was split from #1492 by @mohamedelabbas1996, and it carries his API code and tests with the contract changes agreed in review: the filter is named occurrence_set_id, membership can only be set on create, ids are validated in one query, and choices follow the capture set picker. His evaluation and retraining stacks can rebase onto it, with the small renames listed under "For the stacked branches" below.

List of Changes

Change (effect)	How	Notes
Curators (ML data managers, project managers) can create a set from a list of occurrence ids in their project.	POST /api/v2/occurrences/sets/ with name, description, project_id, occurrence_ids. OccurrenceSetPermission checks create_occurrenceset on the project named in the body.	The project is read from the body only, never the query string, so the project checked is the project written to.
Ids that are missing or belong to another project are refused, and the error names up to ten of them.	occurrence_ids is a ListField(IntegerField) checked against the project in one query.	Avoids PrimaryKeyRelatedField(many=True), which runs one query per id. A non-integer or unknown project_id returns 400.
A set's occurrences and project cannot be changed after creation; the name and description can.	occurrence_ids and project_id are refused on PATCH; PUT is not offered (405).	Deleting or merging an occurrence still removes it from every set. That is documented and left to a follow-up ticket.
A project's sets can be listed with their size, and offered in a picker.	GET /occurrences/sets/?project_id= (project required) and GET /occurrences/sets/choices/?project_id=.	Choices mirror the capture set choices: most recently updated first, ChoicesPagination, names only.
Public sets (sets with no project) are offered to every project and anyone can open them, but they cannot be changed through the API.	OccurrenceSetPermission allows retrieve, since the queryset already limits what a caller can reach. Edits fall through to check_permission, which refuses an object with no project.	Public sets are made in the Django admin (previous PR).
A set shared by two projects is edited under the project named in the request, so the edit buttons a list row offers are the ones that work.	When project_id names one of the set's projects, the edit check uses the caller's role in that project; otherwise the set's first project governs.	Only the Django admin can share a set between projects.
The occurrence list (and occurrence stats) can be narrowed to one set.	OccurrenceSetFilter backend reading occurrence_set_id, added to OCCURRENCE_FILTER_BACKENDS, documented in the OpenAPI schema.	A backend rather than a FilterSet field so the export framework can run the same filter in a later PR.

Detailed Description

Permissions. Create is the only action without a saved object to check, so OccurrenceSetPermission handles it: anonymous callers are refused, project_id is parsed from the request body with SingleParamSerializer (a non-integer gives 400), an unknown project gives 400, and otherwise the caller needs create_occurrenceset on that project. Reading a set needs no object check: the detail queryset goes through visible_for_user(), so a caller only reaches sets of projects they can see, plus public sets, which anyone may read. An edit is checked against the project in the request when the set belongs to it, so that a set shared by two projects (possible only through the admin) can be edited by the curators of either, under their own project, which matches the user_permissions on its row in that project's list. Without such a project, BaseModel.check_permission checks the set's first project and refuses a public set. OccurrenceSetViewSet.create skips DefaultViewSet's unsaved-instance check, because both the project and the occurrences are many-to-many and cannot be put on an unsaved instance.

Visibility. Listing needs no permission branch: the queryset goes through visible_for_user(), which hides the sets of draft projects from non-members, and then for_project(), which adds public sets. A non-member listing a draft project sees only public sets.

Per-row permissions. A set can belong to several projects, so the user_permissions on each row are resolved against the active project with add_m2m_object_permissions, the same helper taxa lists use. The project and the caller's permissions are looked up once per request, and the set's projects are prefetched, so the list does not grow by a query per row. A detail read without project_id uses the set's own project, which keeps the offered actions in line with what check_permission allows (none for a public set).

Counts. occurrences_count comes from the with_occurrences_count() annotation when it is present. The create response uses the number of ids just added rather than running a COUNT.

Routing. occurrences/sets is registered before occurrences, like occurrences/stats, so the occurrence detail route does not read sets as a primary key.

What is measured and what is not. The query-count tests compare counts directly (create with 5 ids against 50, and list and choices with 1 set against 3) and require them to be equal. Removing the projects prefetch makes both list tests fail (14 against 12 and 16 against 14), which shows the tests detect a per-row query. No timings were taken on production-sized data. The filter adds one join through the membership table, and its cost on large projects has not been measured.

For the stacked branches

The evaluation (#1493, #1506) and retraining (#1501 to #1504) branches build on #1492's version of this code. When they rebase onto this PR and the model PR below it:

1. The occurrence-list filter parameter is occurrence_set_id, not occurrence_set. In the evaluation branch this is the UI filter key (ui/src/utils/useFilters.ts, filter-control.tsx) and the occurrence_set = RelatedIdFilter(field_name="evaluation_sets") FilterSet field, which this PR's OccurrenceSetFilter backend replaces.
2. The reverse accessor from an occurrence is occurrence_sets, not evaluation_sets (for example occurrence__evaluation_sets in ami/ml/training/dataset.py on the retraining branch).
3. The migrations are 0100_occurrence_set and 0101_grant_occurrence_set_permissions, as on the retraining branches. The evaluation branch still has them as 0098 and 0099.
4. Every branch adds ami/main/test_occurrence_sets.py, which will conflict with this PR's version of the file. Take this PR's version and re-add any tests the branch needs on top of it.
5. Worth considering: rename the ?occurrence_set= parameter of ml/training-data/summary to occurrence_set_id, so every endpoint names a set the same way.

OccurrenceSet.objects.for_project(), visible_for_user() and the occurrences/sets route are unchanged from #1492, so code that only uses those should keep working.

How to Test

docker compose run --rm django python manage.py test ami.main.test_occurrence_sets ami.main.models_future.tests.test_occurrence_sets

By hand, as a user with the ML data manager role in a project:

1. POST /api/v2/occurrences/sets/ with {"name": "Blind set", "project_id": <id>, "occurrence_ids": [<ids>]} returns 201 with occurrences_count.
2. GET /api/v2/occurrences/?project_id=<id>&occurrence_set_id=<set id> returns only those occurrences that the project's default filters keep.
3. PATCH /api/v2/occurrences/sets/<set id>/ with {"occurrence_ids": [...]} returns 400, and with {"name": "Renamed"} returns 200.
4. GET /api/v2/occurrences/sets/choices/?project_id=<id> lists the project's sets and any public sets, most recently updated first.
5. GET /api/v2/occurrences/sets/<public set id>/ returns 200 for any caller, and PATCH on it returns 403, even for a superuser.

Related Issues

Closes #1534
Part of #763
Split from #1492

Stacked on

feat/occurrence-set-model (the occurrence set model, migrations, permissions and Django admin). Retarget this PR to main once that one merges.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MeUoKCfkiZaZjhja7e6AGh

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	8acbebd
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6ad7e86c5b5be067254ba5cc

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}