Skip to content

feat: training set SDK support (DE-8692) - #480

Open
luke-e-schaefer wants to merge 10 commits into
masterfrom
lukeschaefer/de-8692-training-sets
Open

feat: training set SDK support (DE-8692)#480
luke-e-schaefer wants to merge 10 commits into
masterfrom
lukeschaefer/de-8692-training-sets

Conversation

@luke-e-schaefer

@luke-e-schaefer luke-e-schaefer commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Training Set SDK support (DE-8692)

Adds SDK support for training sets — a mutable, versioned, model-scoped collection of dataset_item ids spanning one or more source datasets. Similar to the Benchmark resource.

Ticket: DE-8692

What's added

  • nucleus/training_set.pyTrainingSet dataclass: from_json, refresh, update, delete, items, add_items, remove_items, new_version, family, lineage fields.
  • NucleusClient methods: create_training_set, get_training_set, list_training_sets, get_model_training_set, repin_training_set, update/delete_training_set, list/add/remove_training_set_items, create_training_set_version, list_training_set_family.
  • Model entry points: Model.create_training_set(...), Model.training_set, Model.repin_training_set(...).

Greptile Summary

The PR adds model-scoped training-set creation, mutation, versioning, export, and media-download support.

  • Exposes training sets through NucleusClient, Model, and the top-level package.
  • Adds payload and response DTOs for membership, lineage, and asynchronous operations.
  • Adds paginated JSONL export and per-dataset media downloads with collision handling.
  • Adds unit coverage for the new SDK workflows.

Confidence Score: 4/5

The PR is not yet safe to merge because downloads can still silently replace members whose generated paths alias on case-insensitive filesystems.

Collision tracking compares exact path strings, but supported filesystems can resolve differently-cased strings to the same entry, allowing the later media transfer to overwrite the earlier one.

Files Needing Attention: nucleus/training_set.py

Important Files Changed

Filename Overview
nucleus/training_set.py Adds the training-set resource and local export/download helpers; collision detection can still overwrite case-aliased destinations on case-insensitive filesystems.
nucleus/init.py Adds client APIs for training-set CRUD, membership, versioning, pagination, and export.
nucleus/model.py Adds model-level convenience entry points for creating, reading, and repinning training sets.
nucleus/data_transfer_object/training_set.py Adds the typed training-set membership-page response.
tests/test_training_sets.py Covers the added workflows and exact-string download collisions but does not exercise case-insensitive destination aliases.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  User[SDK caller] --> Model[Model]
  Model --> TrainingSet[TrainingSet]
  TrainingSet --> Membership[Add or remove members]
  TrainingSet --> Version[Create version]
  TrainingSet --> Export[Page export records]
  Export --> JSONL[Write JSONL]
  Export --> Download[Download media by dataset]
Loading

Fix all with Greploop Fix All in Cursor Fix All in Claude Code Fix All in Codex

Prompt To Fix All With AI
### Issue 1
nucleus/training_set.py:330
**Case-aliased downloads still overwrite**

If `download_items` runs on a case-insensitive filesystem, members whose generated paths differ only by case, such as `Frame.jpg` and `frame.jpg`, bypass the exact-string `used_paths` check. The second transfer then replaces the first filesystem entry while the method reports both members as downloaded.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (6): Last reviewed commit: "flexing" | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.

Context used:

luke-e-schaefer and others added 3 commits August 27, 2026 00:06
…collections (DE-8692)

Add a TrainingSet resource mirroring Benchmark: a mutable, versioned,
model-scoped collection of dataset_item ids spanning one or more datasets.

- nucleus/training_set.py: TrainingSet dataclass (from_json, refresh, update,
  delete, items, add_items, remove_items, new_version, family) with lineage
  fields (parent_training_set_id, version_major/minor/label).
- NucleusClient methods: create_training_set (model-scoped, unified source
  signature incl. training_set_ids), get/list/update/delete, add/remove/list
  items, create_training_set_version, list_training_set_family, repin, and
  get_model_training_set.
- Model.create_training_set, Model.training_set, Model.repin_training_set.
- TrainingSetItemsPage DTO; new *_KEY constants (TRAINING_SET_ID(S)_KEY,
  PARENT_TRAINING_SET_ID_KEY); reuse existing item/slice/dataset/version keys.
- Exports, version bump 0.21.2 -> 0.21.3, CHANGELOG entry.
- tests/test_training_sets.py: 28 fully-mocked tests (no live API).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nload_items) (DE-8692)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@luke-e-schaefer luke-e-schaefer self-assigned this Aug 31, 2026
@luke-e-schaefer
luke-e-schaefer marked this pull request as ready for review September 2, 2026 16:11
Comment thread nucleus/training_set.py Outdated
Merge blockers:
- pylint: make export_training_set_records public (drop cross-object
  protected access) and open() export file with encoding="utf-8".
- Export pagination (iter_training_set_export_records): advance offset by
  records returned, terminate on a short/empty page, and only trust a
  positive `total` as a secondary early-out — fixes first-page-only
  truncation on missing/0 total, item-skipping on short pages, and the
  runaway request loop on an over-counting total.
- download_items: namespace files under a per-dataset_id subdirectory and
  sanitize the reference_id basename, fixing silent overwrites on
  cross-dataset reference_id collisions and a path-traversal write.
- Align the create/version async contract: create_training_set_version no
  longer raises when the change is synchronous (no job_id); create/add/
  remove/version now agree. Corrected the misleading "synchronous" comment
  and docstrings.
- remove_training_set_items: return_raw_response=True and parse a job_id
  only when there is a body, so a 204/empty DELETE response can't crash
  .json() after the removal already succeeded.

Robustness / simplification:
- Reuse model_weights._stream_weights_to_file (retries + Content-Length
  short-read check) instead of a degraded local copy.
- Stream export_to_file / download_items via the record generator instead
  of materializing the whole set in memory.
- download_items now genuinely skips media-less records (matches docstring).
- export_to_file writes the raw record verbatim (drops redundant key list).

Nits:
- Model.create_training_set/training_set/repin_training_set return
  annotations via TYPE_CHECKING.
- Use ITEMS_KEY/TOTAL_KEY constants; add TOTAL_KEY.
- Revert unrelated black-churn hunk in update_evaluation_v2_preset.
- CHANGELOG 0.23.0: correct tag link + date, fold Changed/Removed into Added.

Tests: pagination edge cases (missing/0/over-counting total, short page),
sync create/version (no job_id), cross-dataset dedupe, path-traversal
sanitization, media-less skip, and empty-DELETE-body handling.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread nucleus/training_set.py Outdated
…ownload

Greptile follow-up: os.path.basename collapsed distinct reference_ids that
share a basename (e.g. "camera_a/frame" and "camera_b/frame" in one dataset)
to the same filename, so one silently overwrote the other while download_items
counted both.

Flatten path separators to "_" instead of taking basename (keeps distinct
reference_ids distinct while still preventing traversal), and guarantee no
member ever overwrites another with a used-path set that falls back to the
globally-unique dataset_item_id on any residual name collision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread nucleus/training_set.py Outdated
Greptile follow-up: if a member's reference-derived path collided AND its
dataset_item_id fallback path was itself already taken, the fallback write
overwrote the earlier file while both were counted. Loop a numeric suffix
until the path is unused, so no member can ever clobber another.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread nucleus/training_set.py
luke-e-schaefer and others added 2 commits September 2, 2026 17:14
Greptile flagged that download_items overwrites a destination path already
present on disk (e.g. from a prior call). This is intended, idempotent
re-download behavior — pre-seeding used_paths from existing files would break
it (every re-run would accumulate _1/_2 suffixes). Document the overwrite
semantics instead of changing behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread nucleus/training_set.py
# disambiguate with the globally-unique dataset_item_id, then a
# numeric suffix as a last resort (in case that fallback path is
# itself already taken), until the path is unused.
if path in used_paths:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Case-aliased downloads still overwrite

If download_items runs on a case-insensitive filesystem, members whose generated paths differ only by case, such as Frame.jpg and frame.jpg, bypass the exact-string used_paths check. The second transfer then replaces the first filesystem entry while the method reports both members as downloaded.

Prompt To Fix With AI
This is a comment left during a code review.
Path: nucleus/training_set.py
Line: 330

Comment:
**Case-aliased downloads still overwrite**

If `download_items` runs on a case-insensitive filesystem, members whose generated paths differ only by case, such as `Frame.jpg` and `frame.jpg`, bypass the exact-string `used_paths` check. The second transfer then replaces the first filesystem entry while the method reports both members as downloaded.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

@luke-e-schaefer
luke-e-schaefer requested a review from a team September 3, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant