Add a Claude Code skill for SPHEREx data-release notebook updates

.claude/skills/spherex-data-release/SKILL.md

@@ -0,0 +1,124 @@
+---
+name: spherex-data-release
+description: Update the SPHEREx tutorial notebooks in irsa-tutorials for a new SPHEREx data release (DR2, DR3, a new Quick Release, a reprocessing). Use this whenever someone mentions a new SPHEREx release, new SPHEREx collections or DOIs, a new SPHEREx Explanatory Supplement, new FLAGS bits, or asks to "update the SPHEREx notebooks", "test SPHEREx against irsatest", or prepare the SPHEREx release-day PR -- even if they only ask about one of those pieces.
+---
+
+# SPHEREx data-release update
+
+This skill repeats the work done for DR1 (issue #392, released 2026-11-12): find every place in the
+SPHEREx notebooks that depends on the release, test against pre-release IRSA hosts, collect the
+answers that only other people can give, and land everything in one PR on release day.
+
+Most of the time goes into waiting on other people, not editing code. So the main job is to keep a
+written plan current, ask for missing information early, and do everything that does not depend on
+those answers right away.
+
+## Files in this skill
+
+- `references/touchpoints.md` -- every release-dependent spot in each notebook, with what it looked
+  like for DR1. Read this first; it is the work list.
+- `references/decisions.md` -- rules settled during DR1 (which release to show, which DOI to cite,
+  which flags count as bad, who owns which notebook) and gotchas that cost time. Read before
+  asking the user anything; most questions were already answered once.
+- `scripts/check_release.sh <rel>` -- what is live for release `<rel>` (e.g. `dr2`) on the
+  pre-release and production hosts, S3 buckets and docs.
+- `scripts/run_sdt_release_test.py` -- executes the whole Source Discovery Tool notebook against
+  pre-release data and prints a summary of what to check.
+
+## Workflow
+
+### 1. Start a plan file
+
+Create `SPHEREX_<REL>_PLAN.md` at the repo root and add it (and any local test folder) to
+`.git/info/exclude` so it never gets committed. Put a short "RESUME HERE" block at the top: branch,
+last commit, what is blocked and on whom, next actions. Update it at the end of every work chunk.
+The work spans weeks and many sessions, and this file is the only thing that carries over.
+
+Work on a branch named `spherex-<rel>` off `main`. Push it to the user's fork only until release;
+the test-host settings must not reach `main` early.
+
+### 2. Find out what exists
+
+Run `scripts/check_release.sh <rel>` (needs IRSA VPN for the pre-release hosts). It reports, for
+TAP, SIA, IBE, S3 and the supplement PDF, whether the new release is visible. Record the results as
+a "data products needed" table in the plan (product, which notebook uses it, test location,
+available yes/no, date checked). Re-run it whenever something might have changed.
+
+### 3. Ask early for what only people can provide
+
+Send these questions as soon as the release date is known. Draft each as a paste-ready message
+in a fenced code block. Keep each one short and specific.
+
+- **IRSA** (archive/backend staff): Will the new SIA collections be loaded on the pre-release host
+  (irsadev) before release, so the SIA notebooks can be tested? What will the collections be
+  named? Will `cloud_access` be filled in those rows? (The Source Discovery Tool reads its S3
+  paths from it. Check it in SIA results, e.g. with `scripts/check_release.sh`: TAP's
+  `ivoa.obscore` leaves it blank even when SIA fills it.) Will the data be on the staging
+  S3 bucket, and when does it move to the public one?
+- **IRSA** (DOIs): the DOI for the new release. It is usually not available until close to release.
+- **SSDC**: a copy of the new Explanatory Supplement, or confirmation that the online one is
+  current. Ask about any new FLAGS bits and which ones should count as bad for photometry.
+
+Then decide with the user what happens if the pre-release SIA load does not come through
+(for DR1 the fallback was letting the tutorials go out a few days after release). Set a date to
+follow up and write it in the plan.
+
+### 4. Make the changes that do not depend on answers
+
+Go through `references/touchpoints.md` notebook by notebook. For each one, make the change, then
+test it against the pre-release host. In order of how early they can be done:
+
+1. TAP notebooks (`spherex_cutouts.md`, `spherex_psf.md`, SDT sec. 12): point at the pre-release
+   host through an `IRSA_HOST` variable inside a clearly marked `# TESTING ONLY` block, and add the
+   release filter on `a.uri`. Run them end to end with
+   `jupytext --to notebook --execute <file>`.
+2. Header-driven code and text: open one new-release image and compare its extensions, FLAGS
+   `MSKN*` keywords and `L2 N_*` counts with what the notebooks say and parse (see
+   touchpoints). New flags or renamed keywords break code silently -- the DR1 photometry recorded
+   no flags at all until the parser was fixed.
+3. SIA notebooks (`spherex_intro.md`, SDT secs. 6-11): switch to the guessed collection names now
+   (pattern below), so only a check is left for release day.
+4. Narrative: supplement link and table number, DOIs, release wording on the landing page.
+
+Collection names follow the pattern of the previous release: for DR1 they were guessed as
+`spherex_dr1`, `spherex_dr1_deep`, `spherex_dr1_cal` from `spherex_qr3`, `spherex_qr3_deep`,
+`spherex_qr3_cal`. Put a "verify the guessed names" step in the release-day checklist with the
+exact lines to change if they differ.
+
+Commit in small, single-purpose commits and push to the fork. Ask before committing, and say in
+the plan what each commit did.
+
+### 5. Test the Source Discovery Tool
+
+Run the whole notebook, not selected cells. A DR1 test that ran only some cells and stood in a
+2MASS star for secs. 9-11 passed while sec. 6 was silently picking a D5 image instead of D4
+(see decisions.md). `scripts/run_sdt_release_test.py` executes every cell on a copy, with the
+pre-release host's SIA, its IBE mirror in place of S3, and its Firefly viewer. It needs the
+release's SIA collection on that host and SExtractor's `sex` command (`brew install sextractor`
+is enough; the notebook's conda env is not needed). Run it from the repo root:
+
+```bash
+.venv/bin/python .claude/skills/spherex-data-release/scripts/run_sdt_release_test.py \
+  tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool_demo.md --release dr2 \
+  --out <your excluded test folder>/sdt_run
+```
+
+Check its summary: sec. 6 must pick D2 and D4, the hard-coded `selected_ids` source must exist,
+and its spectrum should look like a real source. Ask the user to look at the two Firefly tabs.
+For DR1: 33 notebook code cells, 0 errors; 79 SExtractor sources, #65 was 40th by flux; 276
+points, 0.75-4.99 um, median S/N about 11, 50 points with a `bad_flags` pixel in the aperture.
+
+### 6. Release day
+
+Keep a release-day checklist in the plan from the start and refine it as answers arrive. Its
+final shape for DR1 is in `references/decisions.md` under "Release-day checklist". Open a draft PR
+well before release so it can be reviewed, with the checklist in the PR description. Merge any
+pending cleanup PRs into the release branch first so the diff shows only release work.
+
+As the last checklist item, update this skill with what was learned (see below).
+
+## Keeping this skill current
+
+After each release, update `references/touchpoints.md` with the new values and line numbers, add
+new decisions and gotchas to `references/decisions.md`, and fix anything in this file that was
+wrong. Leave out people's names: say which group to ask (IRSA, SSDC), not who.

.claude/skills/spherex-data-release/references/decisions.md

@@ -0,0 +1,80 @@
+# Decisions and gotchas from DR1
+
+## Rules (settled; don't re-ask unless something changed)
+
+- **Show only the newest release.** Older collections stay searchable, but the tutorials
+  demonstrate only the newest one. TAP notebooks filter on `a.uri LIKE 'ibe/data/spherex/<rel>/%'`.
+- **DOIs:** cite only the DOI of each release the notebook actually loads (both if it loads two).
+- **`bad_flags` in the Source Discovery Tool:** use the flag list that sets the supplement's
+  CONTAINS_BAD_PIXEL warning (supplement v2.0, p. 43). This came from the SSDC developer who wrote
+  the SDT. Do not use the shorter "flags for masking pixels for photometry calculations" list on
+  the same page. For DR1 the list was: TRANSIENT, OVERFLOW, SUR_ERROR, NONFUNC, DICHROIC,
+  MISSING_DATA, HOT, COLD, NONLINEAR, CROSSTALK, GHOST, GHOST_FPA, GHOST_EXT, STREAK, PERSIST,
+  BLOOM, SNOWBALL, HALO, SATELLITE_HALO. FULLSAMPLE is informational and was removed.
+- **Supplement:** link `SPHEREx_Expsupp.pdf` (the `_QR` suffix was dropped for DR1). For flag
+  definitions, the online supplement is the reference -- not plans for a future release. Cite
+  the flag table by number (Table 16 in v2.0) and recheck the number when a new version appears.
+- **Ownership:** IRSA updates the Source Discovery Tool notebook, not SSDC.
+- **Test hosts** go in a `# TESTING ONLY` block with an `IRSA_HOST` variable, so release day is a
+  mechanical revert. Push only to a personal fork until release.
+- **`cloud_access`:** assume the new SIA collection fills it. If a pre-release collection has it
+  empty, build the S3 path from the URI for local testing only, and don't commit that.
+- **Demo mosaic cube:** not regenerated per release.
+
+## Gotchas
+
+- irsatest/irsadev are reachable only on the IRSA network or VPN. Cloud notebook servers get 403 or
+  cannot resolve them.
+- IRSA production TAP/SIA can return 502 for a while. For non-SPHEREx lookups in tests (like
+  2MASS), irsatest works too.
+- `spherex.plane.datarelease` is a date. Filter by URI path or `obs_publisher_did`.
+- Each release has a `_cal` SIA collection listing calibration files (ePSF, SAPM) per detector --
+  the quickest way to learn calibration version names.
+- IBE mirrors the S3 paths, so `s3://nasa-irsa-spherex/<rel>/...` reads can be pointed at
+  `https://<host>/ibe/data/spherex/<rel>/...` for testing.
+- Pre-release S3 data sat in a staging bucket (`nasa-irsa-spherex-dr1-staging`) that moved to the
+  public `nasa-irsa-spherex` on release day.
+- FLAGS keywords changed between releases (`MP_*` in QR2, `MSKN*`/`MSKD*` from QR3). New flags
+  can appear in the supplement before they appear in image headers (SATELLITE_HALO, bit 29, was in
+  supplement v2.0 but not in DR1 v7.0.5 headers) -- code must tolerate missing flag names.
+- To test SIA notebooks against irsadev, set `Irsa.sia_url = "https://irsadev.ipac.caltech.edu/SIA"`
+  in a temporary copy (don't commit it). `Irsa.list_collections()` still lists production
+  collections, so check irsadev collections with a TAP query on `ivoa.obscore` instead. irsadev SIA
+  `access_url`s point at a dev IBE host. IRSA may load only part of the sky, so confirm the notebook
+  positions are covered.
+- Select detectors by name, never by position. The SDT once took D4 as
+  `results.group_by('energy_bandpassname').groups[3]`, which is D4 only if all six detectors have
+  images at the position. DR1 at the demo position had D1, D2, D4 and D5, so it silently used D5
+  with every label still saying D4. Fixed for DR1 by matching `groups.keys` to `'SPHEREx-D4'`.
+- The SDT's sec. 12.1 measures a hard-coded SExtractor source number (`selected_ids = [65]`). The
+  numbering changes whenever the images change, so check the number on every release, including
+  release day. For DR1 it was left as 65 (a real source, mid-brightness) with a release-day check.
+- Production Firefly (irsa.ipac.caltech.edu/irsaviewer) can't load images from pre-release hosts:
+  it shows "failed". Use the same host's viewer (`https://irsatest.ipac.caltech.edu/irsaviewer`)
+  when testing, and check production Firefly on release day.
+- MOST on the pre-release hosts doesn't index the new release, so `MOST_queries.md` can't be
+  tested end to end before release. Its SPHEREx cutout cell can be: take the exposures production
+  MOST returns, find the new-release file with the same observation ID
+  (`level2_<week>_<n>_<det>`) on the pre-release IBE, and run the cell on it.
+- Expect overlapping cleanup PRs. Merge them into the release branch before opening the release
+  PR, or the same changes show up twice.
+
+## Release-day checklist (DR1's final shape)
+
+1. Confirm the release is live: `scripts/check_release.sh <rel>`; verify the guessed SIA collection
+   names and fix intro + SDT if they differ; confirm production SIA rows have `cloud_access`.
+2. Revert test hosts: delete each `# TESTING ONLY` block and `IRSA_HOST`, inline
+   `https://irsa.ipac.caltech.edu`, and check that
+   `grep -rn "irsatest\|irsadev\|IRSA_HOST" tutorials/` finds nothing.
+3. Put in the new DOI (SDT acknowledgements).
+4. Bump `**Updated:**` in every notebook that changed.
+5. Run each notebook against production with `jupytext --to notebook --execute` (SDT needs
+   SExtractor: its conda env, `conda-spherex_sdt.yml`, or `brew install sextractor`). Check the
+   Firefly tabs show images, and that the SDT's hard-coded `selected_ids` still gives a real
+   spectrum.
+6. `techniques-and-tools/MOST_queries.md` belongs to other authors and isn't in the release PR.
+   For DR1 a separate issue asked them to re-run it once MOST serves the release and check which
+   release MOST returns.
+7. Recheck the supplement's flag table number.
+8. Mark the draft PR ready for review (it closes the release issue).
+9. Update this skill with what was learned.

.claude/skills/spherex-data-release/references/touchpoints.md

@@ -0,0 +1,85 @@
+# Release-dependent spots in the SPHEREx notebooks
+
+Values shown are what DR1 used (branch `spherex-dr1`, Nov 2026). Line numbers drift; search for the
+quoted text instead. Paths are relative to `tutorials/spherex/`.
+
+## spherex_cutouts.md (TAP + IBE cutouts)
+
+- `IRSA_HOST` test block (`# TESTING ONLY`): sets `IRSA_HOST` and `Irsa.tap_url`. Note
+  `Irsa.tap_url` must be set directly; `astroquery.ipac.irsa.conf.tap_url` is read only at import.
+- TAP release filter: `AND a.uri LIKE 'ibe/data/spherex/dr1/%'`. `spherex.plane.datarelease` is a
+  date, not a release name, so filter on the URI path. Without a filter the query mixes releases
+  and `results[...][0]` picks the oldest.
+- Cutout URL prefix built from `{IRSA_HOST}/` || `a.uri`.
+- Supplement link (`SPHEREx_Expsupp.pdf`).
+- `**Updated:**` date at the bottom.
+
+## spherex_psf.md (TAP + IBE cutouts + S3)
+
+- Same `IRSA_HOST` block, release filter and cutout URL prefix as cutouts.
+- Sec. 5.1 reads the image's own `EPSF` extension (image version >= 7.0). Sec. 5.2 is a fallback
+  for older images that reads ePSF calibration files from S3 (`<rel>/epsf/cal-epsf-v*-*`). It must
+  stay guarded so it does not overwrite a new image's own ePSF (a DR1 bug).
+- "primary HDU plus 6 extensions (IMAGE, FLAGS, VARIANCE, ZODI, EPSF, WCS-WAVE)" -- check against a
+  new image.
+- Supplement link, `**Updated:**`.
+
+## spherex_intro.md (SIA)
+
+- Collection list and the `Irsa.query_sia(..., collection='spherex_dr1')` call; bullet list of
+  "two collections" (wide `spherex_dr1`, deep `spherex_dr1_deep`). Show only the newest release.
+- "This notebook uses data from SPHEREx Data Release 1 (DR1)."
+- Sec. 6 extension list and sec. 12 describe the `EPSF` bintable (DR1: 441 rows = 21x21 regions,
+  33x33 ePSF, 5x oversampled via `OVSMPX`/`OVSMPY`; columns XCENTER/YCENTER/CWAVE/CBAND).
+- Sec. 8d: prints `spectral_image_header['L2 N_*']` and explains it. DR1 had 26 keys: one per flag
+  for most flags (STREAK has none), `_NB` = not also flagged bad, `_NS` = not also flagged SOURCE,
+  `N_SAT_HALO` = SATELLITE_HALO. Cites the supplement's flag table (Table 16 in supplement v2.0).
+- Sec. 9: loops over FLAGS header `MSKNnnnn` (name) / `MSKDnnnn` (description). QR2 used `MP_*`.
+  DR1 defined 22 flags.
+- Supplement link, `**Updated:**`.
+
+## spherex_source_discovery/spherex_source_discovery_tool_demo.md (SIA + S3 + TAP)
+
+- Sec. 5 parameters: `data_release = "dr1"`, `sia_collection = f"spherex_{data_release}"`,
+  `sapm_collection = "cal-sapm-v3-2026-191"`. `data_release` also builds the S3 path
+  `nasa-irsa-spherex/{data_release}/solid_angle_pixel_map/{sapm_collection}/...` (folder is
+  `solid_angle_pixel_map/`, not `sapm/`). If only the SIA name changes, set `sia_collection`
+  directly. Remove any "valid as of <date>" note that no longer holds.
+- "Note that we have used the collection `spherex_dr1`" sentence after the SIA query.
+- Sec. 6 picks the D2 and D4 images by `energy_bandpassname` (D4 through `groups.keys`, since DR1).
+  Check that the new release's search still returns both detectors at the demo position.
+- Sec. 7 `get_s3_fpath(cloud_access)`: needs `cloud_access` filled in SIA results.
+- Sec. 8 masks: D2 mask from `flags_data2`, D4 mask from `flags_data4` (DR1 fixed a typo here).
+- Sec. 12.1 TAP query: `IRSA_HOST` test block (uses `pyvo.dal.TAPService(f"{IRSA_HOST}/TAP")`),
+  release filter, cutout URL prefix.
+- Sec. 12.1 `selected_ids = [65]`: a SExtractor source number from sec. 10, so it changes when the
+  images do. Check it still names a real source.
+- Secs. 6 and 11.2 use `FireflyClient.make_lab_client()`, which works only inside JupyterLab.
+- Sec. 12.2 `bad_flags`: the supplement's CONTAINS_BAD_PIXEL list (see decisions.md). Any listed
+  flag in the aperture drops the point from the plotted spectrum (`bokeh_viz.plot_spectrum`).
+- Acknowledgements: DOI sentence. Cite only the DOIs of releases the notebook actually loads.
+- Supplement link, `**Updated:**`.
+- Package `spherex_source_discovery_tool/aperture_photometry.py`:
+  `build_phot_table` reads flag names from `MSKN*` (falls back to `MP_*`); `grab_star` skips
+  `bad_flags` entries an image does not define. Check both against new FLAGS headers.
+
+## spherex_mosaic.md
+
+- Supplement link. The static `M101_irsa.fits` demo cube was not regenerated for DR1.
+
+## spherex.md (landing page)
+
+- Wording about Quick Releases and Data Releases.
+
+## techniques-and-tools/MOST_queries.md
+
+- Not a SPHEREx tutorial (other authors) and not changed for DR1. Sec. 5 queries MOST with
+  `catalog="spherex"` (Vesta, Aug 2025); `plot_path()` uses only MOST's output columns; the cutout
+  cell reads `hdul[1]` and `PC1_1`/`PC1_2`. No release-specific code. Before DR1, production MOST
+  returned QR2 images; whether it returns the new release, the old one, or both is only known
+  after release (see decisions.md for the pre-release check).
+
+## Repo files
+
+- `deploy_to_fornax_manifest.in` and `notebook_metadata.yml` list the SPHEREx files; update them if
+  files are added, renamed or removed (a stale manifest entry fails CI).