Update SPHEREx tutorials for DR1

.gitignore

@@ -55,6 +55,10 @@ _build/
 #pycache
 __pycache__
 
+# Local Python virtual environments
+.venv/
+venv/
+
 # Local working notes, not part of the tutorials
 CLAUDE.md
 OU_PR_REVIEW.md

tutorials/spherex/spherex.md

@@ -4,7 +4,8 @@
 SPHEREx observes the sky from roughly 0.75–5.0 µm using a single instrument that provides low-resolution spectroscopy (R ≈ 40–150) in hundreds of spectral channels for every point on the sky.
 Its science goals span cosmology, galaxy evolution, and the interstellar medium, enabling measurements of large-scale structure, the cosmic history of star formation, and the distribution of key molecules and ices in the Milky Way and nearby galaxies.
 
-SPHEREx data releases include weekly [Quick Release spectral image products](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-products/) (multi-extension FITS files containing calibrated near-infrared surface brightness, variance, flags, modeled backgrounds, PSFs, and wavelength WCS) along with ancillary calibration and metadata files such as gain matrices, dark current maps, solid angle pixel maps, and detailed spectral WCS products for each detector.
+SPHEREx data releases include weekly [Quick Release spectral image products](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-products/) as well as periodic Data Releases (e.g., DR1) (multi-extension FITS files containing calibrated near-infrared surface brightness, variance, flags, modeled backgrounds, PSFs, and wavelength WCS) along with ancillary calibration and metadata files such as gain matrices, dark current maps, solid angle pixel maps, and detailed spectral WCS products for each detector.
+QR3 and DR1 supersede QR2, and SPHEREx recommends using QR3 and DR1 over QR2.
 
 ```{notebook-gallery} notebook_metadata.yml
 tutorials/spherex/spherex_intro.md

tutorials/spherex/spherex_cutouts.md

@@ -38,7 +38,7 @@ SPHEREx data will be used to:
 
 The community will also mine SPHEREx data and combine it with synergistic data sets to address a variety of additional topics in astrophysics.
 
-More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
+More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).
 
 ## 3. Imports
 
@@ -104,21 +104,33 @@ output_filename = 'spherex_cutouts_mef.fits'
 Here we show how to use `astroquery`'s IRSA TAP interface to retrieve all images that overlap with the position defined above.
 This query will retrieve a table of URLs that link to the MEF cutouts.
 Each row in the table corresponds to a single cutout and includes the data access URL and an observation timestamp.
+The SPHEREx metadata tables include images from every release.
+QR3 and DR1 supersede QR2, and SPHEREx recommends using QR3 and DR1 over QR2; this query keeps only DR1 images.
 The results are sorted from oldest to newest.
 
 ```{code-cell} ipython3
+# TESTING ONLY -- DR1 is not yet on production. Point at irsatest for now and
+# revert IRSA_HOST to "https://irsa.ipac.caltech.edu" before merging at DR1 release.
+# Note: `Irsa.tap_url` must be set directly -- `astroquery.ipac.irsa.conf.tap_url` is only read
+# once when the module is first imported, so changing it afterward has no effect.
+IRSA_HOST = "https://irsatest.ipac.caltech.edu"
+Irsa.tap_url = f"{IRSA_HOST}/TAP"
+
 # Define a query that will search the appropriate SPHEREx metadata tables
 # for spectral images that cover the chosen coordinates and match the
 # specified bandpass. Return the cutout data access URL and the time of observation.
 # Sort by observation time.
+# spherex.plane.datarelease is a date, not a release name, so DR1 rows are
+# selected by matching the DR1 path in the artifact URI instead.
 query = f"""
 SELECT
-    'https://irsa.ipac.caltech.edu/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
+    '{IRSA_HOST}/' || a.uri || '?center={coord.ra.deg},{coord.dec.deg}d&size={size.value}' AS uri,
     p.time_bounds_lower
 FROM spherex.artifact a
 JOIN spherex.plane p ON a.planeid = p.planeid
 WHERE 1 = CONTAINS(POINT('ICRS', {coord.ra.deg}, {coord.dec.deg}), p.poly)
         AND p.energy_bandpassname = '{bandpass}'
+        AND a.uri LIKE 'ibe/data/spherex/dr1/%'
 ORDER BY p.time_bounds_lower
 """
 
@@ -166,7 +178,7 @@ def process_cutout(row, coord, cache):
     with fits.open(row["uri"], cache=cache) as hdulist:
         # There are seven HDUs:
         # 0 contains minimal metadata in the header and no data.
-        # 1 through 6 are: IMAGE, FLAGS, VARIANCE, ZODI, PSF, WCS-WAVE
+        # 1 through 6 are: IMAGE, FLAGS, VARIANCE, ZODI, EPSF, WCS-WAVE
         header = hdulist[1].header
 
         # Compute pixel coordinates corresponding to cutout position.

tutorials/spherex/spherex_intro.md

@@ -44,7 +44,8 @@ The community will also mine SPHEREx data and combine it with synergistic data s
 
 More information about SPHEREx, the data releases, and the data products is available in the [SPHEREx Archive at IRSA User Guide](https://caltech-ipac.github.io/spherex-archive-documentation).
 
-This notebook can be used with both Quick Release 3 (QR3) and Quick Release 2 (QR2) data.
+This notebook uses data from SPHEREx Data Release 1 (DR1).
+QR3 and DR1 supersede the earlier Quick Release 2 (QR2), and SPHEREx recommends using QR3 and DR1 over QR2.
 
 +++
 
@@ -109,15 +110,13 @@ The IRSA SIA collections can be listed using using the ``list_collections`` meth
 +++
 
 The collections are documented at [SPHEREx Data Access: Application Program Interfaces (APIs)](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-access#application-program-interfaces-apis)
-There are currently four collections available:
+The DR1 Spectral Image MEFs are available in two collections:
 
-* `'spherex_qr3'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
-* `'spherex_qr3_deep'` -- Quick Release 3 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
-* `'spherex_qr2'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
-* `'spherex_qr2_deep'` -- Quick Release 2 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
+* `'spherex_dr1'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Wide Survey**
+* `'spherex_dr1_deep'` -- DR1 Spectral Image MEFs that are part of the SPHEREx **Deep Survey**
 
 ```{code-cell} ipython3
-results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_qr2')
+results = Irsa.query_sia(pos=(coord, search_radius), collection='spherex_dr1')
 ```
 
 :::{note}
@@ -188,7 +187,7 @@ You can see that the Level 2 Spectral Image files are multi-extension FITS files
 * **FLAGS:** A bitmap with per-pixel status and processing flags.
 * **VARIANCE:** A per-pixel estimate of the variance.
 * **ZODI:** An model of the zodiacal dust background signal. Note that this is not subtracted from the IMAGE extension.
-* **PSF:** An image cube (3D array) where each plane represents a Point Spread Function (PSF) for a cutout in over-sampled pixel space.
+* **EPSF:** A table of effective Point Spread Functions (ePSFs), one for each of 441 regions of the detector, in over-sampled pixel space.
 * **WCS-WAVE:** Spectral WCS lookup table that maps pixel coordinates to the central wavelength and bandwidth of each pixel.
 
 We will tour each extension in the sections below.
@@ -362,7 +361,7 @@ Now let's take a look at some header keywords that provide information about how
 spectral_image_header['L2 N_*']
 ```
 
-There are 14 flags in total. Typically, most of the pixels are identified as SOURCE pixels, which are pixels mapped to a known source. The remaining flags are described in Table 8 of the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
+These keywords count the pixels carrying most of the flags defined in the FLAGS header (see section 9), plus a few combined counts: keywords ending in `_NB` count pixels that are not also flagged as bad, and those ending in `_NS` count pixels that are not also flagged SOURCE. Typically, most of the pixels are identified as SOURCE pixels, which are pixels mapped to a known source. The flags are described in Table 16 of the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).
 
 +++
 
@@ -382,10 +381,12 @@ for keyword in keywords_to_print:
     print(f"{keyword}: {value}")
 ```
 
-Conveniently, the definitions of the flags are also provided in the FLAGS header.
+Conveniently, the definitions of the flags are also provided in the FLAGS header. For each bit number n, the keyword `MSKNnnnn` gives the flag name and `MSKDnnnn` gives its description.
 
 ```{code-cell} ipython3
-flags_header['MP*']
+for key in flags_header['MSKN*']:
+    bit = key[4:]
+    print(f"{int(bit):2d}  {flags_header[key]:<13} {flags_header['MSKD' + bit]}")
 ```
 
 Tip: In the Firefly visualization, if you are looking at the first extension (IMAGE), you can open up the layers icon (top right) to enable overlaying a visualization of the flags.
@@ -420,14 +421,14 @@ for keyword in keywords_to_print:
     print(f"{keyword}: {value}")
 ```
 
-## 12. Explore the fifth extension: PSF
+## 12. Explore the fifth extension: EPSF
 
-The fifth extension of the MEF contains 121 Point-spread functions (PSFs); each PSF is represented as a 101 x 101 image and all 121 are assembled together into a cube. Each of the 121 layers represents a "super-resolution" PSF estimate in a different region (defined by an 11x11 grid) of the detector. Each PSF is a two-dimensional array with size of 101 × 101 pixels. The PSFs are oversampled such that 10 PSF pixels cover the same spatial extent as one spectral image pixel (0.615 arcsec). Let's look at some specific header keywords for the PSF:
+The fifth extension of the MEF contains the effective Point Spread Function (ePSF), stored as a binary table. The detector is divided into a 21 x 21 grid of regions (each about 97 x 97 spectral image pixels), and each of the 441 rows of the table holds the ePSF for one region, along with that region's position on the detector (`XCENTER`, `YCENTER`) and its central wavelength and bandwidth (`CWAVE`, `CBAND`). Each ePSF is a 33 x 33 array, oversampled by a factor of 5 relative to the spectral image pixels (header keywords `OVSMPX` and `OVSMPY`). Unlike an optical PSF, the ePSF is a model of the pixel-convolved PSF: it predicts the pixel values of a point source at any sub-pixel position. The [SPHEREx ePSF tutorial](spherex_psf.md) shows how to find and use the ePSF for a given position. Let's look at some specific header keywords for the EPSF:
 
 ```{code-cell} ipython3
-psf_header = hdulist[5].header
+psf_header = hdulist["EPSF"].header
 
-keywords_to_print = ['EXTNAME', 'NAXIS1', 'NAXIS2', 'NAXIS3', 'BUNIT']
+keywords_to_print = ['EXTNAME', 'NAXIS2', 'DETECTOR', 'OVSMPX', 'OVSMPY']
 
 for keyword in keywords_to_print:
     value = psf_header.get(keyword, 'Keyword not found')

tutorials/spherex/spherex_mosaic.md

@@ -42,7 +42,7 @@ SPHEREx data will be used to:
 
 The community will also mine SPHEREx data and combine it with synergistic data sets to address a variety of additional topics in astrophysics.
 
-More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
+More information is available in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp.pdf).
 
 +++