From b787f6ee59eb826c7733847919d907374d40cc1a Mon Sep 17 00:00:00 2001 From: xoubish Date: Mon, 5 Oct 2026 17:19:10 -0700 Subject: [PATCH 1/5] notebook 1 (MER mosaics) updated to DR1 --- tutorials/euclid/1_Euclid_intro_MER_images.md | 334 ++++++++++++------ 1 file changed, 223 insertions(+), 111 deletions(-) diff --git a/tutorials/euclid/1_Euclid_intro_MER_images.md b/tutorials/euclid/1_Euclid_intro_MER_images.md index 2a3a84ad..2d570c01 100644 --- a/tutorials/euclid/1_Euclid_intro_MER_images.md +++ b/tutorials/euclid/1_Euclid_intro_MER_images.md @@ -22,7 +22,17 @@ authors: - name: Vandana Desai --- -# Euclid Q1: MER Mosaics +# Euclid DR1: MER Mosaics + +```{danger} TODO before the DR1 release (remove this box when done) +**1. Remove the cell that points `Irsa.sia_url` to irsadev, once DR1 is public on the production IRSA services.** + +**2. Switch to, or add, `access_url` downloads once IRSA redirects the DR1 `access_url` links to the cloud (expected in about a week).** + +**3. Update the Kümmel et al. reference (journal, volume, arXiv ID), confirm it can be cited, and verify the DR1 DOI.** + +**4. Re-run the notebook against production IRSA and check the outputs and plots.** +``` +++ @@ -31,10 +41,13 @@ authors: +++ By the end of this tutorial, you will: -- understand the basic characteristics of Euclid Q1 MER mosaics; -- know how to download full MER mosaics; +- understand the basic characteristics of Euclid DR1 MER mosaics; +- understand the difference between wide and deep MER tiles; +- know how to identify the official wide survey mosaic when a tile has several mosaics per band; +- know how to access full MER mosaics from the cloud; - know how to make smaller cutouts of MER mosaics; - know how to use matplotlib to plot a grid of cutouts; +- know how to make a color composite from the Euclid VIS and NISP cutouts; - know how to identify sources in the cutouts and make basic measurements. +++ @@ -48,20 +61,37 @@ The primary science goals of Euclid are to better understand the composition and The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. -Euclid Quick Release 1 (Q1) consists of consists of ~30 TB of imaging, spectroscopy, and catalogs covering four non-contiguous fields: -Euclid Deep Field North (22.9 sq deg), Euclid Deep Field Fornax (12.1 sq deg), Euclid Deep Field South (28.1 sq deg), and LDN1641. +Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 63 sq deg in the three Euclid Deep Fields (North, South, and Fornax). +DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. +It follows Quick Release 1 (Q1), which covered four much smaller fields; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. -Among the data products included in the Q1 release are the Level 2 MER mosaics. -These are multiwavelength mosaics created from images taken with the Euclid instruments (VIS and NISP), as well as a variety of ground-based telescopes. -All of the mosaics have been created according to a uniform tiling on the sky, and mapped to a common pixel scale. +Among the data products included in DR1 are the Level 2 MER mosaics. +These are multiwavelength mosaics created from images taken with the Euclid instruments (VIS and NISP), as well as a variety of ground-based surveys (DES, CFIS, Pan-STARRS, and HSC-based surveys) that provide the u, g, r, i, and z bands. +All of the mosaics have been created according to a uniform tiling on the sky, and mapped to a common pixel scale of 0.1 arcsec. +The MER processing for DR1 is described in detail in Euclid Collaboration: Kümmel et al. (2026), see the [References](#references) section below. This notebook provides a quick introduction to accessing MER mosaics from IRSA. If you have questions about it, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). +++ +## Wide and deep MER tiles + +DR1 MER mosaics come in two kinds of tiles, which IRSA serves as two separate collections: + +* **Wide tiles** (`euclid_dr1_wide_DpdMerBksMosaic`): tile IDs start with `102`, about 32 x 32 arcmin, the same tiling as in Q1. +* **Deep tiles** (`euclid_dr1_deep_DpdMerBksMosaic`): tile IDs start with `101`, about 17 x 17 arcmin, covering the deep fields. + They stack all of the deep field observations (about 8 times the VIS exposure time of a wide tile), so they reach much fainter sources. + +The deep fields are also covered by wide tiles, and because they have been observed many times, these wide tiles have **several mosaics in every band**. +Only one is the official wide survey mosaic; the others are each built from a single pass over the deep field. +The `PPLANID` keyword in the FITS header tells them apart: the official mosaic has `DATA_RELEASE` in it, the others `DATA_RELEASE_DEEP`. + ++++ + ## Data volume -Each MER image is approximately 1.47 GB. Downloading can take some time. +Each wide MER mosaic is approximately 1.47 GB, and each deep MER mosaic is approximately 0.42 GB. +Downloading can take some time. +++ @@ -73,11 +103,11 @@ We rely on ``astroquery`` and ``sep`` features that have been recently added, so ```{code-cell} ipython3 # Uncomment the next line to install dependencies if needed. -# !pip install numpy 'astropy>=5.3' matplotlib 'astroquery>=0.4.10' 'sep>=1.4' fsspec +# !pip install numpy 'astropy>=5.3' matplotlib 'astroquery>=0.4.10' 'sep>=1.4' fsspec s3fs ``` ```{code-cell} ipython3 -import re +import json import numpy as np @@ -87,88 +117,139 @@ from matplotlib.patches import Ellipse from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.nddata import Cutout2D -from astropy.utils.data import download_file +from astropy.stats import sigma_clipped_stats from astropy.visualization import ImageNormalize, PercentileInterval, AsinhStretch, ZScaleInterval, SquaredStretch from astropy.wcs import WCS from astropy import units as u from astroquery.ipac.irsa import Irsa +import s3fs import sep ``` -## 1. Search for multiwavelength Euclid Q1 MER mosaics that cover the star HD 168151 -Below we set a search radius of 10 arcsec and convert the name "HD 168151" into coordinates. + + +```{code-cell} ipython3 +# DR1 is not public yet, so we query IRSA's development server. +Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' +``` + +## 1. Search for multiwavelength Euclid DR1 MER mosaics in the Euclid Deep Field North +Below we set a search radius of 10 arcsec around a position in the Euclid Deep Field North, an area covered by both deep and wide tiles. ```{code-cell} ipython3 search_radius = 10 * u.arcsec -coord = SkyCoord.from_name('HD 168151') +coord = SkyCoord(273.1670, 64.3687, unit='deg', frame='icrs') ``` -Use IRSA's Simple Image Access (SIA) API to search for all Euclid MER mosaics that overlap with the search region you have specified. We specify the euclid_DpdMerBksMosaic "collection" because it lists all of the multiwavelength MER mosaics, along with their associated catalogs. +Use IRSA's Simple Image Access (SIA) API to search for all Euclid MER mosaics that overlap with the search region you have specified. ```{tip} -The IRSA SIA collections can be listed using using the ``list_collections`` method, we can filter on the ones containing "euclid" in the collection name: +The IRSA SIA collections can be listed using the ``list_collections`` method, we can filter on the ones containing "euclid" in the collection name: Irsa.list_collections(filter='euclid') ``` -There are currently four collections available: +We search both DR1 MER collections described above, one for the deep and one for the wide tiles. +IRSA serves other Euclid data products, as well as the Q1 and Early Release Observations data, in further collections. +For more information, please read the [IRSA User Guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf). -* `'euclid_ero'` -- the Early Release Observations, for more information see [Euclid ERO documentation at IRSA](https://irsa.ipac.caltech.edu/data/Euclid/docs/overview_ero.html). -* `'euclid_DpdMerBksMosaic'` -- This is the collection of interest for us in this notebook, it contains all the Q1 MER mosaic data. For more information on MER mosaics, see the "Introduction" above. -* `'euclid_DpdVisCalibratedQuadFrame'` -- This collection contains the VIS instrument Calibrated images. Please note these data are not stacked, so there are four dither positions for every observation ID and two calibration frames. -* `'euclid_DpdNirCalibratedFrame'` -- This collection contains the NIR instrument Calibrated images. Please note these data are not stacked, so there are four dither positions for every observation ID. +```{code-cell} ipython3 +deep_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_dr1_deep_DpdMerBksMosaic') +wide_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_dr1_wide_DpdMerBksMosaic') +``` -For more information on the VIS and NIR calibrated frames, please read the [IRSA User Guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf) +These tables list all MER mosaic images available in this search position. These mosaics include the Euclid VIS, Y, J, H images, as well as ground-based telescopes which have been put on the same pixel scale. For more information, see the [Euclid documentation at IPAC](https://euclid.caltech.edu/page/euclid-faq-tech/). -In this notebook, we will focus on downloading and visualizing the MER mosaic data. +Note that there are various image types are returned as well, we filter out the `science` images from these: ```{code-cell} ipython3 -image_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_DpdMerBksMosaic') +science_images = deep_table[deep_table['dataproduct_subtype'] == 'science'] +wide_science_images = wide_table[wide_table['dataproduct_subtype'] == 'science'] ``` -This table lists all MER mosaic images available in this search position. These mosaics include the Euclid VIS, Y, J, H images, as well as ground-based telescopes which have been put on the same pixel scale. For more information, see the [Euclid documentation at IPAC](https://euclid.caltech.edu/page/euclid-faq-tech/). +The tile ID is the first part of the `obs_id`: deep tile IDs start with `101` and wide tile IDs with `102`. +Adjacent tiles overlap, and our position happens to lie in the overlap of two deep tiles and of two wide tiles: -Note that there are various image types are returned as well, we filter out the `science` images from these: +```{code-cell} ipython3 +for name, images in [('Deep', science_images), ('Wide', wide_science_images)]: + tile_ids, n_vis = np.unique([obs_id[:9] for obs_id in images[images['energy_bandpassname'] == 'VIS']['obs_id']], return_counts=True) + print(f'{name} tiles and their number of VIS mosaics:', dict(zip(tile_ids.tolist(), n_vis.tolist()))) +``` + +Each deep tile has a single VIS mosaic, while the wide tiles have several. +For the deep tile, we keep the one whose center is closest to our position: ```{code-cell} ipython3 -science_images = image_table[image_table['dataproduct_subtype'] == 'science'] +tile_centers = SkyCoord(science_images['s_ra'], science_images['s_dec'], unit='deg') +deep_tileID = science_images['obs_id'][np.argmin(tile_centers.separation(coord))][:9] +science_images = science_images[[obs_id.startswith(deep_tileID) for obs_id in science_images['obs_id']]] + +print(f'Deep MER tile ID: {deep_tileID}') science_images ``` -## 2. Retrieve a Euclid Q1 MER mosaic image in the VIS bandpass +## 2. Locate the mosaics in the cloud -+++ - -### Let's first look at one example full image, the VIS image +IRSA hosts the DR1 MER mosaics in an AWS S3 bucket. +The `cloud_access` column of the search results gives the bucket name and file location of each mosaic, so let's extract the S3 file path from it: -Note that 'access_estsize' is in units of kb + ```{code-cell} ipython3 -filename = science_images[science_images['energy_bandpassname'] == 'VIS']['access_url'][0] -filesize = science_images[science_images['energy_bandpassname'] == 'VIS']['access_estsize'][0] / 1000000 -print(filename) +def get_s3_fpath(cloud_access): + cloud_info = json.loads(cloud_access) # converts str to dict + bucket_name = cloud_info['aws']['bucket_name'] + key = cloud_info['aws']['key'] -print(f'Please note this image is {filesize} GB. With 230 Mbps internet download speed, it takes about 1 minute to download.') -``` + return f'{bucket_name}/{key}' -```{code-cell} ipython3 -science_images + +science_images['s3_fpath'] = [get_s3_fpath(row['cloud_access']) for row in science_images] +wide_science_images['s3_fpath'] = [get_s3_fpath(row['cloud_access']) for row in wide_science_images] ``` -### Extract the tileID of this image from the filename +## 3. Identify the official wide survey VIS mosaic + +To find the official wide survey VIS mosaic, we read the `PPLANID` keyword from the header of each wide VIS mosaic returned by our search. +Reading only the header with a small block size avoids downloading the large image data. ```{code-cell} ipython3 -tileID = science_images[science_images['energy_bandpassname'] == 'VIS']['obs_id'][0][:9] +s3 = s3fs.S3FileSystem(anon=True) + + +def get_processing_plan(s3_fpath): + try: + with s3.open(s3_fpath, block_size=28800) as f: + return fits.Header.fromfile(f).get('PPLANID', '') + except FileNotFoundError: + # A few files listed in the search results may not be available in the bucket + return '' + -print(f'The MER tile ID for this object is : {tileID}') +wide_vis_images = wide_science_images[wide_science_images['energy_bandpassname'] == 'VIS'] +wide_vis_images['plan'] = [get_processing_plan(path) for path in wide_vis_images['s3_fpath']] + +wide_vis_images['official'] = ['-DATA_RELEASE-' in plan for plan in wide_vis_images['plan']] +wide_tileID = wide_vis_images[wide_vis_images['official']]['obs_id'][0][:9] + +wide_vis_images['obs_id', 'plan', 'official'] ``` -Retrieve the MER image -- note this file is about 1.46 GB +## 4. Retrieve a Euclid DR1 MER mosaic image in the VIS bandpass + ++++ + +### Let's first look at one example full image, the deep VIS image + +We open the full deep VIS mosaic directly from the S3 bucket. Note this file is about 0.42 GB. ```{code-cell} ipython3 -fname = download_file(filename, cache=True) -hdu_mer_irsa = fits.open(fname) +filename = science_images[science_images['energy_bandpassname'] == 'VIS']['s3_fpath'][0] +print(filename) + +hdu_mer_irsa = fits.open(f's3://{filename}', fsspec_kwargs={'anon': True}) print(hdu_mer_irsa.info()) header_mer_irsa = hdu_mer_irsa[0].header @@ -188,7 +269,7 @@ Have a look at the header information for this image. header_mer_irsa ``` -Lets extract just the primary image. +Lets extract just the primary image. This reads the full image from the cloud, so it may take a minute. ```{code-cell} ipython3 im_mer_irsa = hdu_mer_irsa[0].data @@ -196,11 +277,13 @@ im_mer_irsa = hdu_mer_irsa[0].data print(im_mer_irsa.shape) ``` -Due to the large field of view of the MER mosaic, let's cut out a smaller section (2"x2")of the MER mosaic to inspect the image +Due to the large field of view of the MER mosaic, let's cut out a smaller section (2'x2') around our search position to inspect the image ```{code-cell} ipython3 -plt.imshow(im_mer_irsa[0:1200,0:1200], cmap='gray', origin='lower', - norm=ImageNormalize(im_mer_irsa[0:1200,0:1200], interval=PercentileInterval(99.9), stretch=AsinhStretch())) +im_2arcmin = Cutout2D(im_mer_irsa, position=coord, size=2 * u.arcmin, wcs=WCS(header_mer_irsa)).data + +plt.imshow(im_2arcmin, cmap='gray', origin='lower', + norm=ImageNormalize(im_2arcmin, interval=PercentileInterval(99.9), stretch=AsinhStretch())) colorbar = plt.colorbar() ``` @@ -212,7 +295,7 @@ Uncomment the code below to plot an image of the entire field of view of the MER # colorbar = plt.colorbar() ``` -## 3. Create multiwavelength Euclid Q1 MER cutouts of a region of interest +## 5. Create multiwavelength Euclid DR1 MER cutouts of a region of interest +++ @@ -220,12 +303,6 @@ Uncomment the code below to plot an image of the entire field of view of the MER We'd like to take a look at the multiwavelength images of our object, but the full MER mosaics are very large, so we will inspect the multiwavelength cutouts. ``` -```{code-cell} ipython3 -urls = science_images['access_url'] - -urls -``` - Create an array with the instrument and filter name so we can add this to the plots. ```{code-cell} ipython3 @@ -237,58 +314,40 @@ science_images['filters'][science_images['filters']== 'VIS_VIS'] = "VIS" science_images['filters'] ``` -## 4. Define cutout parameters for a smaller region of interest +## 6. Define cutout parameters for a smaller region of interest ```{code-cell} ipython3 ######################## User defined section ############################ # Set the image cutout size im_cutout = 1.0 * u.arcmin -# Set the cutout center coordinates -# For now choose a random location on the image -# because the star itself is saturated -ra = 273.8667 -dec = 64.525 - -# Bright star position -# ra = 273.474451 -# dec = 64.397273 - -coords_cutout = SkyCoord(ra, dec, unit='deg', frame='icrs') +# Set the cutout center coordinates, here our search position +coords_cutout = coord ########################################################################## -# Iterate through each filter - -cutout_list = [] -for url in urls: - # Use fsspec to interact with the fits file without downloading the full file - hdu = fits.open(url, use_fsspec=True) - print(f"Opened {url}") - - # Store the header - header = hdu[0].header - - # Read in the cutout of the image that you want - cutout_data = Cutout2D(hdu[0].section, position=coords_cutout, size=im_cutout, wcs=WCS(hdu[0].header)) - - # Close the file - # hdu.close() +def make_cutout(s3_fpath): + # Read in only the cutout of the image that you want, without downloading the full file + with fits.open(f's3://{s3_fpath}', fsspec_kwargs={'anon': True}) as hdu: + header = hdu[0].header + cutout_data = Cutout2D(hdu[0].section, position=coords_cutout, size=im_cutout, wcs=WCS(header)) # Define a new fits file based on this smaller cutout, with accurate WCS based on the cutout size new_hdu = fits.PrimaryHDU(data=cutout_data.data, header=header) new_hdu.header.update(cutout_data.wcs.to_header()) + return new_hdu - # Append the cutout to the list - cutout_list.append(new_hdu) + +# Iterate through each filter +cutout_list = [make_cutout(s3_fpath) for s3_fpath in science_images['s3_fpath']] # Combine all cutouts into a single HDUList and display information final_hdulist = fits.HDUList(cutout_list) final_hdulist.info() ``` -## 5. Visualize multiwavelength Euclid Q1 MER cutouts +## 7. Visualize multiwavelength Euclid DR1 MER cutouts We need to determine the number of images for the grid layout, and then plot each cutout. @@ -316,47 +375,92 @@ plt.tight_layout() plt.show() ``` -## 6. Identify and measure sources in Euclid Q1 MER cutouts with sep +### Make a color composite -First we list all the filters so you can choose which cutout you want to extract sources on. We will choose VIS. +Because all MER mosaics share the same pixel grid, we can combine the deep cutouts directly into a false-color RGB image, with NISP H as red, NISP J as green, and VIS as blue. ```{code-cell} ipython3 -filt_index = np.where(science_images['filters'] == 'VIS')[0][0] +def scale_band(filt): + image_data = final_hdulist[np.where(science_images['filters'] == filt)[0][0]].data + return ImageNormalize(image_data, interval=PercentileInterval(99.5), stretch=AsinhStretch(), clip=True)(image_data) -img1 = final_hdulist[filt_index].data -``` -### Extract some sources from the cutout using sep (python package based on source extractor) +rgb = np.dstack([scale_band('NISP_H'), scale_band('NISP_J'), scale_band('VIS')]) -Following the sep tutorial, first create a background for the cutout -https://sep.readthedocs.io/en/stable/tutorial.html +plt.figure(figsize=(6, 6)) +plt.imshow(rgb, origin='lower') +plt.title('Deep tile: H (red), J (green), VIS (blue)') +plt.axis('off') +plt.show() +``` -Need to do some initial steps (swap byte order) with the cutout to prevent sep from crashing. Then create a background model with sep. +### Compare the deep and wide VIS cutouts + +Let's plot the same region in all of the VIS mosaics of the wide tile: first the official wide survey mosaic, then the mosaics built from single passes over the deep field, and finally the deep mosaic. +All of them are in the same units, so we display them with the same linear stretch from -2 to +5 times the sky noise of the official wide mosaic. +This makes the noise visible: the wide mosaics look grainy, while the deep mosaic is smoother and reveals more faint sources. +The sky noise of each mosaic, measured with sigma clipping, is given in the panel titles. ```{code-cell} ipython3 -img2 = img1.byteswap().view(img1.dtype.newbyteorder()) -c_contiguous_data = np.array(img2, dtype=np.float32) +filt_index = np.where(science_images['filters'] == 'VIS')[0][0] +deep_vis = final_hdulist[filt_index].data + +# Wide VIS mosaics of the tile with the official mosaic, official first, skipping files not available in the bucket +wide_tile_vis = wide_vis_images[[obs_id.startswith(wide_tileID) for obs_id in wide_vis_images['obs_id']]] +wide_tile_vis = wide_tile_vis[wide_tile_vis['plan'] != ''] +wide_tile_vis.sort('official', reverse=True) -bkg = sep.Background(c_contiguous_data) +images = [make_cutout(s3_fpath).data for s3_fpath in wide_tile_vis['s3_fpath']] + [deep_vis] +titles = (['Wide (official)'] + [f'Wide (single pass {i})' for i in range(1, len(wide_tile_vis))] + + [f'Deep tile {deep_tileID}']) -bkg_image = bkg.back() +noise = [sigma_clipped_stats(image_data)[2] for image_data in images] +norm = ImageNormalize(vmin=-2 * noise[0], vmax=5 * noise[0]) -plt.imshow(bkg_image, interpolation='nearest', cmap='gray', origin='lower') -plt.colorbar() +columns = 4 +rows = -(-len(images) // columns) +fig, axes = plt.subplots(rows, columns, figsize=(4 * columns, 4 * rows)) +axes = axes.flatten() +for ax, image_data, title, sigma in zip(axes, images, titles, noise): + ax.imshow(image_data, cmap='gray', origin='lower', norm=norm) + ax.set_title(f'{title}\nnoise = {sigma:.1e}') + ax.set_xticks([]) + ax.set_yticks([]) +for ax in axes[len(images):]: + fig.delaxes(ax) +plt.tight_layout() +plt.show() ``` -### Inspect the background rms as well +## 8. Identify and measure sources in Euclid DR1 MER cutouts with sep -```{code-cell} ipython3 -bkg_rms = bkg.rms() +We extract sources on the deep VIS cutout. -plt.imshow(bkg_rms, interpolation='nearest', cmap='gray', origin='lower') -plt.colorbar(); +```{code-cell} ipython3 +img1 = deep_vis ``` -### Subtract the background +### Extract some sources from the cutout using sep (python package based on source extractor) + +Following the sep tutorial, first create a background for the cutout +https://sep.readthedocs.io/en/stable/tutorial.html + +Need to do some initial steps (swap byte order) with the cutout to prevent sep from crashing. Then create a background model with sep, inspect it and its rms, and subtract it from the cutout. ```{code-cell} ipython3 +img2 = img1.byteswap().view(img1.dtype.newbyteorder()) +c_contiguous_data = np.array(img2, dtype=np.float32) + +# Use a background mesh (bw, bh) larger than our galaxy, so its light is not absorbed into the background +bkg = sep.Background(c_contiguous_data, bw=128, bh=128) + +fig, axes = plt.subplots(1, 2, figsize=(10, 4)) +for ax, image_data, title in zip(axes, [bkg.back(), bkg.rms()], ['Background', 'Background rms']): + im = ax.imshow(image_data, interpolation='nearest', cmap='gray', origin='lower') + ax.set_title(title) + fig.colorbar(im, ax=ax) +plt.show() + data_sub = img2 - bkg ``` @@ -383,7 +487,7 @@ sources_thr = sources[sources['flux'] > flux_threshold] print("Found", len(sources_thr), "objects above flux threshold") ``` -## 7. Review detected sources on the VIS cutout +## 9. Review detected sources on the VIS cutout We plot the VIS cutout with the sources detected overplotted with a red ellipse @@ -405,8 +509,16 @@ for i in range(len(sources_thr)): ax.add_artist(e) ``` +## References + + + +- Euclid Collaboration: Kümmel, M., et al. 2026, *Euclid Data Release (DR1): The MERge processing*, A&A, in prep. +- Euclid Collaboration: Romelli, E., Kümmel, M., Dole, H., et al. 2026, A&A, 711, A4 (the Q1 MER processing) +- Euclid Data Release 1, 2026, [doi:10.57780/esa-d079c51](https://doi.org/10.57780/esa-d079c51) + ## About this Notebook -**Updated**: 2025-03-31 +**Updated**: 2026-10-05 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. From 9b16d39b92dc4de487c285d05a6786440c96547f Mon Sep 17 00:00:00 2001 From: xoubish Date: Thu, 15 Oct 2026 08:45:02 -0700 Subject: [PATCH 2/5] starting with MER catalog, text only --- .../euclid/2_Euclid_intro_MER_catalog.md | 137 +++++++++++++++--- 1 file changed, 118 insertions(+), 19 deletions(-) diff --git a/tutorials/euclid/2_Euclid_intro_MER_catalog.md b/tutorials/euclid/2_Euclid_intro_MER_catalog.md index 892b0990..ac296329 100644 --- a/tutorials/euclid/2_Euclid_intro_MER_catalog.md +++ b/tutorials/euclid/2_Euclid_intro_MER_catalog.md @@ -22,7 +22,21 @@ authors: - name: Vandana Desai --- -# Euclid Q1: MER Catalogs +# Euclid DR1: MER Catalogs + +```{danger} TODO before the DR1 release (remove this box when done) +The notebook has not been run against DR1 yet: the IRSA Euclid database was still being indexed when the text was updated (2026-10-15). Once IRSA says the indexing is done: + +**1. Run the notebook against production IRSA TAP (the DR1 tables are listed there) and check the outputs and the plot.** + +**2. Check that the column names used in the queries (`ra`, `dec`, `flux_detection_total`, `vis_det`, the `flux_*_templfit` and `flux_vis_to_*_templfit` fluxes, `point_like_flag`, `point_like_prob`, `extended_prob`) still exist in DR1, and update the column count and highlights in the tip.** + +**3. Confirm with IRSA which table holds the deep visits (single visits of the deep fields processed like wide survey data), and where the EMDS / HSC fields (HSC3, HSC4) ended up. Update the table descriptions below accordingly.** + +**4. Ask IRSA whether DR1 morphology and cutouts tables (as `euclid_q1_mer_morphology` / `euclid_q1_mer_cutouts` in Q1) will be added, and mention them in the introduction if so.** + +**5. Update the Kümmel et al. reference (journal, volume, arXiv ID) and verify the DR1 DOI.** +``` +++ @@ -31,7 +45,8 @@ authors: +++ By the end of this tutorial, you will: -- Understand the basic characteristics of Euclid Q1 MER catalogs. +- Understand the basic characteristics of Euclid DR1 MER catalogs. +- Know which of the four DR1 MER catalog tables to use. - What columns are available in the MER catalog. - How to query with ADQL in the MER catalog. - How to make a simple color-magnitude diagram with the data. @@ -47,14 +62,52 @@ The primary science goals of Euclid are to better understand the composition and The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. -Euclid Quick Release 1 (Q1) consists of consists of ~30 TB of imaging, spectroscopy, and catalogs covering four non-contiguous fields: -Euclid Deep Field North (22.9 sq deg), Euclid Deep Field Fornax (12.1 sq deg), Euclid Deep Field South (28.1 sq deg), and LDN1641. +Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 63 sq deg in the three Euclid Deep Fields (North, South, and Fornax). +DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. +It follows Quick Release 1 (Q1), which covered four much smaller fields; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. -Among the data products included in the Q1 release are the three MER catalogs: the final catalog, the morphology catalog, and the cutouts catalog. -This notebook provides an introduction to the MER final catalog. -Each entry is a single source with associated photometry from the multiwavelength MER Mosaics (VIS, Y, J, H, and any accompanying external ground-based measurements), along with other basic measurements, like size and shape. +Among the data products included in DR1 are the MER catalogs. +The MER processing detects sources on the VIS and NISP mosaics (see the MER mosaics notebook) and measures their photometry in the Euclid VIS, Y, J, H bands and in the accompanying ground-based u, g, r, i, z bands, along with other basic measurements, like size and shape. +This notebook provides an introduction to the MER final catalog, in which each entry is a single source. +The MER processing for DR1 is described in detail in Euclid Collaboration: Kümmel et al. (2026), see the [References](#references) section below. If you have questions about this notebook, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). + + ++++ + +## The four DR1 MER catalog tables + +Unlike Q1, which had a single MER final catalog table, DR1 splits the MER catalog into four tables, according to how the data were observed and which MER tiling was used. +MER processes the sky in tiles: **wide** tiles (tile IDs starting with `102`, about 32 x 32 arcmin, the same tiling as in Q1) and smaller **deep** tiles (tile IDs starting with `101`, about 17 x 17 arcmin), used for fields too deep to process as wide tiles. + +* `euclid_dr1_mer_catalogue_wide_survey` -- **the Euclid Wide Survey**, the standard-depth survey intended for cosmology (wide tiles). + **If you are looking for the standard Euclid survey, this is the table you want**, and it is the one used in this notebook. +* `euclid_dr1_mer_catalogue_deep_survey` -- **the Euclid Deep Survey**: the deep stacks of the Euclid Deep Fields North, South, and Fornax (deep tiles). +* `euclid_dr1_mer_catalogue_deep_mode` -- other fields deep enough to need the deep tiling, but not part of the Deep Survey: the Euclid auxiliary fields COSMOS, CDFS, and AEGIS. +* `euclid_dr1_mer_catalogue_wide_mode` -- other fields processed with the wide tiling that are neither part of the Wide Survey nor visits of the deep fields, for example some of the fields observed during the thermal stabilization period of the PSF calibration campaign, just before the main survey started (January/February 2024). + + + +The deep fields are covered by both the Deep Survey (deep tiles) and the wide tiling, so the same object can appear in more than one table, measured on data of different depth. +Table 5 of Kümmel et al. (2026) lists all of the DR1 fields with their tiling and exposure times. + ++++ + +## Things to know about the DR1 MER photometry + +Kümmel et al. (2026) describe several changes and known issues compared to Q1. +The most important ones for using the catalog are: + +* **The PSFs are no longer normalized to unit flux**, which corrects the large offsets seen in Q1 for bands with extended PSFs (such as Pan-STARRS i), but means that DR1 fluxes differ systematically from the Q1 ones. Colors measured with aperture (A-PHOT) or template-fitting (T-PHOT) photometry are largely unaffected. +* **The Euclid NIR bands have offsets of a few to about 13 percent** with respect to Gaia, depending on the photometric method. The offsets are similar in the three NIR bands, so NIR colors are less affected than absolute magnitudes. +* **Some tiles are only partially covered**, at the survey borders (see the `filling_factor` column) or in individual bands, for example after solar proton events. Requiring a detection in all bands therefore removes more sources than in Q1. +* **Spurious sources**: `spurious_flag` is set for `spurious_prob > 0.5`, which has proven too aggressive; a threshold of 0.7 is suggested instead. The spurious probability is optimized for wide data. + +++ ## Imports @@ -87,8 +140,10 @@ Irsa.list_catalogs(filter='euclid') ### Choose the Euclid MER table +We use the Wide Survey table. To use another one of the four DR1 MER tables described above, change the table name here. + ```{code-cell} ipython3 -table_mer = 'euclid_q1_mer_catalogue' +table_mer = 'euclid_dr1_mer_catalogue_wide_survey' ``` ### Learn some information about the MER catalog: @@ -104,12 +159,18 @@ print(len(columns_info)) columns_info ``` + + ```{tip} -The MER catalog contains 476 columns, below are a few highlights: +The Q1 MER catalog contains 476 columns, below are a few highlights: - object_id -- flux_vis_psf, mer.flux_y_templfit,flux_j_templfit, mer.flux_h_templfit +- flux_detection_total: the total (Kron) flux in the detection band +- flux_y_templfit, flux_j_templfit, flux_h_templfit: template-fitting fluxes +- flux_vis_to_y_templfit, flux_vis_to_j_templfit, flux_vis_to_h_templfit: template-fitting fluxes measured on the VIS image degraded to the resolution of each NIR band - fwhm +- point_like_flag, point_like_prob, spurious_prob ``` +++ @@ -117,19 +178,40 @@ The MER catalog contains 476 columns, below are a few highlights: ### Define the following ADQL query to find the first 10k stars in the MER catalog Since we are just using the MER catalog alone, it does not have a column for classification. -We can use the `point_like_flag = 1` or `point_like_prob > 0.99` for stars. +We can use `point_like_flag = 1` for stars, which selects VIS-detected objects with `point_like_prob > 0.96`, the threshold that reached a 99% star purity in simulations. +A lower `point_like_prob` threshold gives a more complete but less pure star sample. + +To get total fluxes in the NIR bands, we follow the recipe of the [MER Photometry Cookbook](https://euclid.esac.esa.int/dr/q1/dpdd/merdpd/merphotometrycookbook.html). +The total flux of an object is measured with a Kron aperture in the detection band, which is VIS for VIS-detected objects (`vis_det = 1`), and is given in `flux_detection_total`. +The total flux in another band X is then obtained by scaling the detection-band total flux with the color measured by template fitting: + +$$F_{X,\mathrm{tot}} = \mathtt{flux\_detection\_total} \times \frac{\mathtt{flux\_X\_templfit}}{\mathtt{flux\_vis\_to\_X\_templfit}}$$ -Set all the fluxes to be greater than 0 so the object is detected in all four Euclid MER mosaic images. +where `flux_vis_to_X_templfit` is the template-fitting flux measured on the VIS image degraded to the resolution of band X. +The cookbook gives an equivalent recipe with the aperture fluxes (`flux_X_nfwhm_aper / flux_vis_nfwhm_aper`). + +We select VIS-detected objects and require all of these fluxes to be greater than 0, so the object is detected in all four Euclid MER mosaic images. + + ```{code-cell} ipython3 -adql_stars = ("SELECT TOP 10000 mer.object_id, mer.ra, mer.dec, mer.flux_vis_psf, mer.fluxerr_vis_psf, " +adql_stars = ("SELECT TOP 10000 mer.object_id, mer.ra, mer.dec, " + "mer.flux_detection_total, mer.fluxerr_detection_total, " "mer.flux_y_templfit,mer.fluxerr_y_templfit, mer.flux_j_templfit, mer.fluxerr_j_templfit, " - "mer.flux_h_templfit, mer.fluxerr_h_templfit, mer.point_like_prob, mer.extended_prob " + "mer.flux_h_templfit, mer.fluxerr_h_templfit, " + "mer.flux_vis_to_y_templfit, mer.flux_vis_to_j_templfit, mer.flux_vis_to_h_templfit, " + "mer.point_like_prob, mer.extended_prob " f"FROM {table_mer} AS mer " - "WHERE mer.flux_vis_psf > 0 " + "WHERE mer.vis_det = 1 " + "AND mer.flux_detection_total > 0 " "AND mer.flux_y_templfit > 0 " "AND mer.flux_j_templfit > 0 " "AND mer.flux_h_templfit > 0 " + "AND mer.flux_vis_to_y_templfit > 0 " + "AND mer.flux_vis_to_j_templfit > 0 " + "AND mer.flux_vis_to_h_templfit > 0 " "AND mer.point_like_flag = 1 ") ``` @@ -142,17 +224,24 @@ result_stars[:5] ## 2. Make a color-magnitude diagram using the catalogs pulled from IRSA +- Compute the total Y and H fluxes with the recipe above - Convert from flux in uJy to magnitudes using the zero point correction - Convert the error bars to magnitudes as well - Plot the color-magnitude diagram ```{code-cell} ipython3 -mag_y = -2.5 * np.log10(result_stars["flux_y_templfit"]) + 23.9 -mag_h = -2.5 * np.log10(result_stars["flux_h_templfit"]) + 23.9 +def total_flux(band): + return (result_stars["flux_detection_total"] * result_stars[f"flux_{band}_templfit"] + / result_stars[f"flux_vis_to_{band}_templfit"]) + + +mag_y = -2.5 * np.log10(total_flux("y")) + 23.9 +mag_h = -2.5 * np.log10(total_flux("h")) + 23.9 x = mag_y - mag_h # Y - H y = mag_y +# For simplicity, the error bars only include the template-fitting flux errors xerr = (2.5 / np.log(10) * np.sqrt((result_stars["fluxerr_y_templfit"] / result_stars["flux_y_templfit"])**2 + (result_stars["fluxerr_h_templfit"] / result_stars["flux_h_templfit"])**2)) yerr = (2.5 / np.log(10) * (result_stars["fluxerr_y_templfit"] / result_stars["flux_y_templfit"])) @@ -165,11 +254,21 @@ plt.ylabel('Y') # Note that these limits exclude a handful of points with large error bars. plt.xlim(-2, 2) plt.ylim(24, 16) -plt.title('10k Stars in MER catalog -- IRSA') +plt.title('10k Stars in DR1 MER Wide Survey catalog -- IRSA') ``` + + +## References + + + +- Euclid Collaboration: Kümmel, M., et al. 2026, *Euclid Data Release (DR1): The MERge processing*, A&A, in prep. +- Euclid Collaboration: Romelli, E., Kümmel, M., Dole, H., et al. 2026, A&A, 711, A4 (the Q1 MER processing) +- Euclid Data Release 1, 2026, [doi:10.57780/esa-d079c51](https://doi.org/10.57780/esa-d079c51) + ## About this Notebook -**Updated**: 2025-04-09 +**Updated**: 2026-10-15 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. From e08e385fbd08eef9c5f21bddc6628acdf2b5a67d Mon Sep 17 00:00:00 2001 From: xoubish Date: Fri, 23 Oct 2026 09:55:37 -0700 Subject: [PATCH 3/5] change --- .../euclid/2_Euclid_intro_MER_catalog.md | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/tutorials/euclid/2_Euclid_intro_MER_catalog.md b/tutorials/euclid/2_Euclid_intro_MER_catalog.md index ac296329..8d322f4c 100644 --- a/tutorials/euclid/2_Euclid_intro_MER_catalog.md +++ b/tutorials/euclid/2_Euclid_intro_MER_catalog.md @@ -25,11 +25,9 @@ authors: # Euclid DR1: MER Catalogs ```{danger} TODO before the DR1 release (remove this box when done) -The notebook has not been run against DR1 yet: the IRSA Euclid database was still being indexed when the text was updated (2026-10-15). Once IRSA says the indexing is done: +**1. Remove the cell that points `Irsa.tap_url` to irsadev, once DR1 is public on the production IRSA services, then re-run the notebook against production and check the outputs and the plot.** -**1. Run the notebook against production IRSA TAP (the DR1 tables are listed there) and check the outputs and the plot.** - -**2. Check that the column names used in the queries (`ra`, `dec`, `flux_detection_total`, `vis_det`, the `flux_*_templfit` and `flux_vis_to_*_templfit` fluxes, `point_like_flag`, `point_like_prob`, `extended_prob`) still exist in DR1, and update the column count and highlights in the tip.** +**2. The column names used in the query were checked against the irsadev TAP schema (2026-10-23). Re-check once the production tables are final.** **3. Confirm with IRSA which table holds the deep visits (single visits of the deep fields processed like wide survey data), and where the EMDS / HSC fields (HSC3, HSC4) ended up. Update the table descriptions below accordingly.** @@ -91,7 +89,10 @@ MER processes the sky in tiles: **wide** tiles (tile IDs starting with `102`, ab identified by its sky patch; the ESA archive help puts them in wide mode, but Euclid MER says they should not be there), and where the EMDS / HSC fields (HSC3, HSC4: wide tiling but with EXT bands that need the MER deep catalog) ended up. Do not quote sky patch numbers until confirmed: the MER paper's "patch" numbers (44, 64, 71, 126) differ from the SkyPatch -values given for Q1 (49 EDF-N, 58 EDF-F, 62 EDF-S). Once confirmed, add a bullet for the deep visits here. --> +values given for Q1 (49 EDF-N, 58 EDF-F, 62 EDF-S). Once confirmed, add a bullet for the deep visits here. +irsadev (2026-10-23) also has euclid_dr1_mer_catalogue_deep_survey_photometry and euclid_dr1_mer_catalogue_deep_mode_photometry +(716 columns each); ask IRSA what they hold (EXT bands for the deep tiles?) and describe them here. --> + The deep fields are covered by both the Deep Survey (deep tiles) and the wide tiling, so the same object can appear in more than one table, measured on data of different depth. Table 5 of Kümmel et al. (2026) lists all of the DR1 fields with their tiling and exposure times. @@ -105,9 +106,12 @@ The most important ones for using the catalog are: * **The PSFs are no longer normalized to unit flux**, which corrects the large offsets seen in Q1 for bands with extended PSFs (such as Pan-STARRS i), but means that DR1 fluxes differ systematically from the Q1 ones. Colors measured with aperture (A-PHOT) or template-fitting (T-PHOT) photometry are largely unaffected. * **The Euclid NIR bands have offsets of a few to about 13 percent** with respect to Gaia, depending on the photometric method. The offsets are similar in the three NIR bands, so NIR colors are less affected than absolute magnitudes. -* **Some tiles are only partially covered**, at the survey borders (see the `filling_factor` column) or in individual bands, for example after solar proton events. Requiring a detection in all bands therefore removes more sources than in Q1. +* **Some tiles are only partially covered**, at the survey borders or in individual bands, for example after solar proton events. Requiring a detection in all bands therefore removes more sources than in Q1. * **Spurious sources**: `spurious_flag` is set for `spurious_prob > 0.5`, which has proven too aggressive; a threshold of 0.7 is suggested instead. The spurious probability is optimized for wide data. + + +++ ## Imports @@ -128,6 +132,13 @@ import matplotlib.pyplot as plt from astroquery.ipac.irsa import Irsa ``` + + +```{code-cell} ipython3 +# DR1 is not public yet, so we query IRSA's development server. +Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' +``` + ## 1. Download MER catalog from IRSA directly to this notebook +++ @@ -159,11 +170,11 @@ print(len(columns_info)) columns_info ``` - + ```{tip} -The Q1 MER catalog contains 476 columns, below are a few highlights: +Each DR1 MER catalog table contains 480 columns, below are a few highlights: - object_id - flux_detection_total: the total (Kron) flux in the detection band @@ -192,8 +203,7 @@ The cookbook gives an equivalent recipe with the aperture fluxes (`flux_X_nfwhm_ We select VIS-detected objects and require all of these fluxes to be greater than 0, so the object is detected in all four Euclid MER mosaic images. - ```{code-cell} ipython3 From cd05ac52c102c699636df8e323e79ccfee7cb4d3 Mon Sep 17 00:00:00 2001 From: xoubish Date: Fri, 23 Oct 2026 17:04:21 -0700 Subject: [PATCH 4/5] MER catalog --- .../euclid/2_Euclid_intro_MER_catalog.md | 386 ++++++++++++++---- 1 file changed, 305 insertions(+), 81 deletions(-) diff --git a/tutorials/euclid/2_Euclid_intro_MER_catalog.md b/tutorials/euclid/2_Euclid_intro_MER_catalog.md index 8d322f4c..1d825cc8 100644 --- a/tutorials/euclid/2_Euclid_intro_MER_catalog.md +++ b/tutorials/euclid/2_Euclid_intro_MER_catalog.md @@ -25,15 +25,9 @@ authors: # Euclid DR1: MER Catalogs ```{danger} TODO before the DR1 release (remove this box when done) -**1. Remove the cell that points `Irsa.tap_url` to irsadev, once DR1 is public on the production IRSA services, then re-run the notebook against production and check the outputs and the plot.** +**1. Remove the cell that points `Irsa.tap_url` and `Irsa.sia_url` to irsadev, once DR1 is public on the production IRSA services, then re-run the notebook against production and check the outputs and the plots.** -**2. The column names used in the query were checked against the irsadev TAP schema (2026-10-23). Re-check once the production tables are final.** - -**3. Confirm with IRSA which table holds the deep visits (single visits of the deep fields processed like wide survey data), and where the EMDS / HSC fields (HSC3, HSC4) ended up. Update the table descriptions below accordingly.** - -**4. Ask IRSA whether DR1 morphology and cutouts tables (as `euclid_q1_mer_morphology` / `euclid_q1_mer_cutouts` in Q1) will be added, and mention them in the introduction if so.** - -**5. Update the Kümmel et al. reference (journal, volume, arXiv ID) and verify the DR1 DOI.** +**2. Update the Kümmel et al. reference (journal, volume, arXiv ID) and verify the DR1 DOI.** ``` +++ @@ -46,8 +40,11 @@ By the end of this tutorial, you will: - Understand the basic characteristics of Euclid DR1 MER catalogs. - Know which of the four DR1 MER catalog tables to use. - What columns are available in the MER catalog. -- How to query with ADQL in the MER catalog. -- How to make a simple color-magnitude diagram with the data. +- How to search the MER catalog around a position on the sky. +- How to use the MER quality columns to clean a catalog. +- How to select stars and compute their total fluxes in the Euclid and ground-based bands. +- How to look at the MER VIS images of catalog sources. +- How to query the MER catalog with ADQL and make a color-color diagram of stars. +++ @@ -67,6 +64,7 @@ It follows Quick Release 1 (Q1), which covered four much smaller fields; all of Among the data products included in DR1 are the MER catalogs. The MER processing detects sources on the VIS and NISP mosaics (see the MER mosaics notebook) and measures their photometry in the Euclid VIS, Y, J, H bands and in the accompanying ground-based u, g, r, i, z bands, along with other basic measurements, like size and shape. This notebook provides an introduction to the MER final catalog, in which each entry is a single source. +In this notebook we follow a typical path through the MER catalog: we search for the sources around a position on the sky, clean the sample with the quality columns, select the stars, look at the spectral energy distributions and VIS images of a few of them, and finally make a color-color diagram of many stars. The MER processing for DR1 is described in detail in Euclid Collaboration: Kümmel et al. (2026), see the [References](#references) section below. If you have questions about this notebook, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). @@ -74,10 +72,10 @@ If you have questions about this notebook, please contact the [IRSA helpdesk](ht +++ -## The four DR1 MER catalog tables +## The MER catalog tables -Unlike Q1, which had a single MER final catalog table, DR1 splits the MER catalog into four tables, according to how the data were observed and which MER tiling was used. -MER processes the sky in tiles: **wide** tiles (tile IDs starting with `102`, about 32 x 32 arcmin, the same tiling as in Q1) and smaller **deep** tiles (tile IDs starting with `101`, about 17 x 17 arcmin), used for fields too deep to process as wide tiles. +The DR1 MER catalog is split into four tables, according to how the data were observed and which MER tiling was used. +MER processes the sky in tiles: **wide** tiles (tile IDs starting with `102`, about 32 x 32 arcmin) and smaller **deep** tiles (tile IDs starting with `101`, about 17 x 17 arcmin), used for fields too deep to process as wide tiles. * `euclid_dr1_mer_catalogue_wide_survey` -- **the Euclid Wide Survey**, the standard-depth survey intended for cosmology (wide tiles). **If you are looking for the standard Euclid survey, this is the table you want**, and it is the one used in this notebook. @@ -97,20 +95,8 @@ irsadev (2026-10-23) also has euclid_dr1_mer_catalogue_deep_survey_photometry an The deep fields are covered by both the Deep Survey (deep tiles) and the wide tiling, so the same object can appear in more than one table, measured on data of different depth. Table 5 of Kümmel et al. (2026) lists all of the DR1 fields with their tiling and exposure times. -+++ - -## Things to know about the DR1 MER photometry - -Kümmel et al. (2026) describe several changes and known issues compared to Q1. -The most important ones for using the catalog are: - -* **The PSFs are no longer normalized to unit flux**, which corrects the large offsets seen in Q1 for bands with extended PSFs (such as Pan-STARRS i), but means that DR1 fluxes differ systematically from the Q1 ones. Colors measured with aperture (A-PHOT) or template-fitting (T-PHOT) photometry are largely unaffected. -* **The Euclid NIR bands have offsets of a few to about 13 percent** with respect to Gaia, depending on the photometric method. The offsets are similar in the three NIR bands, so NIR colors are less affected than absolute magnitudes. -* **Some tiles are only partially covered**, at the survey borders or in individual bands, for example after solar proton events. Requiring a detection in all bands therefore removes more sources than in Q1. -* **Spurious sources**: `spurious_flag` is set for `spurious_prob > 0.5`, which has proven too aggressive; a threshold of 0.7 is suggested instead. The spurious probability is optimized for wide data. - - +The MER catalog from the earlier Quick Release 1 (Q1) is also available at IRSA, as the single table `euclid_q1_mer_catalogue`. +All of the Q1 areas have been reprocessed for DR1, so Q1 and DR1 fluxes should not be mixed (see the [notes on the DR1 MER photometry](#notes-on-the-dr1-mer-photometry) below). +++ @@ -122,13 +108,21 @@ We rely on ``astroquery`` features that have been recently added, so please make ```{code-cell} ipython3 # Uncomment the next line to install dependencies if needed -# !pip install numpy matplotlib 'astroquery>=0.4.10' +# !pip install numpy matplotlib astropy 'astroquery>=0.4.10' fsspec s3fs ``` ```{code-cell} ipython3 +import json + import numpy as np import matplotlib.pyplot as plt +import astropy.units as u +from astropy.coordinates import SkyCoord +from astropy.io import fits +from astropy.nddata import Cutout2D +from astropy.visualization import ImageNormalize, PercentileInterval, AsinhStretch +from astropy.wcs import WCS from astroquery.ipac.irsa import Irsa ``` @@ -137,12 +131,15 @@ from astroquery.ipac.irsa import Irsa ```{code-cell} ipython3 # DR1 is not public yet, so we query IRSA's development server. Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' +Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' ``` -## 1. Download MER catalog from IRSA directly to this notebook +## 1. Find the MER catalog tables at IRSA +++ +The DR1 MER catalog is far too large to download, so we use IRSA's services to run queries on the server and retrieve only the rows and columns we need. + First, have a look at what Euclid catalogs are available. With the ``list_catalogs`` functionality, we'll receive a list of the name of the catalogs as well as their brief desciption. ```{code-cell} ipython3 @@ -186,88 +183,315 @@ Each DR1 MER catalog table contains 480 columns, below are a few highlights: +++ -### Define the following ADQL query to find the first 10k stars in the MER catalog +## 2. Search the MER catalog around a position + +We start with all of the sources within 0.5 deg of a position in the Wide Survey. +This area contains a few hundred thousand sources, so rather than retrieving all 480 columns, we write an ADQL query that returns only the columns we need for this notebook: +the positions and tile IDs, the quality columns, the star-galaxy classification, and the template-fitting fluxes in the Euclid and ground-based bands (see section 5). +The `CONTAINS` and `CIRCLE` functions select the sources in a circle on the sky. + + + +```{code-cell} ipython3 +coord = SkyCoord(ra=90.7299, dec=-45.3529, unit='deg') +search_radius = 0.5 * u.deg + +# Template-fitting fluxes in the Euclid NIR bands and in all of the ground-based bands +flux_cols = ([f'flux_{band}_templfit' for band in 'yjh'] + [f'flux_vis_to_{band}_templfit' for band in 'yjh'] + + [col for col in columns_info + if col.startswith('flux_') and '_ext_' in col and col.endswith('_templfit')]) +columns = (['object_id', 'ra', 'dec', 'tileid', 'vis_det', 'flux_detection_total', + 'point_like_flag', 'point_like_prob', 'spurious_prob', 'det_quality_flag', + 'flag_vis', 'flag_y', 'flag_j', 'flag_h'] + flux_cols) + +adql = (f"SELECT {', '.join(columns)} " + f"FROM {table_mer} " + "WHERE CONTAINS(POINT('ICRS', ra, dec), " + f"CIRCLE('ICRS', {coord.ra.deg}, {coord.dec.deg}, {search_radius.to_value(u.deg)})) = 1") + +# Alternatively, uncomment the query below to get 10,000 stars from anywhere in the table instead of all of the +# sources around a position. The sky map below assumes a search around a position, so skip it in that case. +# adql = f"SELECT TOP 10000 {', '.join(columns)} FROM {table_mer} WHERE point_like_flag = 1" +``` + +We run the query with the TAP service, which takes about a minute. + +```{code-cell} ipython3 +result = Irsa.query_tap(adql).to_table() + +# Keep only the sources with a positive total flux in the detection band, so that we can compute magnitudes +result = result[result['flux_detection_total'] > 0] +print(len(result)) +``` + +```{tip} +For a small search area, the `query_region` method of `astroquery` is a simpler alternative to writing ADQL, and it can return all of the columns, for example: + + Irsa.query_region(coordinates=coord, spatial='Cone', catalog=table_mer, radius=3 * u.arcmin, columns='*') +``` + +Let's look at the density of sources on the sky, as offsets from the center of the search. +The MER catalog is built tile by tile, and the map shows the tile structure, areas without sources (for example around bright stars), and the edge of the DR1 coverage. + +```{code-cell} ipython3 +source_coords = SkyCoord(ra=result['ra'], dec=result['dec'], unit='deg') +offset_ra, offset_dec = coord.spherical_offsets_to(source_coords) +offset_ra = offset_ra.to_value(u.arcmin) +offset_dec = offset_dec.to_value(u.arcmin) + +limit = 1.05 * search_radius.to_value(u.arcmin) +fig, ax = plt.subplots(figsize=(7, 6)) +_, _, _, image = ax.hist2d(offset_ra, offset_dec, bins=np.linspace(-limit, limit, 120), cmap='Greys') +fig.colorbar(image, ax=ax, label='Number of sources per bin') +ax.add_patch(plt.Circle((0, 0), search_radius.to_value(u.arcmin), fill=False, linestyle='--', color='tab:blue')) +ax.set_xlim(limit, -limit) # RA increases to the left +ax.set_ylim(-limit, limit) +ax.set_aspect('equal') +ax.set_xlabel('RA offset (arcmin)') +ax.set_ylabel('Dec offset (arcmin)') +``` + +## 3. Clean the sample with the quality columns + +Before using the sources, we check the MER columns that identify sources whose detection or photometry may be unreliable: + +- `spurious_prob` is the probability that a source is a spurious detection, for example an artifact around a bright object. + `spurious_flag` is set for `spurious_prob > 0.5`, but this has proven too aggressive, and a threshold of 0.7 is suggested instead (see the [notes on the DR1 MER photometry](#notes-on-the-dr1-mer-photometry) below). +- `flag_vis`, `flag_y`, `flag_j`, and `flag_h` keep track of the flagged pixels (for example bad or saturated pixels) in the flag image of each band. +- `det_quality_flag` flags problems in the detection step that could corrupt the magnitude used for the star-galaxy separation. + +Nonzero flags do not necessarily mean that a source is unusable, but they are worth checking for your science case. + + + +```{code-cell} ipython3 +is_spurious = result['spurious_prob'] > 0.7 +print(f"spurious_prob > 0.5: {np.sum(result['spurious_prob'] > 0.5)} of {len(result)} sources") +print(f"spurious_prob > 0.7: {np.sum(is_spurious)} of {len(result)} sources") +print(f"flag_vis != 0: {np.sum(result['flag_vis'] != 0)} of {len(result)} sources") +print(f"det_quality_flag != 0: {np.sum(result['det_quality_flag'] != 0)} of {len(result)} sources") +``` + +We remove the likely spurious sources from our sample: + +```{code-cell} ipython3 +clean_sources = result[~is_spurious] +print(len(clean_sources)) +``` + +## 4. Select the stars -Since we are just using the MER catalog alone, it does not have a column for classification. -We can use `point_like_flag = 1` for stars, which selects VIS-detected objects with `point_like_prob > 0.96`, the threshold that reached a 99% star purity in simulations. +The MER catalog does not have a column for classification, but it provides `point_like_prob`, the probability that a VIS-detected source is point-like. +`point_like_flag = 1` selects sources with `point_like_prob > 0.96`, the threshold that reached a 99% star purity in simulations. A lower `point_like_prob` threshold gives a more complete but less pure star sample. -To get total fluxes in the NIR bands, we follow the recipe of the [MER Photometry Cookbook](https://euclid.esac.esa.int/dr/q1/dpdd/merdpd/merphotometrycookbook.html). +```{code-cell} ipython3 +stars = clean_sources[clean_sources['point_like_flag'].filled(0) == 1] +print(len(stars)) +``` + +## 5. Look at the SEDs and VIS images of three stars + +### Total fluxes in the Euclid and ground-based bands + +Besides the Euclid VIS, Y, J, and H bands, the MER catalog contains photometry measured on ground-based images in the u, g, r, i, and z bands. +Which ground-based surveys are available depends on the position on the sky, for example DECam in the south, or a combination of MegaCam, HSC, and Pan-STARRS in the north. +The catalog therefore has columns for all of the possible survey and band combinations, named `flux__ext__...`, and only some of them are filled at a given position. +Let's check which ground-based bands have measurements for our stars. + +```{code-cell} ipython3 +ext_flux_cols = [col for col in stars.colnames + if col.startswith('flux_') and '_ext_' in col and col.endswith('_templfit') + and not col.startswith('flux_vis_to_')] + +for col in ext_flux_cols: + n_measured = np.sum(~np.ma.getmaskarray(stars[col])) + print(f"{col}: {n_measured} of {len(stars)} stars") +``` + +At this position, the ground-based photometry comes from DECam, in the g, r, i, and z bands. + +To get total fluxes, we follow the recipe of the [MER Photometry Cookbook](https://euclid.esac.esa.int/dr/q1/dpdd/merdpd/merphotometrycookbook.html). The total flux of an object is measured with a Kron aperture in the detection band, which is VIS for VIS-detected objects (`vis_det = 1`), and is given in `flux_detection_total`. -The total flux in another band X is then obtained by scaling the detection-band total flux with the color measured by template fitting: +The total flux in another band X, from Euclid or from the ground, is then obtained by scaling the detection-band total flux with the color measured by template fitting: $$F_{X,\mathrm{tot}} = \mathtt{flux\_detection\_total} \times \frac{\mathtt{flux\_X\_templfit}}{\mathtt{flux\_vis\_to\_X\_templfit}}$$ where `flux_vis_to_X_templfit` is the template-fitting flux measured on the VIS image degraded to the resolution of band X. The cookbook gives an equivalent recipe with the aperture fluxes (`flux_X_nfwhm_aper / flux_vis_nfwhm_aper`). - -We select VIS-detected objects and require all of these fluxes to be greater than 0, so the object is detected in all four Euclid MER mosaic images. +The fluxes are in uJy, which we convert to AB magnitudes. ```{code-cell} ipython3 -adql_stars = ("SELECT TOP 10000 mer.object_id, mer.ra, mer.dec, " - "mer.flux_detection_total, mer.fluxerr_detection_total, " - "mer.flux_y_templfit,mer.fluxerr_y_templfit, mer.flux_j_templfit, mer.fluxerr_j_templfit, " - "mer.flux_h_templfit, mer.fluxerr_h_templfit, " - "mer.flux_vis_to_y_templfit, mer.flux_vis_to_j_templfit, mer.flux_vis_to_h_templfit, " - "mer.point_like_prob, mer.extended_prob " - f"FROM {table_mer} AS mer " - "WHERE mer.vis_det = 1 " - "AND mer.flux_detection_total > 0 " - "AND mer.flux_y_templfit > 0 " - "AND mer.flux_j_templfit > 0 " - "AND mer.flux_h_templfit > 0 " - "AND mer.flux_vis_to_y_templfit > 0 " - "AND mer.flux_vis_to_j_templfit > 0 " - "AND mer.flux_vis_to_h_templfit > 0 " - "AND mer.point_like_flag = 1 ") +def total_mag(sources, band): + """Total AB magnitude of VIS-detected sources in a band, following the MER Photometry Cookbook.""" + flux = sources['flux_detection_total'] + if band != 'vis': + flux = flux * sources[f'flux_{band}_templfit'] / sources[f'flux_vis_to_{band}_templfit'] + return -2.5 * np.log10(flux) + 23.9 ``` -We can run the query with the TAP service, and then look at some of the results. +### Choose three stars + +We pick three stars with positive fluxes in all bands, no flagged pixels in the Euclid bands, no detection problems, and of intermediate brightness. +To span the range of stellar colors, we take the stars closest to three points along the stellar locus in the VIS - J vs. J - H color-color diagram that we make below: a blue, an intermediate, and a red star. ```{code-cell} ipython3 -result_stars = Irsa.query_tap(adql_stars).to_table() -result_stars[:5] +# Approximate central wavelengths of the bands, in micron +band_wavelengths = {'g_ext_decam': 0.48, 'r_ext_decam': 0.64, 'vis': 0.72, 'i_ext_decam': 0.78, + 'z_ext_decam': 0.92, 'y': 1.08, 'j': 1.37, 'h': 1.77} + +has_all_bands = np.all([np.ma.filled(stars[f'flux_{band}_templfit'], 0) > 0 + for band in band_wavelengths if band != 'vis'], axis=0) +mag_vis = total_mag(stars, 'vis') +no_flags = ((stars['flag_vis'] == 0) & (stars['flag_y'] == 0) & (stars['flag_j'] == 0) & (stars['flag_h'] == 0) + & (stars['det_quality_flag'] == 0)) +candidates = stars[has_all_bands & no_flags & (mag_vis > 18.5) & (mag_vis < 21)] + +color_vis_j = total_mag(candidates, 'vis') - total_mag(candidates, 'j') +color_j_h = total_mag(candidates, 'j') - total_mag(candidates, 'h') +# (VIS - J, J - H) of a blue, an intermediate, and a red star on the stellar locus +locus_points = [(0.0, -0.2), (0.5, -0.03), (1.5, 0.03)] +three_stars = candidates[[np.argmin(np.hypot(color_vis_j - x, color_j_h - y)) for x, y in locus_points]] +three_stars['object_id', 'ra', 'dec', 'tileid'] ``` -## 2. Make a color-magnitude diagram using the catalogs pulled from IRSA +The MER `object_id` is built from the position of the source: its digits are the RA and the absolute value of the Dec, both in units of 1e-7 deg, and its sign is the sign of the Dec. +All of our sources are in the southern sky, so they have negative IDs. -- Compute the total Y and H fluxes with the recipe above -- Convert from flux in uJy to magnitudes using the zero point correction -- Convert the error bars to magnitudes as well -- Plot the color-magnitude diagram +### Get VIS cutouts of the three stars + +The MER mosaics are available through IRSA's Simple Image Access (SIA) service and are hosted in the cloud (see the MER mosaics notebook for more details). +For each star, we search for the wide VIS mosaics at its position, and read a small cutout directly from the cloud, without downloading the full mosaic. +Each source in the MER catalog belongs to one tile, so we use the mosaic of the tile given by its `tileid`. ```{code-cell} ipython3 -def total_flux(band): - return (result_stars["flux_detection_total"] * result_stars[f"flux_{band}_templfit"] - / result_stars[f"flux_vis_to_{band}_templfit"]) +def get_s3_fpath(cloud_access): + cloud_info = json.loads(cloud_access) # converts str to dict + return f"{cloud_info['aws']['bucket_name']}/{cloud_info['aws']['key']}" -mag_y = -2.5 * np.log10(total_flux("y")) + 23.9 -mag_h = -2.5 * np.log10(total_flux("h")) + 23.9 +def vis_cutout(source, size=3 * u.arcsec): + position = SkyCoord(ra=source['ra'], dec=source['dec'], unit='deg') + sia_results = Irsa.query_sia(pos=(position, 1 * u.arcsec), collection='euclid_dr1_wide_DpdMerBksMosaic') + # The tile ID is the first part of the obs_id of the mosaic + is_vis_mosaic = ((sia_results['dataproduct_subtype'] == 'science') & (sia_results['energy_bandpassname'] == 'VIS') + & [obs_id.startswith(str(source['tileid'])) for obs_id in sia_results['obs_id']]) + mosaic = sia_results[is_vis_mosaic][0] + with fits.open(f"s3://{get_s3_fpath(mosaic['cloud_access'])}", fsspec_kwargs={'anon': True}) as hdu: + return Cutout2D(hdu[0].section, position=position, size=size, wcs=WCS(hdu[0].header)).data -x = mag_y - mag_h # Y - H -y = mag_y -# For simplicity, the error bars only include the template-fitting flux errors -xerr = (2.5 / np.log(10) * np.sqrt((result_stars["fluxerr_y_templfit"] / result_stars["flux_y_templfit"])**2 - + (result_stars["fluxerr_h_templfit"] / result_stars["flux_h_templfit"])**2)) -yerr = (2.5 / np.log(10) * (result_stars["fluxerr_y_templfit"] / result_stars["flux_y_templfit"])) +cutouts = [vis_cutout(star) for star in three_stars] +``` -plt.errorbar(x, y, xerr=xerr, yerr=yerr, - fmt='o', markersize=1.5, ecolor='lightgrey', elinewidth=0.5, capsize=2) +### Plot the SEDs and the images -plt.xlabel('Y-H') -plt.ylabel('Y') -# Note that these limits exclude a handful of points with large error bars. -plt.xlim(-2, 2) -plt.ylim(24, 16) -plt.title('10k Stars in DR1 MER Wide Survey catalog -- IRSA') +```{code-cell} ipython3 +star_colors = ['tab:blue', 'tab:green', 'tab:red'] + +# The SEDs on the left, and the three VIS cutouts stacked in a narrow column on the right +fig = plt.figure(figsize=(12, 6)) +grid = fig.add_gridspec(3, 2, width_ratios=[4, 1]) +ax_sed = fig.add_subplot(grid[:, 0]) +for star, color in zip(three_stars, star_colors): + mags = [total_mag(star, band) for band in band_wavelengths] + ax_sed.plot(list(band_wavelengths.values()), mags, 'o-', color=color, label=f"object_id {star['object_id']}") +ax_sed.invert_yaxis() +ax_sed.set_xscale('log') +ax_sed.set_xticks(list(band_wavelengths.values()), ['g', 'r', 'VIS', 'i', 'z', 'Y', 'J', 'H']) +ax_sed.minorticks_off() +ax_sed.set_xlabel('Band (approximate central wavelength on a log scale)') +ax_sed.set_ylabel('Total AB magnitude') +ax_sed.legend() + +for i, (star, cutout, color) in enumerate(zip(three_stars, cutouts, star_colors)): + ax = fig.add_subplot(grid[i, 1]) + norm = ImageNormalize(cutout, interval=PercentileInterval(99.9), stretch=AsinhStretch()) + ax.imshow(cutout, origin='lower', cmap='gray', norm=norm) + ax.set_title(f"VIS = {total_mag(star, 'vis'):.1f}", color=color, fontsize='medium') + ax.set_xticks([]) + ax.set_yticks([]) + for spine in ax.spines.values(): + spine.set_edgecolor(color) + spine.set_linewidth(3) +fig.tight_layout() ``` - +All three stars are point sources in the VIS images, but their SEDs have very different shapes. +The bluest star is hot, with a nearly flat SED in AB magnitudes, while the cooler stars are much fainter in the blue bands than in the near-infrared. +A color-color diagram captures these differences for many stars at once. + +## 6. Make a color-color diagram of the stars + +Finally, we make a color-color diagram of all of the stars in our search area. +Some sources have no measurement in a band, for example where a tile is only partially covered (see the [notes on the DR1 MER photometry](#notes-on-the-dr1-mer-photometry) below). +These fluxes are masked in the table, and faint sources can also have negative fluxes, so we keep only the stars with positive fluxes in all of the bands we need. + +```{code-cell} ipython3 +cc_bands = ['j', 'h', 'g_ext_decam', 'r_ext_decam'] +# Masked fluxes are filled with 0, so they are removed as well +has_all_fluxes = np.all([np.ma.filled(stars[f'flux_{band}_templfit'], 0) > 0 for band in cc_bands], axis=0) +many_stars = stars[has_all_fluxes] +print(f"{len(many_stars)} of {len(stars)} stars have VIS, J, H, g, and r fluxes") + +color_x = total_mag(many_stars, 'vis') - total_mag(many_stars, 'j') +color_y = total_mag(many_stars, 'j') - total_mag(many_stars, 'h') +color_g_r = total_mag(many_stars, 'g_ext_decam') - total_mag(many_stars, 'r_ext_decam') +``` + +We plot the stars as points colored by their ground-based g - r color, add contours of the density of stars, and mark the three stars from above. + +```{code-cell} ipython3 +fig, ax = plt.subplots(figsize=(7, 6)) +image = ax.scatter(color_x, color_y, c=color_g_r, s=5, alpha=0.6, cmap='coolwarm', vmin=0, vmax=1.6, edgecolor='none') +fig.colorbar(image, ax=ax, label='g - r (DECam)') + +# Contours of the star counts in coarser bins, which smooths them +counts, xedges, yedges = np.histogram2d(color_x, color_y, + bins=[np.arange(-0.75, 2.6, 0.12), np.arange(-0.6, 0.65, 0.05)]) +ax.contour((xedges[:-1] + xedges[1:]) / 2, (yedges[:-1] + yedges[1:]) / 2, counts.T, + levels=[5, 15, 40], colors='black', linewidths=0.8) + +for star, color in zip(three_stars, star_colors): + ax.scatter(total_mag(star, 'vis') - total_mag(star, 'j'), total_mag(star, 'j') - total_mag(star, 'h'), + s=250, marker='*', color=color, edgecolor='black', linewidth=1.5, zorder=3) + +ax.set_xlim(-0.75, 2.5) +ax.set_ylim(-0.6, 0.6) +ax.set_xlabel('VIS - J') +ax.set_ylabel('J - H') +ax.set_title(f'{len(many_stars)} stars in the DR1 MER Wide Survey catalog -- IRSA') +``` + +The stars follow the stellar locus, a narrow sequence in color-color space set mainly by their temperature. +From hot to cool stars, both colors first become redder together, then J - H stays nearly constant while VIS - J keeps increasing: for the coolest stars, the M dwarfs, molecular absorption shapes the near-infrared spectrum and J - H no longer tracks temperature. +The longer wavelength baseline of VIS - J keeps separating these stars, and our three stars lie at different positions along the locus. +The ground-based g - r color, which also traces temperature, changes smoothly along the locus, from the blue, hot stars to the red, cool ones. + + + +## Notes on the DR1 MER photometry + +Kümmel et al. (2026) describe the DR1 MER photometry and its known limitations in detail. +The most important points for using the catalog are: + +* **NIR calibration offsets**: the Euclid Y, J, and H fluxes have offsets of a few to about 13 percent with respect to Gaia, depending on the photometric method. The offsets are similar in the three bands, so NIR colors are more reliable than absolute NIR magnitudes. +* **Partial coverage**: some tiles are only partially covered, at the survey borders or in individual bands, for example after solar proton events. Requiring a detection in all bands therefore removes sources in these areas. +* **Spurious sources**: `spurious_flag` is set for `spurious_prob > 0.5`, which has proven too aggressive; a threshold of 0.7 is suggested instead. The spurious probability is optimized for wide data. + +```{note} +If you have used the Euclid Quick Release 1 (Q1) MER catalog: the DR1 PSFs are no longer normalized to unit flux. +This corrects the large offsets seen in Q1 for bands with extended PSFs (such as Pan-STARRS i), but means that DR1 fluxes differ systematically from the Q1 ones, so the two releases should not be mixed. +Colors measured with aperture (A-PHOT) or template-fitting (T-PHOT) photometry are largely unaffected. +``` + + + ++++ ## References @@ -279,6 +503,6 @@ plt.title('10k Stars in DR1 MER Wide Survey catalog -- IRSA') ## About this Notebook -**Updated**: 2026-10-15 +**Updated**: 2026-10-23 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. From 586232d91846791a1b6769deaa752c2d81bd5270 Mon Sep 17 00:00:00 2001 From: xoubish Date: Thu, 5 Nov 2026 16:43:49 -0800 Subject: [PATCH 5/5] Address review comments on DR1 MER notebooks 1 and 2 Changes from Jessica's review (PR #398) and Anahita's review (xoubish/irsa-tutorials#1): - Notebook 1: check that an official wide mosaic exists before using it; skip files missing from the bucket in one place; larger sep background mesh; plot a binned view of the full tile; sort the cutout grid by wavelength with short labels; rename science_images to deep_science_images; explain wide/deep tiles, deep visits, noise, and run time; list the Euclid collections in a code cell. - Notebook 2: check all fluxes used by total_mag are positive; select VIS-detected stars explicitly; choose the wide or deep SIA collection from the tile ID; run the TAP query asynchronously; 10 arcsec VIS cutouts; describe the photometry companion tables and the deep visits; resolve the open TODOs. - Both: define MER, fix the Deep Survey area (53 sq deg) and the Q1 description, link the two notebooks, remove the TODO boxes, set the updated date to the DR1 release (2026-11-12). - Update the gallery card descriptions and add DR1 to the Euclid landing page. --- notebook_metadata.yml | 4 +- tutorials/euclid/1_Euclid_intro_MER_images.md | 162 ++++++++++-------- .../euclid/2_Euclid_intro_MER_catalog.md | 93 +++++----- tutorials/euclid/euclid.md | 3 + 4 files changed, 140 insertions(+), 122 deletions(-) diff --git a/notebook_metadata.yml b/notebook_metadata.yml index 12a8f26f..a21fdfc3 100644 --- a/notebook_metadata.yml +++ b/notebook_metadata.yml @@ -5,11 +5,11 @@ - title: MER Image Mosaics file: tutorials/euclid/1_Euclid_intro_MER_images.md section: Euclid - description: Retrieve both a full MER mosaic image and multi-wavelength cutouts, then subtract the background from the cutouts and extract sources. + description: Search the DR1 wide and deep MER mosaics, identify the official wide survey mosaic, make multiwavelength cutouts, compare wide and deep images, and subtract the background and extract sources. - title: MER Catalogs file: tutorials/euclid/2_Euclid_intro_MER_catalog.md section: Euclid - description: Explore the columns in the MER final catalog, query for stars, and create a color-magnitude diagram. + description: Choose among the DR1 MER catalog tables, clean a sample with the quality columns, select stars, plot their SEDs and VIS cutouts, and make a color-color diagram. - title: SIR 1D Spectra file: tutorials/euclid/3_Euclid_intro_1D_spectra.md section: Euclid diff --git a/tutorials/euclid/1_Euclid_intro_MER_images.md b/tutorials/euclid/1_Euclid_intro_MER_images.md index 2d570c01..b363ff9f 100644 --- a/tutorials/euclid/1_Euclid_intro_MER_images.md +++ b/tutorials/euclid/1_Euclid_intro_MER_images.md @@ -24,16 +24,6 @@ authors: # Euclid DR1: MER Mosaics -```{danger} TODO before the DR1 release (remove this box when done) -**1. Remove the cell that points `Irsa.sia_url` to irsadev, once DR1 is public on the production IRSA services.** - -**2. Switch to, or add, `access_url` downloads once IRSA redirects the DR1 `access_url` links to the cloud (expected in about a week).** - -**3. Update the Kümmel et al. reference (journal, volume, arXiv ID), confirm it can be cited, and verify the DR1 DOI.** - -**4. Re-run the notebook against production IRSA and check the outputs and plots.** -``` - +++ ## Learning Goals @@ -59,14 +49,15 @@ By the end of this tutorial, you will: Euclid launched in July 2023 as a European Space Agency (ESA) mission with involvement by NASA. The primary science goals of Euclid are to better understand the composition and evolution of the dark Universe. The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. -These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. +These data are archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. -Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 63 sq deg in the three Euclid Deep Fields (North, South, and Fornax). +Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 53 sq deg in the three Euclid Deep Fields (North, South, and Fornax). DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. -It follows Quick Release 1 (Q1), which covered four much smaller fields; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. +It follows Quick Release 1 (Q1), which covered about 63 sq deg with a single pass over each of the three Euclid Deep Fields and the dark cloud LDN 1641; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. Among the data products included in DR1 are the Level 2 MER mosaics. -These are multiwavelength mosaics created from images taken with the Euclid instruments (VIS and NISP), as well as a variety of ground-based surveys (DES, CFIS, Pan-STARRS, and HSC-based surveys) that provide the u, g, r, i, and z bands. +MER (the Merge processing function of the Euclid ground segment) combines the VIS, NISP, and ground-based images into mosaics on a common grid, and detects and measures the sources on them to build the MER catalog. +The mosaics are created from images taken with the Euclid instruments (VIS and NISP), as well as a variety of ground-based surveys (DES, CFIS, Pan-STARRS, and HSC-based surveys) that provide the u, g, r, i, and z bands. All of the mosaics have been created according to a uniform tiling on the sky, and mapped to a common pixel scale of 0.1 arcsec. The MER processing for DR1 is described in detail in Euclid Collaboration: Kümmel et al. (2026), see the [References](#references) section below. This notebook provides a quick introduction to accessing MER mosaics from IRSA. @@ -79,12 +70,15 @@ If you have questions about it, please contact the [IRSA helpdesk](https://irsa. DR1 MER mosaics come in two kinds of tiles, which IRSA serves as two separate collections: * **Wide tiles** (`euclid_dr1_wide_DpdMerBksMosaic`): tile IDs start with `102`, about 32 x 32 arcmin, the same tiling as in Q1. -* **Deep tiles** (`euclid_dr1_deep_DpdMerBksMosaic`): tile IDs start with `101`, about 17 x 17 arcmin, covering the deep fields. - They stack all of the deep field observations (about 8 times the VIS exposure time of a wide tile), so they reach much fainter sources. +* **Deep tiles** (`euclid_dr1_deep_DpdMerBksMosaic`): tile IDs start with `101`, about 17 x 17 arcmin, used for fields too deep to process as wide tiles: the Euclid Deep Fields and auxiliary fields such as COSMOS, CDFS, and AEGIS. + In the Euclid Deep Fields, they stack all of the observations (about 8 times the VIS exposure time of a wide tile), so they reach much fainter sources. The deep fields are also covered by wide tiles, and because they have been observed many times, these wide tiles have **several mosaics in every band**. -Only one is the official wide survey mosaic; the others are each built from a single pass over the deep field. -The `PPLANID` keyword in the FITS header tells them apart: the official mosaic has `DATA_RELEASE` in it, the others `DATA_RELEASE_DEEP`. +At most one of them is the official wide survey mosaic; the others are each built from a single pass over the deep field. +At our position, only one of the two overlapping wide tiles has its official mosaic in the collection, which the code below selects. +The `PPLANID` keyword in the FITS header tells them apart: the official wide survey mosaic has `-DATA_RELEASE-` in it, the others `DATA_RELEASE_DEEP`. + +In the MER catalog (see the [MER catalogs notebook](2_Euclid_intro_MER_catalog.md)), the sources of the `wide_survey` and `wide_mode` tables are measured on the wide mosaics, and those of the `deep_survey` and `deep_mode` tables on the deep mosaics. +++ @@ -92,6 +86,7 @@ The `PPLANID` keyword in the FITS header tells them apart: the official mosaic h Each wide MER mosaic is approximately 1.47 GB, and each deep MER mosaic is approximately 0.42 GB. Downloading can take some time. +Running the full notebook takes about 20 minutes, most of it spent reading data from the cloud. +++ @@ -131,11 +126,12 @@ import sep ```{code-cell} ipython3 # DR1 is not public yet, so we query IRSA's development server. +Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' ``` ## 1. Search for multiwavelength Euclid DR1 MER mosaics in the Euclid Deep Field North -Below we set a search radius of 10 arcsec around a position in the Euclid Deep Field North, an area covered by both deep and wide tiles. +Below we set a search radius of 10 arcsec around a bright edge-on spiral galaxy in the Euclid Deep Field North, an area covered by both deep and wide tiles. ```{code-cell} ipython3 search_radius = 10 * u.arcsec @@ -144,14 +140,14 @@ coord = SkyCoord(273.1670, 64.3687, unit='deg', frame='icrs') Use IRSA's Simple Image Access (SIA) API to search for all Euclid MER mosaics that overlap with the search region you have specified. -```{tip} -The IRSA SIA collections can be listed using the ``list_collections`` method, we can filter on the ones containing "euclid" in the collection name: +The IRSA SIA collections can be listed using the `list_collections` method; we filter on the ones containing "euclid" in the collection name: - Irsa.list_collections(filter='euclid') +```{code-cell} ipython3 +Irsa.list_collections(filter='euclid') ``` -We search both DR1 MER collections described above, one for the deep and one for the wide tiles. -IRSA serves other Euclid data products, as well as the Q1 and Early Release Observations data, in further collections. +The two DR1 MER collections are `euclid_dr1_wide_DpdMerBksMosaic` and `euclid_dr1_deep_DpdMerBksMosaic`; the others hold the Q1 and Early Release Observations data and other Euclid data products, such as calibrated frames and spectra. +We search both DR1 MER collections, one for the deep and one for the wide tiles. For more information, please read the [IRSA User Guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf). ```{code-cell} ipython3 @@ -164,7 +160,7 @@ These tables list all MER mosaic images available in this search position. These Note that there are various image types are returned as well, we filter out the `science` images from these: ```{code-cell} ipython3 -science_images = deep_table[deep_table['dataproduct_subtype'] == 'science'] +deep_science_images = deep_table[deep_table['dataproduct_subtype'] == 'science'] wide_science_images = wide_table[wide_table['dataproduct_subtype'] == 'science'] ``` @@ -172,21 +168,21 @@ The tile ID is the first part of the `obs_id`: deep tile IDs start with `101` an Adjacent tiles overlap, and our position happens to lie in the overlap of two deep tiles and of two wide tiles: ```{code-cell} ipython3 -for name, images in [('Deep', science_images), ('Wide', wide_science_images)]: +for name, images in [('Deep', deep_science_images), ('Wide', wide_science_images)]: tile_ids, n_vis = np.unique([obs_id[:9] for obs_id in images[images['energy_bandpassname'] == 'VIS']['obs_id']], return_counts=True) print(f'{name} tiles and their number of VIS mosaics:', dict(zip(tile_ids.tolist(), n_vis.tolist()))) ``` -Each deep tile has a single VIS mosaic, while the wide tiles have several. -For the deep tile, we keep the one whose center is closest to our position: +We see that each deep tile has a single VIS mosaic, while the wide tiles have several. +Among the two deep tiles, we keep the one whose center is closest to our position: ```{code-cell} ipython3 -tile_centers = SkyCoord(science_images['s_ra'], science_images['s_dec'], unit='deg') -deep_tileID = science_images['obs_id'][np.argmin(tile_centers.separation(coord))][:9] -science_images = science_images[[obs_id.startswith(deep_tileID) for obs_id in science_images['obs_id']]] +tile_centers = SkyCoord(deep_science_images['s_ra'], deep_science_images['s_dec'], unit='deg') +deep_tileID = deep_science_images['obs_id'][np.argmin(tile_centers.separation(coord))][:9] +deep_science_images = deep_science_images[[obs_id.startswith(deep_tileID) for obs_id in deep_science_images['obs_id']]] print(f'Deep MER tile ID: {deep_tileID}') -science_images +deep_science_images ``` ## 2. Locate the mosaics in the cloud @@ -194,9 +190,6 @@ science_images IRSA hosts the DR1 MER mosaics in an AWS S3 bucket. The `cloud_access` column of the search results gives the bucket name and file location of each mosaic, so let's extract the S3 file path from it: - - ```{code-cell} ipython3 def get_s3_fpath(cloud_access): cloud_info = json.loads(cloud_access) # converts str to dict @@ -206,33 +199,48 @@ def get_s3_fpath(cloud_access): return f'{bucket_name}/{key}' -science_images['s3_fpath'] = [get_s3_fpath(row['cloud_access']) for row in science_images] +deep_science_images['s3_fpath'] = [get_s3_fpath(row['cloud_access']) for row in deep_science_images] wide_science_images['s3_fpath'] = [get_s3_fpath(row['cloud_access']) for row in wide_science_images] ``` +A few files listed in the search results may not be available in the bucket yet, so we check that each file exists and skip the ones that do not: + +```{code-cell} ipython3 +s3 = s3fs.S3FileSystem(anon=True) + + +def drop_missing(images): + exists = np.array([s3.exists(path) for path in images['s3_fpath']], dtype=bool) + for obs_id in images['obs_id'][~exists]: + print(f'Not in the bucket, skipping: {obs_id}') + return images[exists] + + +deep_science_images = drop_missing(deep_science_images) +wide_science_images = drop_missing(wide_science_images) +``` + ## 3. Identify the official wide survey VIS mosaic +Because our position is in the Euclid Deep Field North, which has been observed many times, each wide tile has several VIS mosaics: the official wide survey mosaic and mosaics built from single passes over the deep field (see [Wide and deep MER tiles](#wide-and-deep-mer-tiles) above). To find the official wide survey VIS mosaic, we read the `PPLANID` keyword from the header of each wide VIS mosaic returned by our search. Reading only the header with a small block size avoids downloading the large image data. ```{code-cell} ipython3 -s3 = s3fs.S3FileSystem(anon=True) - - def get_processing_plan(s3_fpath): - try: - with s3.open(s3_fpath, block_size=28800) as f: - return fits.Header.fromfile(f).get('PPLANID', '') - except FileNotFoundError: - # A few files listed in the search results may not be available in the bucket - return '' + with s3.open(s3_fpath, block_size=28800) as f: + return fits.Header.fromfile(f).get('PPLANID', '') wide_vis_images = wide_science_images[wide_science_images['energy_bandpassname'] == 'VIS'] wide_vis_images['plan'] = [get_processing_plan(path) for path in wide_vis_images['s3_fpath']] wide_vis_images['official'] = ['-DATA_RELEASE-' in plan for plan in wide_vis_images['plan']] -wide_tileID = wide_vis_images[wide_vis_images['official']]['obs_id'][0][:9] +official_vis = wide_vis_images[wide_vis_images['official']] +if len(official_vis) == 0: + raise ValueError('No official wide survey VIS mosaic was found at this position; ' + "try another position, or look at wide_vis_images['obs_id', 'plan'].") +wide_tileID = official_vis['obs_id'][0][:9] wide_vis_images['obs_id', 'plan', 'official'] ``` @@ -246,11 +254,11 @@ wide_vis_images['obs_id', 'plan', 'official'] We open the full deep VIS mosaic directly from the S3 bucket. Note this file is about 0.42 GB. ```{code-cell} ipython3 -filename = science_images[science_images['energy_bandpassname'] == 'VIS']['s3_fpath'][0] +filename = deep_science_images[deep_science_images['energy_bandpassname'] == 'VIS']['s3_fpath'][0] print(filename) hdu_mer_irsa = fits.open(f's3://{filename}', fsspec_kwargs={'anon': True}) -print(hdu_mer_irsa.info()) +hdu_mer_irsa.info() header_mer_irsa = hdu_mer_irsa[0].header ``` @@ -273,6 +281,7 @@ Lets extract just the primary image. This reads the full image from the cloud, s ```{code-cell} ipython3 im_mer_irsa = hdu_mer_irsa[0].data +hdu_mer_irsa.close() print(im_mer_irsa.shape) ``` @@ -287,12 +296,16 @@ plt.imshow(im_2arcmin, cmap='gray', origin='lower', colorbar = plt.colorbar() ``` -Uncomment the code below to plot an image of the entire field of view of the MER mosaic. +Let's look at the whole tile, plotting only every 20th pixel to keep it fast. ```{code-cell} ipython3 -# # Full MER mosaic, may take a minute for python to create this image -# plt.imshow(im_mer_irsa, cmap='gray', origin='lower', norm=ImageNormalize(im_mer_irsa, interval=PercentileInterval(99.9), stretch=AsinhStretch())) -# colorbar = plt.colorbar() +# Plot every 20th pixel, which is enough to see the whole tile and fast to display +im_binned = im_mer_irsa[::20, ::20] +plt.imshow(im_binned, cmap='gray', origin='lower', + norm=ImageNormalize(im_binned, interval=PercentileInterval(99.9), stretch=AsinhStretch())) +plt.colorbar() +plt.title(f'Deep VIS mosaic of tile {deep_tileID} (every 20th pixel)') +plt.show() ``` ## 5. Create multiwavelength Euclid DR1 MER cutouts of a region of interest @@ -303,19 +316,28 @@ Uncomment the code below to plot an image of the entire field of view of the MER We'd like to take a look at the multiwavelength images of our object, but the full MER mosaics are very large, so we will inspect the multiwavelength cutouts. ``` -Create an array with the instrument and filter name so we can add this to the plots. +Create short labels with the instrument and filter name so we can add them to the plots, and sort the bands by wavelength. ```{code-cell} ipython3 -science_images['filters'] = science_images['instrument_name'] + "_" + science_images['energy_bandpassname'] - -# VIS_VIS appears in the filters, so update that filter to just say VIS -science_images['filters'][science_images['filters']== 'VIS_VIS'] = "VIS" - -science_images['filters'] +# Short labels for the plots, e.g. 'HSC G' or 'NISP H' +short_names = {'Hyper Suprime-Cam': 'HSC'} +deep_science_images['filters'] = [band if band == 'VIS' else f"{short_names.get(inst, inst)} {band}" + for inst, band in deep_science_images['instrument_name', 'energy_bandpassname']] + +# Sort the bands by wavelength (u, g, r, VIS, i, z from the ground and Euclid, then the NISP bands) +band_order = ['U', 'G', 'R', 'R2', 'VIS', 'I', 'I2', 'Z', 'Y', 'J', 'H'] +deep_science_images['band_order'] = [band_order.index(band) if band in band_order else len(band_order) + for band in deep_science_images['energy_bandpassname']] +deep_science_images.sort('band_order') +deep_science_images['filters'] ``` ## 6. Define cutout parameters for a smaller region of interest +Each cutout is read directly from the cloud, without downloading the full mosaic, but reading the cutouts in all of the bands can still take a few minutes. +Larger cutouts take longer. +A cutout cannot be larger than the tile itself (about 17 arcmin for deep tiles and 32 arcmin for wide tiles); near the edge of a tile it is trimmed to the part that overlaps the tile. + ```{code-cell} ipython3 ######################## User defined section ############################ # Set the image cutout size @@ -340,7 +362,7 @@ def make_cutout(s3_fpath): # Iterate through each filter -cutout_list = [make_cutout(s3_fpath) for s3_fpath in science_images['s3_fpath']] +cutout_list = [make_cutout(s3_fpath) for s3_fpath in deep_science_images['s3_fpath']] # Combine all cutouts into a single HDUList and display information final_hdulist = fits.HDUList(cutout_list) @@ -359,7 +381,7 @@ rows = -(-num_images // columns) fig, axes = plt.subplots(rows, columns, figsize=(4 * columns, 4 * rows), subplot_kw={'projection': WCS(final_hdulist[0].header)}) axes = axes.flatten() -for idx, (ax, filt) in enumerate(zip(axes, science_images['filters'])): +for idx, (ax, filt) in enumerate(zip(axes, deep_science_images['filters'])): image_data = final_hdulist[idx].data norm = ImageNormalize(image_data, interval=PercentileInterval(99.9), stretch=AsinhStretch()) ax.imshow(image_data, cmap='gray', origin='lower', norm=norm) @@ -381,11 +403,11 @@ Because all MER mosaics share the same pixel grid, we can combine the deep cutou ```{code-cell} ipython3 def scale_band(filt): - image_data = final_hdulist[np.where(science_images['filters'] == filt)[0][0]].data + image_data = final_hdulist[np.where(deep_science_images['filters'] == filt)[0][0]].data return ImageNormalize(image_data, interval=PercentileInterval(99.5), stretch=AsinhStretch(), clip=True)(image_data) -rgb = np.dstack([scale_band('NISP_H'), scale_band('NISP_J'), scale_band('VIS')]) +rgb = np.dstack([scale_band('NISP H'), scale_band('NISP J'), scale_band('VIS')]) plt.figure(figsize=(6, 6)) plt.imshow(rgb, origin='lower') @@ -400,14 +422,16 @@ Let's plot the same region in all of the VIS mosaics of the wide tile: first the All of them are in the same units, so we display them with the same linear stretch from -2 to +5 times the sky noise of the official wide mosaic. This makes the noise visible: the wide mosaics look grainy, while the deep mosaic is smoother and reveals more faint sources. The sky noise of each mosaic, measured with sigma clipping, is given in the panel titles. +The official wide survey mosaic is itself built from a single wide survey visit, so its noise is similar to that of the single-pass mosaics; small differences between them come from the observing conditions of each visit. +Artifacts such as satellite trails can appear in any single visit. +This cell reads a cutout from each of the wide VIS mosaics of the tile, so it can take several minutes to run. ```{code-cell} ipython3 -filt_index = np.where(science_images['filters'] == 'VIS')[0][0] +filt_index = np.where(deep_science_images['filters'] == 'VIS')[0][0] deep_vis = final_hdulist[filt_index].data -# Wide VIS mosaics of the tile with the official mosaic, official first, skipping files not available in the bucket +# Wide VIS mosaics of the tile with the official mosaic, official first wide_tile_vis = wide_vis_images[[obs_id.startswith(wide_tileID) for obs_id in wide_vis_images['obs_id']]] -wide_tile_vis = wide_tile_vis[wide_tile_vis['plan'] != ''] wide_tile_vis.sort('official', reverse=True) images = [make_cutout(s3_fpath).data for s3_fpath in wide_tile_vis['s3_fpath']] + [deep_vis] @@ -451,8 +475,8 @@ Need to do some initial steps (swap byte order) with the cutout to prevent sep f img2 = img1.byteswap().view(img1.dtype.newbyteorder()) c_contiguous_data = np.array(img2, dtype=np.float32) -# Use a background mesh (bw, bh) larger than our galaxy, so its light is not absorbed into the background -bkg = sep.Background(c_contiguous_data, bw=128, bh=128) +# Use a background mesh (bw, bh) larger than our galaxy (about 250 pixels long), so its light is not absorbed into the background +bkg = sep.Background(c_contiguous_data, bw=300, bh=300) fig, axes = plt.subplots(1, 2, figsize=(10, 4)) for ax, image_data, title in zip(axes, [bkg.back(), bkg.rms()], ['Background', 'Background rms']): @@ -519,6 +543,6 @@ for i in range(len(sources_thr)): ## About this Notebook -**Updated**: 2026-10-05 +**Updated**: 2026-11-12 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. diff --git a/tutorials/euclid/2_Euclid_intro_MER_catalog.md b/tutorials/euclid/2_Euclid_intro_MER_catalog.md index 1d825cc8..c7096cb8 100644 --- a/tutorials/euclid/2_Euclid_intro_MER_catalog.md +++ b/tutorials/euclid/2_Euclid_intro_MER_catalog.md @@ -24,12 +24,6 @@ authors: # Euclid DR1: MER Catalogs -```{danger} TODO before the DR1 release (remove this box when done) -**1. Remove the cell that points `Irsa.tap_url` and `Irsa.sia_url` to irsadev, once DR1 is public on the production IRSA services, then re-run the notebook against production and check the outputs and the plots.** - -**2. Update the Kümmel et al. reference (journal, volume, arXiv ID) and verify the DR1 DOI.** -``` - +++ ## Learning Goals @@ -55,42 +49,36 @@ By the end of this tutorial, you will: Euclid launched in July 2023 as a European Space Agency (ESA) mission with involvement by NASA. The primary science goals of Euclid are to better understand the composition and evolution of the dark Universe. The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. -These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. +These data are archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. -Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 63 sq deg in the three Euclid Deep Fields (North, South, and Fornax). +Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 53 sq deg in the three Euclid Deep Fields (North, South, and Fornax). DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. -It follows Quick Release 1 (Q1), which covered four much smaller fields; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. +It follows Quick Release 1 (Q1), which covered about 63 sq deg with a single pass over each of the three Euclid Deep Fields and the dark cloud LDN 1641; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. Among the data products included in DR1 are the MER catalogs. -The MER processing detects sources on the VIS and NISP mosaics (see the MER mosaics notebook) and measures their photometry in the Euclid VIS, Y, J, H bands and in the accompanying ground-based u, g, r, i, z bands, along with other basic measurements, like size and shape. +MER (the Merge processing function of the Euclid ground segment) combines the VIS, NISP, and ground-based images into mosaics on a common grid, and detects and measures the sources on them to build the MER catalog. +The MER processing detects sources on the VIS and NISP mosaics (see the [MER mosaics notebook](1_Euclid_intro_MER_images.md)) and measures their photometry in the Euclid VIS, Y, J, H bands and in the accompanying ground-based u, g, r, i, z bands, along with other basic measurements, like size and shape. This notebook provides an introduction to the MER final catalog, in which each entry is a single source. In this notebook we follow a typical path through the MER catalog: we search for the sources around a position on the sky, clean the sample with the quality columns, select the stars, look at the spectral energy distributions and VIS images of a few of them, and finally make a color-color diagram of many stars. The MER processing for DR1 is described in detail in Euclid Collaboration: Kümmel et al. (2026), see the [References](#references) section below. If you have questions about this notebook, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). - - +++ ## The MER catalog tables -The DR1 MER catalog is split into four tables, according to how the data were observed and which MER tiling was used. -MER processes the sky in tiles: **wide** tiles (tile IDs starting with `102`, about 32 x 32 arcmin) and smaller **deep** tiles (tile IDs starting with `101`, about 17 x 17 arcmin), used for fields too deep to process as wide tiles. +The DR1 MER catalog is split into four tables, according to how the data were observed and which MER tiling was used, plus two companion photometry tables for the deep tiles. +MER processes the sky in tiles: **wide** tiles (tile IDs starting with `102`, about 32 x 32 arcmin, the same tiling as in Q1) and smaller **deep** tiles (tile IDs starting with `101`, about 17 x 17 arcmin), used for fields too deep to process as wide tiles. * `euclid_dr1_mer_catalogue_wide_survey` -- **the Euclid Wide Survey**, the standard-depth survey intended for cosmology (wide tiles). **If you are looking for the standard Euclid survey, this is the table you want**, and it is the one used in this notebook. * `euclid_dr1_mer_catalogue_deep_survey` -- **the Euclid Deep Survey**: the deep stacks of the Euclid Deep Fields North, South, and Fornax (deep tiles). * `euclid_dr1_mer_catalogue_deep_mode` -- other fields deep enough to need the deep tiling, but not part of the Deep Survey: the Euclid auxiliary fields COSMOS, CDFS, and AEGIS. * `euclid_dr1_mer_catalogue_wide_mode` -- other fields processed with the wide tiling that are neither part of the Wide Survey nor visits of the deep fields, for example some of the fields observed during the thermal stabilization period of the PSF calibration campaign, just before the main survey started (January/February 2024). +* `euclid_dr1_mer_catalogue_deep_survey_photometry` and `euclid_dr1_mer_catalogue_deep_mode_photometry` -- companion tables for the deep tiles, with photometry in the additional external bands that are only available in the deep and auxiliary fields (for example Spitzer/IRAC, GALEX, VISTA, and the Subaru intermediate bands). They can be joined to the corresponding main table on `object_id`. - - +The `wide_survey` and `wide_mode` sources are measured on the wide-tile mosaics, and the `deep_survey` and `deep_mode` sources on the deep-tile mosaics. +In the deep fields, the wide tiles also have single-pass "deep visit" mosaics (see the [MER mosaics notebook](1_Euclid_intro_MER_images.md)); the wide-tile catalog in these fields comes from the official wide survey mosaic only. The deep fields are covered by both the Deep Survey (deep tiles) and the wide tiling, so the same object can appear in more than one table, measured on data of different depth. Table 5 of Kümmel et al. (2026) lists all of the DR1 fields with their tiling and exposure times. @@ -148,7 +136,9 @@ Irsa.list_catalogs(filter='euclid') ### Choose the Euclid MER table -We use the Wide Survey table. To use another one of the four DR1 MER tables described above, change the table name here. +We use the Wide Survey table. You can change the table name here to use one of the other DR1 MER tables described above, but the rest of this notebook assumes a field with DECam ground-based photometry (g, r, i, z), as at our search position; elsewhere, adjust the bands in the following cells to the ground-based bands available there. +The deep tables have many more sources per area, so use a much smaller search radius with them (for example 0.05 deg) and a position in one of the deep fields. +We run the query asynchronously (`async_job=True`), so that larger queries do not time out. ```{code-cell} ipython3 table_mer = 'euclid_dr1_mer_catalogue_wide_survey' @@ -167,9 +157,6 @@ print(len(columns_info)) columns_info ``` - - ```{tip} Each DR1 MER catalog table contains 480 columns, below are a few highlights: @@ -190,8 +177,6 @@ This area contains a few hundred thousand sources, so rather than retrieving all the positions and tile IDs, the quality columns, the star-galaxy classification, and the template-fitting fluxes in the Euclid and ground-based bands (see section 5). The `CONTAINS` and `CIRCLE` functions select the sources in a circle on the sky. - - ```{code-cell} ipython3 coord = SkyCoord(ra=90.7299, dec=-45.3529, unit='deg') search_radius = 0.5 * u.deg @@ -217,7 +202,7 @@ adql = (f"SELECT {', '.join(columns)} " We run the query with the TAP service, which takes about a minute. ```{code-cell} ipython3 -result = Irsa.query_tap(adql).to_table() +result = Irsa.query_tap(adql, async_job=True).to_table() # Keep only the sources with a positive total flux in the detection band, so that we can compute magnitudes result = result[result['flux_detection_total'] > 0] @@ -249,6 +234,7 @@ ax.set_ylim(-limit, limit) ax.set_aspect('equal') ax.set_xlabel('RA offset (arcmin)') ax.set_ylabel('Dec offset (arcmin)') +plt.show() ``` ## 3. Clean the sample with the quality columns @@ -262,8 +248,6 @@ Before using the sources, we check the MER columns that identify sources whose d Nonzero flags do not necessarily mean that a source is unusable, but they are worth checking for your science case. - - ```{code-cell} ipython3 is_spurious = result['spurious_prob'] > 0.7 print(f"spurious_prob > 0.5: {np.sum(result['spurious_prob'] > 0.5)} of {len(result)} sources") @@ -282,11 +266,13 @@ print(len(clean_sources)) ## 4. Select the stars The MER catalog does not have a column for classification, but it provides `point_like_prob`, the probability that a VIS-detected source is point-like. -`point_like_flag = 1` selects sources with `point_like_prob > 0.96`, the threshold that reached a 99% star purity in simulations. +`point_like_flag = 1` selects VIS-detected sources with `point_like_prob > 0.96`, the threshold that reached a 99% star purity in simulations. A lower `point_like_prob` threshold gives a more complete but less pure star sample. ```{code-cell} ipython3 -stars = clean_sources[clean_sources['point_like_flag'].filled(0) == 1] +# point_like_flag is only set for VIS-detected sources; masked values count as 0 +stars = clean_sources[(np.ma.filled(clean_sources['point_like_flag'], 0) == 1) + & (clean_sources['vis_det'] == 1)] print(len(stars)) ``` @@ -311,7 +297,7 @@ for col in ext_flux_cols: At this position, the ground-based photometry comes from DECam, in the g, r, i, and z bands. -To get total fluxes, we follow the recipe of the [MER Photometry Cookbook](https://euclid.esac.esa.int/dr/q1/dpdd/merdpd/merphotometrycookbook.html). +To get total fluxes, we follow the recipe of the MER Photometry Cookbook in the Euclid DR1 Data Product Description Document (DPDD). The total flux of an object is measured with a Kron aperture in the detection band, which is VIS for VIS-detected objects (`vis_det = 1`), and is given in `flux_detection_total`. The total flux in another band X, from Euclid or from the ground, is then obtained by scaling the detection-band total flux with the color measured by template fitting: @@ -321,9 +307,6 @@ where `flux_vis_to_X_templfit` is the template-fitting flux measured on the VIS The cookbook gives an equivalent recipe with the aperture fluxes (`flux_X_nfwhm_aper / flux_vis_nfwhm_aper`). The fluxes are in uJy, which we convert to AB magnitudes. - - ```{code-cell} ipython3 def total_mag(sources, band): """Total AB magnitude of VIS-detected sources in a band, following the MER Photometry Cookbook.""" @@ -331,6 +314,15 @@ def total_mag(sources, band): if band != 'vis': flux = flux * sources[f'flux_{band}_templfit'] / sources[f'flux_vis_to_{band}_templfit'] return -2.5 * np.log10(flux) + 23.9 + + +def has_positive_fluxes(sources, bands): + """True for sources whose fluxes used by total_mag are all positive (masked fluxes count as missing).""" + cols = ['flux_detection_total'] + for band in bands: + if band != 'vis': + cols += [f'flux_{band}_templfit', f'flux_vis_to_{band}_templfit'] + return np.all([np.ma.filled(sources[col], 0) > 0 for col in cols], axis=0) ``` ### Choose three stars @@ -343,8 +335,7 @@ To span the range of stellar colors, we take the stars closest to three points a band_wavelengths = {'g_ext_decam': 0.48, 'r_ext_decam': 0.64, 'vis': 0.72, 'i_ext_decam': 0.78, 'z_ext_decam': 0.92, 'y': 1.08, 'j': 1.37, 'h': 1.77} -has_all_bands = np.all([np.ma.filled(stars[f'flux_{band}_templfit'], 0) > 0 - for band in band_wavelengths if band != 'vis'], axis=0) +has_all_bands = has_positive_fluxes(stars, band_wavelengths) mag_vis = total_mag(stars, 'vis') no_flags = ((stars['flag_vis'] == 0) & (stars['flag_y'] == 0) & (stars['flag_j'] == 0) & (stars['flag_h'] == 0) & (stars['det_quality_flag'] == 0)) @@ -363,8 +354,8 @@ All of our sources are in the southern sky, so they have negative IDs. ### Get VIS cutouts of the three stars -The MER mosaics are available through IRSA's Simple Image Access (SIA) service and are hosted in the cloud (see the MER mosaics notebook for more details). -For each star, we search for the wide VIS mosaics at its position, and read a small cutout directly from the cloud, without downloading the full mosaic. +The MER mosaics are available through IRSA's Simple Image Access (SIA) service and are hosted in the cloud (see the [MER mosaics notebook](1_Euclid_intro_MER_images.md) for more details). +For each star, we search for the VIS mosaics at its position (in the wide or deep collection, depending on its tile), and read a small cutout directly from the cloud, without downloading the full mosaic. Each source in the MER catalog belongs to one tile, so we use the mosaic of the tile given by its `tileid`. ```{code-cell} ipython3 @@ -373,9 +364,14 @@ def get_s3_fpath(cloud_access): return f"{cloud_info['aws']['bucket_name']}/{cloud_info['aws']['key']}" -def vis_cutout(source, size=3 * u.arcsec): +def vis_cutout(source, size=10 * u.arcsec): position = SkyCoord(ra=source['ra'], dec=source['dec'], unit='deg') - sia_results = Irsa.query_sia(pos=(position, 1 * u.arcsec), collection='euclid_dr1_wide_DpdMerBksMosaic') + # Deep tile IDs start with 101, wide tile IDs with 102 + if str(source['tileid']).startswith('101'): + collection = 'euclid_dr1_deep_DpdMerBksMosaic' + else: + collection = 'euclid_dr1_wide_DpdMerBksMosaic' + sia_results = Irsa.query_sia(pos=(position, 1 * u.arcsec), collection=collection) # The tile ID is the first part of the obs_id of the mosaic is_vis_mosaic = ((sia_results['dataproduct_subtype'] == 'science') & (sia_results['energy_bandpassname'] == 'VIS') & [obs_id.startswith(str(source['tileid'])) for obs_id in sia_results['obs_id']]) @@ -432,8 +428,7 @@ These fluxes are masked in the table, and faint sources can also have negative f ```{code-cell} ipython3 cc_bands = ['j', 'h', 'g_ext_decam', 'r_ext_decam'] -# Masked fluxes are filled with 0, so they are removed as well -has_all_fluxes = np.all([np.ma.filled(stars[f'flux_{band}_templfit'], 0) > 0 for band in cc_bands], axis=0) +has_all_fluxes = has_positive_fluxes(stars, cc_bands) many_stars = stars[has_all_fluxes] print(f"{len(many_stars)} of {len(stars)} stars have VIS, J, H, g, and r fluxes") @@ -464,6 +459,7 @@ ax.set_ylim(-0.6, 0.6) ax.set_xlabel('VIS - J') ax.set_ylabel('J - H') ax.set_title(f'{len(many_stars)} stars in the DR1 MER Wide Survey catalog -- IRSA') +plt.show() ``` The stars follow the stellar locus, a narrow sequence in color-color space set mainly by their temperature. @@ -471,8 +467,6 @@ From hot to cool stars, both colors first become redder together, then J - H sta The longer wavelength baseline of VIS - J keeps separating these stars, and our three stars lie at different positions along the locus. The ground-based g - r color, which also traces temperature, changes smoothly along the locus, from the blue, hot stars to the red, cool ones. - - ## Notes on the DR1 MER photometry Kümmel et al. (2026) describe the DR1 MER photometry and its known limitations in detail. @@ -488,9 +482,6 @@ This corrects the large offsets seen in Q1 for bands with extended PSFs (such as Colors measured with aperture (A-PHOT) or template-fitting (T-PHOT) photometry are largely unaffected. ``` - - +++ ## References @@ -503,6 +494,6 @@ as of 2026-10-23; mention it here if IRSA adds it. --> ## About this Notebook -**Updated**: 2026-10-23 +**Updated**: 2026-11-12 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. diff --git a/tutorials/euclid/euclid.md b/tutorials/euclid/euclid.md index 63cb0d71..78264cbe 100644 --- a/tutorials/euclid/euclid.md +++ b/tutorials/euclid/euclid.md @@ -6,6 +6,9 @@ It carries two instruments: the VISible instrument (VIS) and the Near-Infrared S [Quick Release 1 (Q1)](https://irsa.ipac.caltech.edu/data/Euclid/docs/overview_q1.html) was released in March 2025 and consists of approximately 35 TB of imaging, spectroscopy, and catalogs, covering four non-contiguous fields totaling 63 square degrees. Data products include MERged mosaics of calibrated and stacked frames; combined infrared spectra (SIR); and catalogs of MER objects, photometric redshifts and classifications (PHZ), and spectroscopic redshifts and line measurements (SPE). +Data Release 1 (DR1) comprises about 1900 square degrees of imaging, spectroscopy, and catalogs from the Euclid Wide Survey and the Euclid Deep Survey, reprocessed with updated pipelines and calibrations, including the Q1 areas. +The MER mosaics and MER catalogs notebooks below use DR1; the other notebooks still use Q1. + Data products include: - MER (merged) mosaic images of calibrated and stacked frames; - SIR combined infrared spectra;