From 6ee87dbd129f1719ca30a006271a836823e78579 Mon Sep 17 00:00:00 2001 From: Russ Laher Date: Wed, 4 Nov 2026 14:21:08 -0800 Subject: [PATCH] xobjects: dev's XAstroObjects and XMerges for the xsources, written by the SFFT difference job Ruling decision-xobjects on decisions.md and its section on foundation.md: the xsources' cross-match (dev's crossMatchXSources) in the difference job, founding xobjects with flux0 and xmemberships with fluxap sums, both signs, reconciled at publish, split by field and read from the children. Alert production unchanged; its move to xobjects and xmemberships, and optional sources, objects and memberships, are planned for later. Co-Authored-By: Claude Opus 5.5 --- system/decisions.md | 3 +++ system/foundation.md | 31 +++++++++++++++++++++++++++---- 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/system/decisions.md b/system/decisions.md index 21ee448..80152cd 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -1086,3 +1086,6 @@ The SFFT rows are runs 145 and 146. Run 146's false positives, 102.7, are one pe (decision-field-children)= **`objects`, `memberships`, `packets` and `forced` are split into child tables by field, inheriting their parents and loaded by `COPY`**: one child per Roman tessellation field, `_`, made by the publisher through `create_field_child`, which runs as the schema's owner, as `create_catalog_child` does. `dev` splits `astroobjects` and `merges` by field as standalone `LIKE` tables that do not inherit a prototype; here each child inherits its parent, as the `sources` children do, so the official views and a query by hand on a parent stay correct, and each carries a `CHECK` on its field. `dev` does not split its packet or forced-photometry records; here they are split with the objects. Every row is filed under its object's field: `memberships`, `packets` and `forced` gain a `field` column, the field of the object (`object_job`, `aid`) the row belongs to, so an object's row, memberships, packet rows and forced photometry share one field's four children. A membership the publisher's reconcile attaches to an object founded in a neighbouring field takes that object's field; if that field has no `memberships` child yet, the loading transaction makes one. The publisher records the fields each job's rows went to in `job_fields`. Readers query the children their task needs, never a parent, as for `sources` ({ref}`ruling `): `_in(fields)`, `_of(jobs)` and `objects_official_in(fields)`. The named query `objects_in_cone` takes the fields the cone touches, which the caller computes. SMDC's existing rows move into their children by `database/2026-11-04-field-children.sql`. (Russ Laher, 2026-11-04) + +(decision-xobjects)= +**`xobjects` and `xmemberships` come to the rebuild as `dev`'s XAstroObjects and XMerges (crossMatchXSources.py, d6364970), written by the SFFT difference job from its `xsources`**: the xsources' cross-match is the sources' association over them, as `dev`'s crossMatchXSources is crossMatchSources over XSources. The xsources with SExtractor `flags` equal to `[associate] xsource_flags` (0, as `dev`) and a finite position and `fluxap` are matched, positive and negative together (`dev` does not partition on `isdiffpos`), within `match_radius` against the run's xobjects in their fields and those fields' neighbours. Each unmatched xsource founds an xobject in its own field, its id `radec_index` of its position and its `flux0` its `fluxap`, as `dev`'s. As for objects, not as `dev`: `dev`'s second pass is replaced by the field rule ({ref}`ruling `); duplicate founders are reconciled at publish, under the same field locks as the objects, rather than merged later; statistics are sufficient sums on each xmembership (`xobjects_official`), not `dev`'s XAstroObjectsMeta; and a superseded source set's xmemberships leave the official view rather than being pruned (`dev`'s pruneNotBestXMerges). Both tables are split into child tables by field, inheriting their parents, loaded by `COPY` and read from the children only ({ref}`ruling `). They reach SMDC by `database/2026-11-04-xobjects.sql`. Alert production is unchanged: packets and forced photometry are still made from `objects` and `memberships`; making them from `xobjects` and `xmemberships`, and making the loading of `sources`, `objects` and `memberships` optional for alerts, are planned for later changes. (Russ Laher, 2026-11-04) diff --git a/system/foundation.md b/system/foundation.md index cb1cc6c..b6de6f3 100644 --- a/system/foundation.md +++ b/system/foundation.md @@ -109,14 +109,16 @@ distinct live streams: | `refimages` | dataset, field, fid | `refimages`, with children `refimimages` (members) and `refimcatalogs` (mosaic catalogs) | | `sources` (source set) | dataset, expid, sca, fid | `sources`, with `xsources` beside it (`dev`'s SExtractor sources of the SFFT image) | | `objects` (the object catalog, not bound) | dataset, survey, field, object id, founding job | `objects`, with `memberships` declared beside each source set | +| `xobjects` (the xsources' object catalog, not bound) | dataset, survey, field, xobject id, founding job | `xobjects`, with `xmemberships` declared beside each source set | | `packets` (packet set) | dataset, expid, sca, fid | `packets`, with children `forced` and `external` | | `catalogs` (reference catalog) | dataset, name, version | `catalogs` | A source is identified by its set and its source id, an object by its founding job and its object id, a packet by its set and its source. -Objects, memberships and xsources carry no binding of their own: a -membership or an xsource is official when its source set is, and an -object when one of its memberships is. A set is +Objects, memberships, xobjects, xmemberships and xsources carry no +binding of their own: a membership, an xmembership or an xsource is +official when its source set is, an object when one of its memberships +is, and an xobject when one of its xmemberships is. A set is identified by the job that wrote it, so a job that writes a set binds exactly one key of that set's product. @@ -217,6 +219,27 @@ their members and their forced rows in the triggers' fields. The operator scripts and the named queries do the same, and the unit test that guards the `sources` parents guards these too. +**The xsources have their own object catalog, `xobjects` and +`xmemberships`** ({ref}`ruling `), `dev`'s +XAstroObjects and XMerges. Under `diff_flavor = "sfft"` the difference +job cross-matches its `xsources` as it associates its sources, as +`dev`'s crossMatchXSources.py does for crossMatchSources.py's sources: +the xsources whose SExtractor `flags` equal `[associate] xsource_flags` +(0, as in `dev`), positive and negative together, are matched within +`match_radius` against the run's xobjects in their fields and the +neighbouring fields, as the job read them. An unmatched xsource founds +an xobject in its own field, its id the hash of its position and its +`flux0` its aperture flux, `fluxap`. Each xmembership records the +xsource's offsets from the xobject's founding position and its `fluxap`, +the xobject's sufficient sums, and `xobjects_official` adds them up. The +publisher reconciles the founded xobjects at publish as it reconciles +objects, under the same field locks, and records the outcome as +`xreconcile`. The two tables are split by field and read like `objects` +and `memberships`: `xobjects_in`, `xobjects_of`, `xmemberships_in`, +`xmemberships_of` and `xobjects_official_in`. They do not yet feed alert +production: packets and forced photometry are still made from `objects` +and `memberships`. + `xsources` are always loaded. `sources` records are loaded unless `[difference] skip_loading_sources_records` is true ({ref}`ruling `). With it true, the job still @@ -674,7 +697,7 @@ image. Each simplification keeps the shape of the full stage. | `import` | stored products for one field: L2 files, difference images, PSFs, the reference and its mosaic catalog, the Gaia mirror's prefix; the cap on sources per image | copies of the image files (the mirror is named, not copied); `exposures`, `l2files`, `diffimages`, `psfs`, `refimages`, `refimcatalogs`, `catalogs` and `sources` (the brightest N per difference image by flux over its error), with limiting magnitude, overlapping fields and sky indices computed; the source system's ids kept; provenance marked unknown | other fields and epochs; recovering provenance | | `psf` | one L2 file's metadata and the CRDS ePSF reference it names (the dataset's overlay names it for RimTimSim; OpenUniverse's is drawn with galsim.roman) | the science and the reference PSF, one stamp twice; two `psfs` rows | a PSF per position on the detector; a reference PSF of the coadd's own | | `reference` | the field's `l2files` rows for one filter and SCA, before the run's reference bound when it has one | the awaicgen mosaic with its coverage and uncertainty maps and SExtractor catalog; `refimages`, `refimimages` (one row per coadded L2 row) and `refimcatalogs` | the Photutils catalog, so the grade's bits from it and the PSF | -| `difference` | one epoch's `l2files` row, a `refimages` row with its mosaic catalog, and the science `psfs` row; the run's objects in the fields its sources fall in and their neighbours, read at execution and recorded | the published differencer's gzipped difference image (`[difference] diff_flavor`, shipped `sfft`), its uncertainty, the mask, both differencers' uncapped catalogs (SFFT's with `dev`'s sumrat, nneg and nbad) and the files `packet` reads beside the image, naive run only for its outcome bits; `diffimages` and the `sources` set, mapped exactly as the import maps `dev`'s, under `sfft` through `dev`'s prefilter; under `sfft`, `xsources`, `dev`'s from both SExtractor catalogs, uncapped; `objects` founded here and this epoch's `memberships` | real/bogus; `rubrat` | +| `difference` | one epoch's `l2files` row, a `refimages` row with its mosaic catalog, and the science `psfs` row; the run's objects in the fields its sources fall in and their neighbours, read at execution and recorded | the published differencer's gzipped difference image (`[difference] diff_flavor`, shipped `sfft`), its uncertainty, the mask, both differencers' uncapped catalogs (SFFT's with `dev`'s sumrat, nneg and nbad) and the files `packet` reads beside the image, naive run only for its outcome bits; `diffimages` and the `sources` set, mapped exactly as the import maps `dev`'s, under `sfft` through `dev`'s prefilter; under `sfft`, `xsources`, `dev`'s from both SExtractor catalogs, uncapped; `objects` founded here and this epoch's `memberships`; under `sfft`, `xobjects` founded here and this epoch's `xmemberships`, from its `xsources` | real/bogus; `rubrat` | | `packet` | the epoch's source set, the science PSF, the dataset's Gaia mirror, the reference and its mosaic catalog; read at execution and recorded: the triggers' objects and official members, their exposures, the window's difference and science images, the official `forced` rows and the mirror's leaves and row groups under the chip's cone | `packets/--.avro` at schema 00.07, the `packets.parquet` index, `forced.parquet`, `external.parquet`; as its last act, the set on the Kafka topic | NED, NED-LVS and solar-system matches | | reader, outside the pipeline | a packet set's Avro and its official index and forced rows | a validation report | nothing |