Simplify fiboa publish for catalog-driven publication - #214
Open
ivorbosloper wants to merge 110 commits into
Open
Simplify fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 110 commits into
fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 110 commits into
Conversation
Co-authored-by: Matthias Mohr <matthias@mohr.ws>
# Conflicts: # CHANGELOG.md # pixi.lock
…easing memory requirements for large data sets
`fiboa publish` now only converts, validates, builds PMTiles and writes a collection.json with relative links, file:size/file:checksum (multihash), a web-map-links v1.3.0 `pmtiles` link with `pmtiles:layers` and a `visual` asset. README/LICENSE generation, the data-survey lookup and the S3 upload are gone; catalogs such as fieldsoftheworld/harmonized-field-data-catalog own those. spdx-license-list is only needed by tests and moves to the dev feature. Also: - FiboaBaseConverter: keep the determination:datetime column that `use_variant_as_determination` adds; it was removed again as unlisted (affected dk, hr). - be_vlg: drop plots without a crop code (one in 2023 failed validation), take the determination date from the variant year, add the 2026 edition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ivorbosloper
changed the title
fiboa publish for catalog-driven publication
…note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
I'm not really involved in the publishing (yet?), so I can't really judge this. But this PR contains a lot of unrelated changes, so I suspect we need to merge something else first to get this easier to review. |
orderByFields costs ~100 s per request on joined layers (the server sorts the whole join every page), which made the Cantabria 2010-2014 layers take days. A range filter on the unique key answers in ~1 s and a window of page_size ids cannot overflow a page, so fetch min/max once (two sorted one-row queries) and page by half-open id windows, skipping empty ones. Pages cached by the old scheme are reused when their ids prove they cover a window exactly. A where filter in rest_params is now ANDed in instead of being clobbered by the paging clause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
…y variant ideg.xunta.gal serves SIXPAC_2014..SIXPAC_2026 (2010-2013 never existed, 2025/2026 were missing). The older campaigns differ: 2014 names the layer RECINTO with SUP_SIGPAC and no DN_OID, 2015 uses SUP_SIX/USO_SIX, 2020 lacks DN_OID but has IDGEOM. Rename per page in file_migration so the land-use filter and id checks see the canonical names; 2014 gets the SIGPAC recinto reference as id, 2020 the geometry id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
Every SIXPAC_<year> MapServer of es_ga has its Recintos layer at id 2, so pages of different years collided in the cache and later years silently re-read the first year's pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
~16k pages of 1000 features are concatenated per edition; the 20 unmapped attribute columns would otherwise stay in memory until the very end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
A null or empty geometry survives conversion but breaks the canonical Hilbert sort at the very last step (geopandas refuses hilbert_distance on such a GeoSeries), so a 90-minute read ends with nothing written. es_ga 2020-2022 each failed this way; the rate is about 1 in 250,000 features. Drop those rows under the same bounded max_dropped_share rule already used for the required non-null properties, so a converter that produces many of them still errors out. Also widen the schema prewarm budget from 5 attempts (~30 s) to 8 with a 60 s cap (~4 min): a vecorel.org blip outlasted the old budget and killed es_ga 2018 and 2019 outright. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSwkoiouWUsspkFd9aQ2AZ
Galicia had no MT (matorral) code until 2023: scrub was coded PR, which the Spanish base filter keeps as grazing land. Sampling the cached pages shows PR at ~23% of features for every edition 2014-2022 and MT entirely absent, then the two swap from 2023 on (MT ~23%, PR ~1%). Keeping PR therefore left the pre-2023 editions about 75% larger than 2023+ (8.76M vs 4.97M fields for es_ga 2022 vs 2023) with no change on the ground. Exclude PR for campaigns before 2023 so the published series is comparable; 2023+ is untouched and still keeps the genuine shrub pasture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSwkoiouWUsspkFd9aQ2AZ
IFAP restructured the file for 2025: the field boundaries moved from "Culturas_<district>" layers into "T<NUTS 3 code>" ones (beside an empty "Culturas" container and a "Codes" lookup table), the crop code column is now PUN_CUL_CO, the crop name is gone, and the file is published in WGS 84 with Shape_Area and Shape_Length still computed in the source units -- degrees, so 2.7e-07 where 2023 says 3046 m2. Widen the layer filter to both shapes, rename the crop code back in migrate(), and recompute both metrics on EPSG:6933 when the file is geographic, the same projection `fiboa improve` uses to fill missing sizes. Take the determination date from the variant instead of the hard-coded 2023, now that a second edition exists; for 2023 that yields the same value it published before. Editions of one converter drift apart exactly like this, so the test list now accepts a "<id>#<label>" entry that runs a second edition from the same folder of input files, and pt is pinned to 2023 with pt#2025 covering the new shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Both were left as "Spain " with the region missing, which is what names every per-edition item in the catalog: "Spain 2026 (GeoParquet)", two spaces and no way to tell Galicia from Castilla-La Mancha. Their titles already carry the region, so take it from there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The converter left `provider = ""`, so the collection credited only Fields of
the World as processor and host, and rashid flagged it: no provider carried the
'producer' role, and the collection had no link back to the publisher. The
attribution string already names them ("©Government of Cantabria. Free
information available at https://mapas.cantabria.es"); take the provider from
there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The provider URL was https://gobiernoabierto.navarra.es/ — Navarra's open government portal, on a converter for the Balearic Islands. The name beside it and the source comments both say Govern de les Illes Balears, whose site is caib.es. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
`sources` was the bare Zenodo URL, so GDAL was handed the zip itself and answered "not recognized as being in a supported file format". Its three siblings (ec_lt, ec_lv, ec_si) all name the shapefile inside their archive; do the same here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Both converters declare a column required that the 2021 EuroCrops release does
not always fill, so writing the GeoParquet fails on the whole dataset:
Field pyarrow.Field<EC_NUTS3: string not null> was non-nullable but pandas
column had 1 null values
One Latvian parcel has no EC_NUTS3; four Slovenian ones have no Latin crop name
(crop_lat_e). The tests never caught it because the fixtures are the first 100
rows, where both columns are filled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
column_migrations named JAHR — German, copied from de_sax — so it never ran
("Column 'JAHR' not found in dataset, skipping migration") and the bare year
reached the STAC writer as determination:datetime: "Invalid isoformat string:
'2021'". Estonia's column is taotlusaas, the shapefile's ten-character
truncation of taotlusaasta.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Slovenia: four of the 828,161 parcels in the 2021 release carry no crop at all — SIFRA_KMRS, RASTLINA, CROP_LAT_E, COLOR and the EC_* columns are all empty for them, and three parcels lack EC_NUTS3. Requiring any of those fails the write for the whole dataset, one column at a time. Only the GERK parcel id is always present, so that is all the schema requires now. Checked against the source rather than guessed, after the same failure recurred on the next column. Estonia: taotletud_kultuur was required and typed but no column ever produces it — the crop code is mapped to crop:code — so the custom schema described a property that cannot exist. Dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
… zip Each edition ships as a zipped GeoPackage, and the variants named the archive alone, so GDAL opened it as /vsizip/...gpkg.zip. A GeoPackage is SQLite, whose reads are random access, and every seek re-inflates the deflate stream from the start: converting the 2018 edition read 85 GB out of the 1.7 GB archive in 40 minutes and had still not finished reading the layer. Naming the file inside the archive (as at_block already does) makes vecorel extract it once into cache/extracted.*/ and read a plain file. The glob keeps it working across the editions, whose inner names differ. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
…a of use
Austria's 2018 edition carries its CRS as a bare projection definition — an
unnamed "MGI / Austria Lambert" that pyproj cannot match to an EPSG code — so
crs_total_bounds raises and `fiboa publish` died after a five-minute conversion
with "Cannot derive total bounds for projected CRS {...}".
The CRS-derived grid exists so that separately converted parts of one dataset
sort into the same order. Where the CRS cannot supply it, the dataset's own
bbox orders the file just as well; only that mergeability is lost. Wrap it in
hilbert_reference_bounds(crs, fallback) and use it from all three call sites
(publish, the per-file merge, the DuckDB converter). Publish warns and skips
ordering when neither is available, rather than losing the whole conversion —
the file is valid either way, just slower to filter spatially.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The converter mapped FLIK and SCHLAGNR to their own columns but never produced `id`, so the collection shipped without the one column every other collection in the catalog has, and catalogize's spatial sample failed with "Referenced column id not found in FROM clause". There is no natural key to use: the 2025 shapefile holds 880,581 features with only 874,222 distinct (FLIK, SCHLAGNR) pairs, so neither the field block nor the pair identifies a parcel. The row index does, per edition, which is what hr does for the same reason; FLIK and SCHLAGNR stay as attributes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
…nverter Both were published without `id`. sk shows why it can happen quietly: it sets `index_as_id = True`, which fills the column, but `columns` never named `id`, and the base converter's "remove unlisted columns" step took it straight back out. de_bb never had either half. Neither source has a natural key. Brandenburg's ref_ident is a field block reference and gp_xx repeats across rows; Slovakia's KODKD is an LPIS block code shared by several fields and sometimes empty. Both take the row index, as de_nds and hr do, keeping the source codes as their own attributes. Nothing downstream catches this — a file without `id` validates — so the base converter now refuses to run without a mapping, naming the fix in the message, and a test walks every converter to keep it that way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The converter claimed CC-BY-NC-4.0, justified by two comments that belong to es_ib: they point at caib.es and quote the Balearic Islands' terms. Cantabria publishes under neither. Its policy (Decreto 87/2013, modified by Decreto 102/2018) defines two licences of its own, both free of charge — a non-commercial one and a commercial one, the latter needed only for reselling the data — so "NC" also overstates the restriction: their FAQ allows indirect economic benefit under the free non-commercial licence. The Esri service the converter reads names no licence at all, only "Gobierno de Cantabria-FEGA" as the copyright holder. Name the licence and link its terms, and use the attribution wording the licence itself prescribes, verbatim, instead of a translation of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The column was mapped to farmer_id, but its values are 16-character FLIKs: DE + BB + LI (Landwirtschaft/InVeKoS) + two digits for the capture year + two for the district + six sequential, e.g. DEBBLI0268030574 — the same structure de_nds and de_nds_block publish as `flik` (DENILI1742040045). Brandenburg's own description of the Feldblockkataster spells that layout out. So the collection was telling readers a field block reference identified the applicant. Map it to `flik` with the flik extension, as its neighbours do, which also drops the custom schema this converter had to carry for the invented property. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Two things stood between this converter and its thirteen archived editions. `index_as_id = True` replaced the source id with a row number. The current land_parcels.gpkg has no identifier, which is why it was set — but every dated arkod_31_12_<year>.gpkg does, one per parcel and unique (1,293,144 of them in 2011, no duplicates, no nulls). Publishing a row number instead threw away the one key that lets a parcel be followed across years, which is most of the point of having the years. Fall back to the row index only when the source brings nothing. The schema also required six properties the archives do not have or do not fill: jpaid exists only in the current edition, sanitary_protection_zone arrived later, and 2011 leaves mines_status empty for 909k of its parcels and water_protect_zone and natura2000 empty for 1.00M. Requiring those fails the write outright, one column per run. The editions carry different subsets by nature, so the custom schema now requires none of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
`BaseConverter.get_urls` returns `sources` when it is set and only falls back
to `variants`, so a converter with both converts the same file no matter which
`--variant` is asked for. hr declared both: its thirteen ARKOD archives would
have been published as thirteen copies of the current land_parcels.gpkg, one
per year, with nothing in the output to show it. Caught by watching the
conversion open land_parcels.gpkg while building 2011.
hr drops `sources`; its variants already cover every year including 2024.
ec_be_vlg keeps its Zenodo `sources` and now says `variants = {}`, because the
yearly downloads it inherits from the Flemish parent are not its editions —
which was true before, but only by precedence.
The base converter refuses the combination outright, and a test walks every
converter, because the failure leaves no trace in the data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Every arkod_31_12_<year>.gpkg sets gpkg_geometry_columns.srs_id to 0, "Undefined geographic SRS", while holding projected metres — the same HTRS96 / Croatia TM the current edition declares as EPSG:3765. Its extent, 264979..731547 easting and 4698420..5155570 northing, reprojects from 3765 onto Croatia (13.6..19.5 E, 42.4..46.5 N) and onto nothing sensible otherwise. Undeclared, that costs three things: the STAC extent keeps projected metres and fails the bbox gate (west longitude 264984 outside [-180, 180]), the Hilbert grid is built from a global geographic extent so every parcel lands in one cell of it, and the missing-area fallback would measure in degrees. Set it when the source leaves it undefined, and leave the declared one alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
…did not
An audit of every published collection found 36 whose latest edition repeats
ids, and the catalog documents `id` as unique within an edition. Nothing
measured it, so the base converter now does, before geometries are exploded, so
it judges what the converter assigned rather than the parts of a split feature.
The seven it rejects, each broken differently:
es_cl DN_OID is 0 in every row of every province; C_REFREC is
SIGPAC's own 23-character recinto reference.
us_usda_cropland the dissolve merges every CSB of one crop in a state and
splits it again, so the surviving CSBID and CNTY belong to
an arbitrary member of the group: 3,093 ids for 7.5M
fields. The dissolved fields are numbered, and CNTY is no
longer published as if it described them.
dk Marknr numbers a field within one application; Journalnr
makes it unique.
fi PERUSLOHKOTUNNUS is the basic parcel holding several
growing parcels — it is the block, and the field is the
pair with LOHKONUMERO.
pt OSA_ID is the land occupation polygon; CUL_ID identifies
the crop parcel and is unique across every layer of both
editions. They were mapped the wrong way round.
ec_lt nothing in the release identifies a parcel (KZS_NR +
NMA_ID + GRUPE + declared area still leaves 3,991
duplicates), so the row index it is — safe, one file.
es_cn no identifier at all, and seven island files were read into
one frame each keeping its own index, which index_as_id
published: 177,543 ids for 461,729 fields.
es_cn also declared admin:subdivision_code "CB" — Cantabria — for the Canary
Islands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The two conversions the new check stopped were both right to be stopped, but not for the reason it gave. es_cl's C_REFREC identifies every one of the 13,022,051 recintos that carries one; 20 in Soria carry none, 11 of them agricultural. Missing ids are already dropped a few lines further on, under the bounded rule, so counting them as repeats rejected a good identifier — and made the message claim an id "appears 1 times", because value_counts had dropped the nulls it was counting. dk's Journalnr only exists from 2014. Before that the source names the applicant — Ansoeger, KUNDE_LB in 2012 and 2013 — and that pair repeats for real: 5,124 keys cover 11,534 of the 678,347 fields of 2008, mostly distinct polygons of distinct size. Those editions therefore number their rows, as ec_lt does, which is safe because an edition is one file. From 2014 the pair stands, except where the application number is missing (70 of 599,008 rows in 2015, the worst): those rows now have no id and are dropped, instead of every one of them sharing an id of "nan". A 2008 fixture covers the older shape, which also has no crop columns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
Coverage had fallen to 79.0%, below the gate, and the two thinnest modules were the ones this branch changed most: the Esri REST mixin at 19% and the tile half of `publish` at 67%. The REST tests serve a small fake service of five features in pages of two, which is enough to hold every bug the mixin has had: pages of one service overwriting another's in the cache (layer ids repeat — every SIXPAC_<year> MapServer has its Recintos at 2), the converter's own `where` being replaced by the window clause instead of ANDed with it, an error page or a broken download surviving as if it held data, and a legacy sorted-paging page being reused when its ids do not actually cover the window. The tile tests cover the `-t $TMPDIR` that tippecanoe needs (it ignores the variable, and /tmp on the conversion server is 3.8 GB — the cause of every early PMTiles failure), the removal of a half-written .pmtiles, and the reuse of one that is already there. Coverage: 79.0% -> 82.9%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
fiboa publishnow only converts, validates, builds PMTiles and writes acollection.jsonwith relative links,file:size/file:checksum(sha256 multihash), a web-map-links v1.3.0pmtileslink withpmtiles:layers, and avisualasset. README/LICENSE generation, the data-survey lookup and theaws s3 syncupload are removed;--generate-meta,--data-url,--s3-upload-path,--yes,--editor,--converted-by,--data-survey-urlare gone,--no-pmtilesand--tippecanoe-optsare new. README and CHANGELOG updated.spdx-license-listis only used by tests and moves to the pixidevfeature.FiboaBaseConverter: thedetermination:datetimecolumn added byuse_variant_as_determinationwas dropped again by the base converter's "remove unlisted columns" step; it is now listed incolumnsautomatically (affected dk, hr).be_vlg: drop plots without a crop code (one plot in the 2023 edition failed validation), derive the determination date from the variant year instead of a constant, add the 2026 edition.Why
Publishing moves to the git-backed Portolan catalog at https://github.com/fieldsoftheworld/harmonized-field-data-catalog (source.coop/ftw/harmonized-field-data), which owns the catalog layer (styles, thumbnails, AGENTS.md/README, uploads). fiboa-cli stays the single home of conversion logic and produces the per-dataset building blocks.
Verification
The three pilot collections built with this branch pass
rashid check --livewith 0 errors against the published catalog.🤖 Generated with Claude Code