migrate Zarr backend to ZarrCore and add ZarrZip extension support by lazarusA · Pull Request #73 · JuliaDataCubes/YAXArrayBase.jl

lazarusA · 2026-10-08T14:29:18Z

as hinted in #72

Plus is needed to properly implement the multi scales work on PyramidSchemes.jl

codecov · 2026-10-08T14:33:43Z

Codecov Report

❌ Patch coverage is 95.18072% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.96%. Comparing base (49d7155) to head (5545ec3).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
ext/ZarrCoreExt.jl	94.73%	3 Missing ⚠️
ext/ZarrZipExt.jl	93.75%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #73      +/-   ##
==========================================
+ Coverage   52.04%   59.96%   +7.92%     
==========================================
  Files          12       13       +1     
  Lines         490      512      +22     
==========================================
+ Hits          255      307      +52     
+ Misses        235      205      -30

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ressed create_dataset now forwards `compressor` (Zarr) and `compress` (NetCDF) to the coordinate arrays, not only to the data variables. Other keyword arguments such as filters stay data-variable only, since they can be type specific. NetCDF iscompressed always returned false because NetCDF.jl does not read the deflate level back when opening a variable. Query it with nc_inq_var_deflate instead.

Zarr v3 allows omitting dimension_names, which zarr-python does for 0-dimensional arrays. get_var_dims threw for such arrays because neither dimension_names nor _ARRAY_DIMENSIONS was present. Return an empty list for arrays without dimensions instead.

…m Zarr

felixcremer · 2026-10-09T08:56:00Z

I changed to using the zarr_format function from ZarrCore and renamed the corresponding keyword to format. Why is this breaking?

lazarusA · 2026-10-09T09:06:48Z

by loading Zarr.jl should not affect end users, but in isolation is breaking, then in order to access other features, like Zip, you need to be explicit about what to load.

Stop gap till change landed in Zarr.jl

felixcremer · 2026-10-09T10:06:39Z

The last change is needed until JuliaIO/Zarr.jl#359 lands.

lazarusA · 2026-10-11T07:48:37Z

Let's do this update only with the latest version of Zarr, then when a new version is patch-release we come back to this overload approach and enable your suggestion.

Now, we should do this transition.

ZGroup has no metadata field in ZarrCore 0.11.1, so zarr_format(ds) raised a FieldError and every add_var call failed. The format is the group's zarr_format field, a ZarrFormat{N} that converts to Int. Co-authored-by: Claude 🤖 <noreply@anthropic.com>

ZarrCore.zcreate takes the format as the zarr_format keyword. The tests that call it directly passed format, which zcreate rejects; format is the keyword of the YAXArrayBase functions only. Co-authored-by: Claude 🤖 <noreply@anthropic.com>

The Zarr v3 zarr.json files carry the format version under the key zarr_format; there is no key named format. Co-authored-by: Claude 🤖 <noreply@anthropic.com>

Zarr 0.11 is an umbrella package: the array, store and codec types live in ZarrCore, and the umbrella re-exports only some of their names. The ZarrDatasets 0.1.7 release depends on ZarrCore and cannot load beside the pre-split Zarr 0.10 that `[sources]` pinned, which is why CI is red on main (`Method overwriting is not permitted during Module precompilation`). - `Project.toml` requires `Zarr = "0.11"`, and `ZarrCore` is a dependency. The names ZarrCore owns are qualified with it (about 190 changed lines in 32 files, nearly all `Zarr.X` -> `ZarrCore.X`), as ExplicitImports requires. - `[sources]` points Zarr, ZarrCore and ZarrZlib at the fork branch `complex-int-dtype-v0.11`: Zarr.jl `main` plus complex-integer arrays, non-finite complex fill values and threaded chunk decoding. ZarrZlib is listed in `[extras]` only so `[sources]` can name it. The docs environment repeats the pin. Not covered: the YAXArrays test is not run, because YAXArrayBase 0.7 requires Zarr 0.10, so YAXArrays cannot be installed beside this package until YAXArrayBase supports Zarr 0.11 (JuliaDataCubes/YAXArrayBase.jl#73). The docs example for it is replaced by a note. Tested locally on Julia 1.13 with 4 threads: 8434 tests pass, none fail, one is marked broken as before, including Aqua and ExplicitImports; the docs build. Julia 1.10 ignores `[sources]` and resolves registered Zarr 0.11.1, which the CI lts jobs exercise. Co-authored-by: Claude 🤖 <noreply@anthropic.com>

Fix CI for #73

Rename the format keyword of create_dataset, create_empty and add_var back to zarr_format, the name ZarrCore uses. With `format` as the keyword, a `zarr_format` passed by the caller reached zcreate, which overrides the group format and split the store into a v2 group with v3 arrays, losing those arrays on reopening. Read the group format with ZarrCore.zarr_format instead of extending it with a method that returned an Int. This needs ZarrCore 0.11.2. Skip the remote Zarr test with a warning when the server can not be reached, instead of failing on TLS or rate limit errors.

lazarusA · 2026-10-16T13:35:17Z

I think is ok now. The only thing I should have done is to make public the zarr_format function in ZarrCore.jl, but that we do in another round, if you all agree on it.

lazarusA · 2026-10-26T10:46:14Z

@felixcremer merge? we also need to bump release DiskArrayEngine.jl 😭, possibly a minor patch release also of Zarr first.

meggart · 2026-11-02T09:37:49Z

This PR contained 2 interface/behavior changes that were silently introduced. There might be reasons for both, but I would like to rather discuss them in separate PRs/issues. 1) The PR special cased compress keyword arguments to be passed to dimension variables as well so that NetCDF dimensions would be compressed by default. No idea if that was intentional, I just do not understand the reasoning. 2) Previously keyword arguments to create_dataset were only forwarded to variable creation. The PR modified the create_empty interface to force it to accept kwargs and forwards the same kwargs to create_empty as well. I am not fully opposed to this but it should be justified and discussed in a separate issue.

meggart · 2026-11-02T09:38:49Z

I have reverted these 2 changes, from my side the PR would be ready to be merged as soon as CI is green, @lazarusA in case you want to have another look.

lazarusA · 2026-11-02T09:49:55Z

those changes were done such that the integration in JuliaDataCubes/PyramidScheme.jl#114 works out of the box, e.g, savedataset. And since the intention of this is to be breaking, I figured it would be good to have them already. Otherwise the turnover rate will take longer, which apparently needs to be.

lazarusA · 2026-11-02T10:11:31Z

+    YAB.backendlist[:zarr2] = ZarrDataset{2}
+    YAB.backendlist[:zarr3] = ZarrDataset{3}


I don't think this makes a lot of sense. The backend should be only :zarr, hence the additional args in the interface that were introduced elsewhere.

How would a user specify the zarr version in your implementation?

This version allows the user to specify driver=:zarr2 and then with driver=:zarr or driver = :all we get the default zarr version as specified in Zarr.jl or set via Preferences.

you just pass the keyword argument zarr_format=3, or zarr_format=2, which is consistent with ZarrCore API naming.

We might have differing opinions here. I personally think this is a good workaround to avoid interface changes. You can still put :zarr as a backend and then the default backend will be chosen, but if you explicitly want v2 or v3 you are explicit. What is wrong with that approach?

meggart · 2026-11-02T13:55:07Z

I think we should still be explicit when we do breaking changes and mention in the PR what we break and most importantly why. Otherwise I will completely lose overview. I also think that "to make XY work" is not a very good reason for a breaking change, as it still does not give me any hint about why the current interface was not sufficient. E.g. I have no idea why compressing dimension variables helps implementing multiscale pyramids.

migrate Zarr backend to ZarrCore and add ZarrZip extension support

c40381d

lazarusA mentioned this pull request Oct 8, 2026

update YAXArrayBase dependency to version 0.8 and add development source JuliaDataCubes/YAXArrays.jl#630

Open

lazarusA added 2 commits October 8, 2026 17:06

add support for Zarr v3 format and dimension names in ZarrCore extension

c529523

lazarusA force-pushed the la/zarrcore branch from a4ef0a8 to 86d4858 Compare October 8, 2026 17:18

lazarusA mentioned this pull request Oct 8, 2026

v0.1 multi scales convention JuliaDataCubes/PyramidScheme.jl#113

Draft

felixcremer reviewed Oct 8, 2026

View reviewed changes

Comment thread ext/ZarrCoreExt.jl Resolved

felixcremer mentioned this pull request Oct 9, 2026

Allow Zarr 0.11 in ZarrExt #72

Closed

Rename zarr_format keyword to format and use zarr_format function fro…

c7b446d

…m Zarr

lazarusA commented Oct 9, 2026

View reviewed changes

Comment thread ext/ZarrCoreExt.jl Resolved

felixcremer added 2 commits October 9, 2026 11:15

Import zarr_format to extend it

0dbc009

Access metadata of wrapped group for zarr_format

0fdbe4e

Stop gap till change landed in Zarr.jl

meggart reviewed Oct 14, 2026

View reviewed changes

Comment thread ext/ZarrCoreExt.jl Resolved

meggart reviewed Oct 14, 2026

View reviewed changes

Comment thread src/datasets/datasetinterface.jl Resolved

felixcremer reviewed Oct 15, 2026

View reviewed changes

Comment thread ext/ZarrCoreExt.jl Resolved

alex-s-gardner mentioned this pull request Oct 15, 2026

Use Zarr 0.11 and ZarrCore alex-s-gardner/ChunkManifests.jl#27

Merged

alex-s-gardner and others added 3 commits October 15, 2026 14:32

Check the zarr_format key of v3 metadata

3e76868

The Zarr v3 zarr.json files carry the format version under the key zarr_format; there is no key named format. Co-authored-by: Claude 🤖 <noreply@anthropic.com>

lazarusA added 2 commits October 16, 2026 13:35

Merge pull request #74 from alex-s-gardner/fix-pr73-ci

6e19ee4

Fix CI for #73

Revert some behavior changes

5545ec3

lazarusA commented Nov 2, 2026

View reviewed changes

meggart merged commit 65d32ac into master Nov 3, 2026
11 checks passed

		YAB.backendlist[:zarr2] = ZarrDataset{2}
		YAB.backendlist[:zarr3] = ZarrDataset{3}

Conversation

lazarusA commented Oct 8, 2026

Uh oh!

codecov Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

felixcremer commented Oct 9, 2026

Uh oh!

Uh oh!

lazarusA commented Oct 9, 2026

Uh oh!

felixcremer commented Oct 9, 2026

Uh oh!

lazarusA commented Oct 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lazarusA commented Oct 16, 2026

Uh oh!

lazarusA commented Oct 26, 2026

Uh oh!

meggart commented Nov 2, 2026

Uh oh!

meggart commented Nov 2, 2026

Uh oh!

lazarusA commented Nov 2, 2026

Uh oh!

lazarusA Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

felixcremer Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

felixcremer Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

lazarusA Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

meggart Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

meggart commented Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Oct 8, 2026 •

edited

Loading