crc32c codec v2 / v3 incompatibility

[d-v-b]

the v3 crc32c codec does not take any parameters, but the numcodecs crc32c codec, which is defacto the v2 crc32c codec, takes a location parameter that can be either start or end.

If we want migration from v2 to v3 to be as smooth as possible, we could add a location parameter to the v3 codec along with language stating that a missing location field has the same meaning as {"location": "end"}.

If we don't want to change the v3 spec, I suspect the consequences are not so severe. The numcodecs default for the location parameter is end, which is compatible with the v3 codec. I suspect most data using this codec used the default, but I could be wrong. Code that maps v2 data to v3 will simply error if location is start in the v2 crc32c codec.

[pvanlaake]

Formally speaking, numcodecs is not part of the core specifications so there is no issue. For those data producers that want to migrate from v2 to v3 and having used the non-default option in numcodecs there is an effort to make, but isn't that just part of the game of staying on top?

[d-v-b]

Formally speaking, numcodecs is not part of the core specifications so there is no issue.

For Zarr V2, codecs did not have a written specification. The behavior of numcodecs was de facto the standard, so from that POV we introduced a regression in Zarr v3 by removing the location parameter parameter from the crc32c codec. While it's possible to shift the burden of adoption onto users, IMO we should strive better to design a backwards-compatible spec.

[jbms]

From my perspective making 100% of zarr v2 data sources metadata-only convertible to zarr v3 wasn't a goal for zarr v3, although I know that some people are interested in making that possible.