zarr_metadata.v3.entity

zarr_metadata.v3.entity

The extension layer: what an entity is, and what is in scope.

Every Zarr v3 extension point -- codecs, data types, chunk grids, chunk key encodings, storage transformers -- is modelled as a class that answers for itself. This module is the public door to that layer, for two kinds of caller, and everything either needs is exported from it.

Reading metadata. ArrayDocumentV3.from_json is the fail-fast front door: one call, and either every extension point is read or a single MetadataValidationError carries every reason it is not, in .problems. The entities it yields know things the document does not spell out -- what a data type's scalars are, which position a codec occupies, what a grid divides an array into. A name the scope does not model is not a failure: it arrives as an Opaque marked out_of_scope, for the reader to resolve elsewhere. Nor is a top-level field outside the spec's: must_understand_fields names the ones the reader must refuse to open the array without recognizing.

from zarr_metadata.v3.entity import ArrayBytesCodec, ArrayDocumentV3, CodecEntity

array = ArrayDocumentV3.from_json(json.loads(raw))   # or raises
array.must_understand_fields    # a field here you do not know: refuse
for codec in array.codecs:
    if isinstance(codec, CodecEntity):
        isinstance(codec, ArrayBytesCodec)   # its pipeline position is its base class
    else:
        codec.json, codec.reason    # 'out_of_scope': resolve it yourself

Three layers, ordered by what each needs, each handing the next a typed value and its problems. well_formed_array_v3(value) needs only the value: JSON refined, arrays as tuples, and the document's shape judged. read_array_v3(document, context) needs a scope: each extension point's name related to a class and the class handed the field. refine_array_v3(array) needs the array: the fill value against the type, the grid against the shape, and the codec pipeline walked with what reaches each codec. Its value, RefinedArrayV3, is the resolved pipeline -- Pipeline of PipelineStage, each a codec and the ArrayParts it is handed, a shard's inner pipelines refined inside it -- which is what a codec pipeline is built from; validation is what the walk finds. from_json and validate_array_metadata_v3 run all three.

What comes back. Problems, not exceptions, wherever a document is being judged rather than demanded. zarr_metadata.rules.validate_array_metadata_v3(document, context=...) returns a tuple of ValidationProblem(loc, message, kind), each loc indexing into the document: ("codecs", 1, "configuration", "level"), and kind one of invalid_type, invalid_value, missing_key, unknown_key and invalid_json. resolve(entry, CodecEntity, SCOPE) reads one metadata field as an entity of that kind, the first two layers for a field on its own: it refines and judges the field, relates the name in it to a class in the scope, and hands that class the field, since the class owns its validation routine. It returns (entity, problems) where entity is the entity or an Opaque -- never None -- with loc relative to the entry: ("configuration", "level"). The class's routine, coerce(value, context), returns (entity or None, problems), that is Coerced, and judges the configuration; the envelope is the field's, and resolve judges it. Constructing an entity by hand raises MetadataValidationError with loc relative to the configuration: ("level",).

Writing an extension. Subclass the kind of thing it is -- a codec's kind (ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec), DataTypeEntity, ChunkGridEntity, ChunkKeyEncodingEntity or StorageTransformerEntity; declare its configuration as a frozen Configuration of its members, with every rule finer than a type in its problems, and name it in the entity's one field, configuration; add the class to a scope. An entity of a bare name defaults the field to the empty record: configuration: Configuration = field(default_factory=Configuration). Complete, and runnable as written:

from collections.abc import Iterator
from dataclasses import dataclass
from typing import ClassVar

from zarr_metadata.rules import validate_array_metadata_v3
from zarr_metadata.v3.entity import (
    CORE_AND_EXTENSIONS,
    UNSET,
    BytesBytesCodec,
    Configuration,
        ValidationProblem,
)

@dataclass(frozen=True)  # the fields are the schema; frozen, so a configuration is a value
class AcmeLz4Options(Configuration):
    acceleration: int | UNSET = UNSET  # optional: absent reads as UNSET

    def problems(self) -> Iterator[ValidationProblem]:
        if self.acceleration is not UNSET and not 1 <= self.acceleration <= 65537:
            yield ValidationProblem(
                ("acceleration",),
                f"expected an integer in [1, 65537], got {self.acceleration}",
                "invalid_value",
            )

@dataclass(frozen=True)
class AcmeLz4Codec(BytesBytesCodec):
    configuration: AcmeLz4Options   # the shape of the metadata: a name, and a configuration

    identifier: ClassVar[str] = "acme.lz4"
    variable_size: ClassVar[bool] = True  # a compressor: its output length is not fixed

SCOPE = CORE_AND_EXTENSIONS.extended_with(AcmeLz4Codec)
document = {
    "zarr_format": 3, "node_type": "array", "shape": [8], "data_type": "uint8",
    "fill_value": 0, "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [8]}},
    "chunk_key_encoding": "default",
    "codecs": ["bytes", {"name": "acme.lz4", "configuration": {"acceleration": 3}}],
}
assert validate_array_metadata_v3(document, context=SCOPE) == ()

An entity has the shape of its metadata: a name, which is the class, and a configuration, which is a record dataclass named in the one field configuration; an entity of a bare name defaults it to the empty Configuration, since the spec makes an absent configuration and an empty one the same. The record's fields are the one place the entity's members are declared; the public *Configuration TypedDict beside it declares the JSON, and a test holds the two to the same keys. Which members exist, which may be left out (the type admits UNSET), how each one is type-checked, and how each is written back are all read off the annotations, and the shapes are the ones JSON takes: int, float (any JSON number), bool, str, JSONValue, a Literal of names, tuple[T, ...] or tuple[T1, T2], a TypedDict or dataclass record, Mapping[str, V], a NewType, and a nested entity, always as inner: CodecEntity | Opaque, because that is what the field holds when the inner name is out of scope -- at any depth, as an array's element or a record's field, and read in the scope the containing entity is read in. Anything else is refused at registration. A required member has no default; an optional one is | UNSET = UNSET, so absence stays distinct from a JSON null, and a member that means something when absent is read that way where it is used, not defaulted. Only | UNSET makes a member optional to a document: a plain default serves hand construction, and a document must still write the member. A member is read as codec.configuration.acceleration, the shape the metadata has; nothing lifts it to the entity. with_configuration(**changes) is the entity with members of its configuration replaced, checked as any construction is.

Everything finer than a type -- a bound, a rule about one member, members read together -- is the record's problems, which yields ValidationProblem(loc, message, kind) as it finds each, in plain code. Locations are relative to the configuration, and kind is "invalid_value" for a value rule. The entity's constructor stops at the first problem it yields, so AcmeLz4Codec(AcmeLz4Options(acceleration=0)) raises MetadataValidationError, and the record's constructor refuses a member of the wrong type, so AcmeLz4Options(acceleration="fast") raises too, whether written by hand, through replace or through with_configuration. create_unchecked(**fields), on a record and on an entity, is the one way around the constructors, for a reader that has just made their checks: coerce is that reader, and a document read makes each check once. coerce runs it to the end and reports every problem in the document; a reader with a record asks options.problems() directly and stops or collects. It runs only on a configuration whose members all read: a member of the wrong type is reported and the entity is not built. A family's rule about its name -- r<N> a multiple of 8 -- is the entity's name_problems(name), a classmethod, located on the entity.

What an entity answers for itself, beyond its configuration. to_json is written once in the base, from the record: the bare name when every member is absent, the object otherwise, a contained entity through its own to_json; an entity whose JSON is not its fields overrides it, and none in the package does. ArrayDocumentV3.to_json puts each envelope back as the document spelled it, so a document read and written comes out as it went in. canonical, the entity in its simplest equivalent form: the entity itself by default, overridden where two spellings of its members mean the same, and in an entity that contains entities to put those in canonical form -- self.with_configuration(inner=self.inner.canonical()). An Opaque answers both as well, with the JSON it kept and with itself, so a field typed CodecEntity | Opaque is written and simplified without asking which it holds. coerce is written once in the base. Then, by kind:

• Every entity: identifier, the name it is registered under. A family -- one class for every acme.fixedN -- overrides accepts(name) and keeps the name in a field marked Annotated[str, FROM_NAME], which coerce fills from the envelope.
• A codec: its kind is its base class. One that holds pipelines of its own, as a shard does, declares them through inner_pipelines(incoming) -- each by the member that holds it, with the parts it is handed -- and refinement walks them; it judges nothing inside them itself. An ArrayArrayCodec defines transition(incoming: ArrayParts) -> ArrayParts | None -- abstract: return incoming if it leaves the array's shape, grid and data type alone, the parts it hands the next codec, or None when the metadata cannot say -- and any codec may define incoming_problems(incoming: ArrayParts | None) -> tuple[ValidationProblem, ...] for what it cannot take, where None is an array the chain lost track of and the answer to it is nothing. Every codec declares variable_size, whether its output length depends on its input, which is what keeps a compressor out of a shard's index.
• A data type: scalar_storage, one of StorageClass -- "single_byte", "multi_byte", "variable_length" -- which the method storage_class() answers (the bytes codec asks it whether an endianness is needed), and fill_value_problems(value, loc) -> tuple[ValidationProblem, ...], abstract: it judges a document's fill_value, and a type that accepts any says so with return (). These composition hooks return tuples; a record's problems yields. The families IntegerDataType, FloatDataType, ComplexDataType and NumpyTimeDataType carry both for the types they cover; a family of your own is a plain subclass that is never registered itself, and passes its class variables down.
• A chunk grid: grid(array_shape), abstract, and shape_problems; see ChunkGridEntity.
• A family, one class for many names: identifier is an invented key no document writes, accepts(name) says which names are its own, a field marked Annotated[str, FROM_NAME] keeps the name as written, and name_problems(name), a classmethod, holds any rule about it.

Registration is the one moment an entity is refused, with a message that says what to write: a class without @dataclass, a codec subclassing CodecEntity instead of a kind, a field other than configuration and a carried name, a configuration that is not a Configuration record, a member whose annotation is not a shape JSON takes -- a nested entity without Opaque among them -- a __post_init__ of the entity's own, a class variable a base annotates and nothing sets, and what a kind leaves abstract. What is left, pyright says in the editor and the constructors say at runtime: a member of the wrong type, a record that is not the entity's own, a value the rules disallow, a canonical returning something else, a hook with the wrong signature. A scope reads what a class is off the class: its kind is its base, its key is its identifier, so extended_with takes the classes and nothing can be misfiled -- and a class whose identifier the scope already has takes the name over, so registering your own "gzip" replaces the package's reading of it. Context.of(*classes) is a scope of exactly those, and a scope is a value: resolve reads in it, and claimant(kind, name) says which class a name belongs to.

Two complete extensions written against this module alone, as tests in the repository: tests/v3/test_acme_affine.py (an array_array codec with a number, an optional member and a nested data type) and tests/v3/test_acme_decimal.py (a configured data type with a fill-value rule).

A name in no scope is not rejected -- that is what extension openness means -- so registering yours is how you get it judged rather than waved through. CORE is what the specification defines; CORE_AND_EXTENSIONS adds the zarr-extensions registry; extended_with(*classes) adds yours.

CORE module-attribute

CORE: Final = Context.of(*_CORE)

Only what the Zarr v3 specification defines.

CORE_AND_EXTENSIONS module-attribute

CORE_AND_EXTENSIONS: Final = Context.of(
    *_CORE, *_EXTENSIONS
)

What the specification defines, plus what zarr-extensions registers.

Coerced module-attribute

Coerced: TypeAlias = tuple[
    EntityT | None, tuple[ValidationProblem, ...]
]

The entity if it could be built, and every problem found.

One direction holds: no entity means at least one problem. The converse does not -- a survivable problem, an unknown key, comes back with the entity, because the entity is still readable and saying so is more useful than refusing. A member of the wrong type is not survivable: the entity's rules are written over a whole configuration, and an entity is never built around a hole.

So test entity is None to decide whether to go on reading, and test the problems to decide the verdict. They are different questions.

Extents module-attribute

Extents: TypeAlias = tuple[frozenset[int] | None, ...]

One entry per dimension: the lengths that dimension's chunks take.

A singleton is a uniform axis. None is an axis whose lengths this package cannot determine — distinct from an empty set, which would claim the axis has no chunks at all.

FROM_NAME module-attribute

FROM_NAME: Final = _FromName()

Marks a field carried by the metadata envelope's name, not its configuration.

data_type_name: Annotated[str, FROM_NAME]

A member all the same -- __post_init__ judges it -- but not a configuration key, so it is neither read from nor written to a configuration object. The raw-bytes family is the case: r<N> keeps its width in its name and has no configuration at all.

JSONValue module-attribute

JSONValue = TypeAliasType(
    "JSONValue",
    int
    | float
    | bool
    | str
    | list["JSONValue"]
    | tuple["JSONValue", ...]
    | Mapping[str, "JSONValue"]
    | None,
)

A recursive type alias for JSON-encodable values.

Defined via TypeAliasType (rather than a plain TypeAlias) so the self-reference is a named recursion point that pydantic can resolve when building a TypeAdapter; a bare recursive TypeAlias raises PydanticUserError/RecursionError at validation time.

Loc module-attribute

Loc: TypeAlias = tuple[str | int, ...]

Where in a document a value sits: the keys and indices down to it.

ProblemKind module-attribute

ProblemKind = Literal[
    "missing_key",
    "invalid_type",
    "invalid_value",
    "invalid_json",
    "unknown_key",
]

Machine-readable classification of a ValidationProblem.

• missing_key: a required key (document key or store key) is absent.
• invalid_type: a value has the wrong structural type (e.g. a string where a mapping is required, a non-JSON-serializable object).
• invalid_value: a value has an acceptable type but an invalid content (e.g. zarr_format: 2 in a v3 document, order: "Q").
• invalid_json: bytes that do not decode as JSON.
• unknown_key: a member this package does not model appears inside an entity whose shape it does model (e.g. an extra key in a blosc configuration). Whether a configuration is closed is unspecified (zarr-developers/zarr-specs#270 has been open since 2023), and this package takes the strict reading: in practice such a member is a typo, or a setting meant for a different entity, and accepting it silently means silently ignoring what the writer asked for. Judging it belongs to the rules layer, so rules.parse_* and the whole-document pydantic field types reject it while model.parse_* — which never interpreted configurations — accepts it. It gets a kind of its own so that a caller who wants the tolerant reading can collect problems with rules.validate_* and filter, and so that it never masks the other findings about the same entity.

StorageClass module-attribute

StorageClass = Literal[
    "single_byte", "multi_byte", "variable_length"
]

How one scalar of a data type occupies bytes.

single_byte and multi_byte are both fixed-size; they differ only in whether a byte order applies, which is what the bytes codec's endian member is about.

UNSET module-attribute

UNSET = Sentinel('UNSET')

Marks a metadata-document key as absent (PEP 661 sentinel; usable directly in type expressions, e.g. tuple[str, ...] | UNSET). Test with is UNSET.

ZarrV3MetadataFieldJSON module-attribute

ZarrV3MetadataFieldJSON = str | ZarrV3NamedConfigJSON

The JSON shape of any v3 metadata extension-point entry: either a bare short-hand name string or a {name, configuration, must_understand} envelope.

Used for data_type, chunk_grid, chunk_key_encoding, individual codec entries, and storage_transformers in v3 array metadata, and for the inner codecs / index_codecs lists of the sharding_indexed codec.

ArrayArrayCodec dataclass

Bases: CodecEntity

A codec that transforms the array: what reaches the next codec is its to say.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class ArrayArrayCodec(CodecEntity):
    """A codec that transforms the array: what reaches the next codec is its to say."""

    @abstractmethod
    def transition(self, incoming: ArrayParts) -> ArrayParts | None:
        """What the next codec in the chain sees.

        `incoming` itself if this codec leaves the array's shape, grid and
        data type alone; the parts it hands on if it changes one; None if
        that cannot be determined from the metadata, which ends the
        judgments downstream rather than inventing them.
        """

transition abstractmethod

transition(incoming: ArrayParts) -> ArrayParts | None

What the next codec in the chain sees.

incoming itself if this codec leaves the array's shape, grid and data type alone; the parts it hands on if it changes one; None if that cannot be determined from the metadata, which ends the judgments downstream rather than inventing them.

Source code in src/zarr_metadata/v3/_entity.py

@abstractmethod
def transition(self, incoming: ArrayParts) -> ArrayParts | None:
    """What the next codec in the chain sees.

    `incoming` itself if this codec leaves the array's shape, grid and
    data type alone; the parts it hands on if it changes one; None if
    that cannot be determined from the metadata, which ends the
    judgments downstream rather than inventing them.
    """

ArrayBytesCodec dataclass

Bases: CodecEntity

The one codec in a pipeline that turns the array into bytes.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class ArrayBytesCodec(CodecEntity):
    """The one codec in a pipeline that turns the array into bytes."""

ArrayDocumentV3 dataclass

A v3 array document with its extension points read as entities: the second layer's value.

A field that could not be read holds an Opaque, which carries the JSON the document wrote and says whether the name was out of scope -- an extension this reader does not model, which is not an error -- or claimed and refused. Both are narrowable: every field is an exhaustive two-case union. refine_array_v3 takes it on to the third layer.

Source code in src/zarr_metadata/v3/_document.py

@dataclass(frozen=True, slots=True)
class ArrayDocumentV3:
    """A v3 array document with its extension points read as entities: the second layer's value.

    A field that could not be read holds an `Opaque`, which carries the
    JSON the document wrote and says whether the name was out of scope --
    an extension this reader does not model, which is not an error -- or
    claimed and refused. Both are narrowable: every field is an exhaustive
    two-case union. `refine_array_v3` takes it on to the third layer.
    """

    document: Mapping[str, JSONValue]
    data_type: DataTypeEntity | Opaque
    chunk_grid: ChunkGridEntity | Opaque
    chunk_key_encoding: ChunkKeyEncodingEntity | Opaque
    codecs: tuple[CodecEntity | Opaque, ...]
    storage_transformers: tuple[StorageTransformerEntity | Opaque, ...]

    def __post_init__(self) -> None:
        """Refuse a field holding anything but an entity of its kind or an `Opaque`.

        `read_array_v3` builds a document that holds what it says by
        construction; this is the same guarantee for one built by hand,
        located at the field.
        """
        found = (
            *_as_object(self.document),
            *held_problems(self.data_type, DataTypeEntity, ("data_type",)),
            *held_problems(self.chunk_grid, ChunkGridEntity, ("chunk_grid",)),
            *held_problems(
                self.chunk_key_encoding, ChunkKeyEncodingEntity, ("chunk_key_encoding",)
            ),
            *(
                entry
                for index, codec in enumerate(self.codecs)
                for entry in held_problems(codec, CodecEntity, ("codecs", index))
            ),
            *(
                entry
                for index, transformer in enumerate(self.storage_transformers)
                for entry in held_problems(
                    transformer, StorageTransformerEntity, ("storage_transformers", index)
                )
            ),
        )
        if len(found) != 0:
            raise MetadataValidationError(found)

    def canonical(self) -> ArrayDocumentV3:
        """This document in the simplest form that means the same thing.

        Each entity in its own canonical form and in its own spelling of
        the envelope -- the bare name when nothing is configured, no
        `must_understand`, which means what absence means -- and the one
        rule that is the document's own: `dimension_names` of nothing
        but nulls says what omitting the field says. A *transformation*,
        asked for by `canonicalize_array_metadata_v3`; `to_json` does
        not apply it. What comes back is a document written that way, so
        writing it changes nothing further.
        """
        simplified = replace(
            self,
            data_type=self.data_type.canonical(),
            chunk_grid=self.chunk_grid.canonical(),
            chunk_key_encoding=self.chunk_key_encoding.canonical(),
            codecs=tuple(codec.canonical() for codec in self.codecs),
            storage_transformers=tuple(entry.canonical() for entry in self.storage_transformers),
        )
        document = {**self.document, **_rendered(simplified)}
        names = document.get("dimension_names")
        if isinstance(names, tuple) and all(
            entry is None for entry in cast("tuple[object, ...]", names)
        ):
            del document["dimension_names"]
        return replace(simplified, document=document)

    def to_json(self) -> dict[str, JSONValue]:
        """The document as it was written, with each entity's members as the entity has them.

        Faithful: a document read and written comes out as it went in,
        an entity's envelope included -- `{"name": "crc32c"}` stays an
        object, an empty `configuration` and a `must_understand` of
        `true` stay written -- because the document knows the spelling
        it read and puts it back around what the entity writes. What
        changed is what changes: a member replaced through
        `with_configuration` is written as the entity now has it, and an
        entity put in by hand is written as it writes itself. A field
        the document did not have is not invented, and one it wrote as
        something no entity could be read from stands as written. Ask
        `canonical` first for the simplest equivalent spelling.
        """
        return {
            **self.document,
            **{
                key: cast("JSONValue", _as_written(self.document[key], value))
                for key, value in _rendered(self).items()
            },
        }

    @property
    def must_understand_fields(self) -> dict[str, ZarrV3ExtensionField]:
        """The fields outside the spec's that do not say `must_understand: false`.

        A reader must refuse to open the array if this holds a field it
        does not recognize. Recognition is the reader's own knowledge: a
        top-level field is no extension point, so no scope claims one,
        and the document partitions by obligation and leaves the verdict
        to its reader, as `ZarrV3ArrayMetadata.must_understand_fields`
        does. An extension point the scope does not claim is the same
        question asked of an `Opaque`.
        """
        extra = {
            key: value
            for key, value in self.document.items()
            if key not in ARRAY_METADATA_STANDARD_KEYS_V3
        }
        return must_understand_subset(cast("Mapping[str, ZarrV3ExtensionField]", extra))

    @classmethod
    def from_json(cls, value: object, *, context: Context = CORE_AND_EXTENSIONS) -> ArrayDocumentV3:
        """A v3 array document read into entities, or raise.

        The reader's front door, and the one entry point that fails fast:
        all three layers, and either every extension point is read and
        the whole composes, or a single `MetadataValidationError` carries
        every reason it does not -- structural and semantic together. Use
        `validate_array_metadata_v3` instead when you want the problems
        as data, and the layers themselves when you want to stop between
        them.

        A name this `context` does not model is *not* a failure. It comes
        back as an `Opaque` marked `out_of_scope`, because a document may
        legitimately use an extension this reader does not know, and
        refusing it would make openness unimplementable. What fails is
        metadata that is wrong, not metadata that is unfamiliar.
        """
        document, problems = well_formed_array_v3(value)
        if document is not None:
            # Read whatever the shape allowed, so a structural problem does
            # not hide the semantic ones behind it.
            array, found = read_array_v3(document, context)
            _, composed = refine_array_v3(array)
            problems = (*problems, *found, *composed)
            if len(problems) == 0:
                return array
        raise MetadataValidationError(problems)

must_understand_fields property

must_understand_fields: dict[str, ZarrV3ExtensionField]

The fields outside the spec's that do not say must_understand: false.

A reader must refuse to open the array if this holds a field it does not recognize. Recognition is the reader's own knowledge: a top-level field is no extension point, so no scope claims one, and the document partitions by obligation and leaves the verdict to its reader, as ZarrV3ArrayMetadata.must_understand_fields does. An extension point the scope does not claim is the same question asked of an Opaque.

__post_init__

__post_init__() -> None

Refuse a field holding anything but an entity of its kind or an Opaque.

read_array_v3 builds a document that holds what it says by construction; this is the same guarantee for one built by hand, located at the field.

Source code in src/zarr_metadata/v3/_document.py

def __post_init__(self) -> None:
    """Refuse a field holding anything but an entity of its kind or an `Opaque`.

    `read_array_v3` builds a document that holds what it says by
    construction; this is the same guarantee for one built by hand,
    located at the field.
    """
    found = (
        *_as_object(self.document),
        *held_problems(self.data_type, DataTypeEntity, ("data_type",)),
        *held_problems(self.chunk_grid, ChunkGridEntity, ("chunk_grid",)),
        *held_problems(
            self.chunk_key_encoding, ChunkKeyEncodingEntity, ("chunk_key_encoding",)
        ),
        *(
            entry
            for index, codec in enumerate(self.codecs)
            for entry in held_problems(codec, CodecEntity, ("codecs", index))
        ),
        *(
            entry
            for index, transformer in enumerate(self.storage_transformers)
            for entry in held_problems(
                transformer, StorageTransformerEntity, ("storage_transformers", index)
            )
        ),
    )
    if len(found) != 0:
        raise MetadataValidationError(found)

canonical

canonical() -> ArrayDocumentV3

This document in the simplest form that means the same thing.

Each entity in its own canonical form and in its own spelling of the envelope -- the bare name when nothing is configured, no must_understand, which means what absence means -- and the one rule that is the document's own: dimension_names of nothing but nulls says what omitting the field says. A transformation, asked for by canonicalize_array_metadata_v3; to_json does not apply it. What comes back is a document written that way, so writing it changes nothing further.

Source code in src/zarr_metadata/v3/_document.py

def canonical(self) -> ArrayDocumentV3:
    """This document in the simplest form that means the same thing.

    Each entity in its own canonical form and in its own spelling of
    the envelope -- the bare name when nothing is configured, no
    `must_understand`, which means what absence means -- and the one
    rule that is the document's own: `dimension_names` of nothing
    but nulls says what omitting the field says. A *transformation*,
    asked for by `canonicalize_array_metadata_v3`; `to_json` does
    not apply it. What comes back is a document written that way, so
    writing it changes nothing further.
    """
    simplified = replace(
        self,
        data_type=self.data_type.canonical(),
        chunk_grid=self.chunk_grid.canonical(),
        chunk_key_encoding=self.chunk_key_encoding.canonical(),
        codecs=tuple(codec.canonical() for codec in self.codecs),
        storage_transformers=tuple(entry.canonical() for entry in self.storage_transformers),
    )
    document = {**self.document, **_rendered(simplified)}
    names = document.get("dimension_names")
    if isinstance(names, tuple) and all(
        entry is None for entry in cast("tuple[object, ...]", names)
    ):
        del document["dimension_names"]
    return replace(simplified, document=document)

from_json classmethod

from_json(
    value: object, *, context: Context = CORE_AND_EXTENSIONS
) -> ArrayDocumentV3

A v3 array document read into entities, or raise.

The reader's front door, and the one entry point that fails fast: all three layers, and either every extension point is read and the whole composes, or a single MetadataValidationError carries every reason it does not -- structural and semantic together. Use validate_array_metadata_v3 instead when you want the problems as data, and the layers themselves when you want to stop between them.

A name this context does not model is not a failure. It comes back as an Opaque marked out_of_scope, because a document may legitimately use an extension this reader does not know, and refusing it would make openness unimplementable. What fails is metadata that is wrong, not metadata that is unfamiliar.

Source code in src/zarr_metadata/v3/_document.py

@classmethod
def from_json(cls, value: object, *, context: Context = CORE_AND_EXTENSIONS) -> ArrayDocumentV3:
    """A v3 array document read into entities, or raise.

    The reader's front door, and the one entry point that fails fast:
    all three layers, and either every extension point is read and
    the whole composes, or a single `MetadataValidationError` carries
    every reason it does not -- structural and semantic together. Use
    `validate_array_metadata_v3` instead when you want the problems
    as data, and the layers themselves when you want to stop between
    them.

    A name this `context` does not model is *not* a failure. It comes
    back as an `Opaque` marked `out_of_scope`, because a document may
    legitimately use an extension this reader does not know, and
    refusing it would make openness unimplementable. What fails is
    metadata that is wrong, not metadata that is unfamiliar.
    """
    document, problems = well_formed_array_v3(value)
    if document is not None:
        # Read whatever the shape allowed, so a structural problem does
        # not hide the semantic ones behind it.
        array, found = read_array_v3(document, context)
        _, composed = refine_array_v3(array)
        problems = (*problems, *found, *composed)
        if len(problems) == 0:
            return array
    raise MetadataValidationError(problems)

to_json

to_json() -> dict[str, JSONValue]

The document as it was written, with each entity's members as the entity has them.

Faithful: a document read and written comes out as it went in, an entity's envelope included -- {"name": "crc32c"} stays an object, an empty configuration and a must_understand of true stay written -- because the document knows the spelling it read and puts it back around what the entity writes. What changed is what changes: a member replaced through with_configuration is written as the entity now has it, and an entity put in by hand is written as it writes itself. A field the document did not have is not invented, and one it wrote as something no entity could be read from stands as written. Ask canonical first for the simplest equivalent spelling.

Source code in src/zarr_metadata/v3/_document.py

def to_json(self) -> dict[str, JSONValue]:
    """The document as it was written, with each entity's members as the entity has them.

    Faithful: a document read and written comes out as it went in,
    an entity's envelope included -- `{"name": "crc32c"}` stays an
    object, an empty `configuration` and a `must_understand` of
    `true` stay written -- because the document knows the spelling
    it read and puts it back around what the entity writes. What
    changed is what changes: a member replaced through
    `with_configuration` is written as the entity now has it, and an
    entity put in by hand is written as it writes itself. A field
    the document did not have is not invented, and one it wrote as
    something no entity could be read from stands as written. Ask
    `canonical` first for the simplest equivalent spelling.
    """
    return {
        **self.document,
        **{
            key: cast("JSONValue", _as_written(self.document[key], value))
            for key, value in _rendered(self).items()
        },
    }

ArrayParts dataclass

Every part of an array a codec will be handed, and their type.

The parts an array is divided into, not the fields of its metadata. Plural deliberately: one pipeline encodes every chunk, so a rule about it quantifies over all of them — a shard's inner chunk shape must divide every chunk, which under a rectilinear grid is several different lengths.

data_type is the coerced data type, so a rule asks it what it is rather than comparing names, and it is None where the element type is undetermined while the array itself is not. That happens inside a shard: the inner grid is the sharding codec's own chunk_shape whatever reached it, so an unreadable codec upstream costs the type and not the parts. None in place of the whole value means something else again — that there is no array here at all, past the array->bytes boundary or beyond a codec that could have changed anything.

Source code in src/zarr_metadata/v3/_parts.py

@dataclass(frozen=True, slots=True)
class ArrayParts:
    """Every part of an array a codec will be handed, and their type.

    The parts an array is divided into, not the fields of its metadata.
    Plural deliberately: one pipeline encodes every chunk, so a rule about
    it quantifies over all of them — a shard's inner chunk shape must
    divide *every* chunk, which under a rectilinear grid is several
    different lengths.

    `data_type` is the coerced data type, so a rule asks it what it is
    rather than comparing names, and it is `None` where the element type
    is undetermined
    while the array itself is not. That happens inside a shard: the inner
    grid is the sharding codec's own `chunk_shape` whatever reached it, so
    an unreadable codec upstream costs the type and not the parts. `None`
    in place of the whole value means something else again — that there is
    no array here at all, past the array->bytes boundary or beyond a codec
    that could have changed anything.
    """

    grid: ChunkGrid
    data_type: DataTypeEntity | None

    def with_grid(self, grid: ChunkGrid) -> ArrayParts:
        return replace(self, grid=grid)

    def with_data_type(self, data_type: DataTypeEntity | None) -> ArrayParts:
        return replace(self, data_type=data_type)

BytesBytesCodec dataclass

Bases: CodecEntity

A codec that transforms bytes, after the array is gone.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class BytesBytesCodec(CodecEntity):
    """A codec that transforms bytes, after the array is gone."""

ChunkGrid dataclass

The division of an array into the parts a codec pipeline encodes.

Nothing here is the metadata: a grid entity keeps its own, and what reaches a codec is the division, not the spelling of it. A derived grid -- the regular one a sharding codec imposes, or a transposed one -- has no metadata to keep anyway.

Source code in src/zarr_metadata/v3/_parts.py

@dataclass(frozen=True, slots=True)
class ChunkGrid:
    """The division of an array into the parts a codec pipeline encodes.

    Nothing here is the metadata: a grid entity keeps its own, and what
    reaches a codec is the division, not the spelling of it. A derived
    grid -- the regular one a sharding codec imposes, or a transposed one
    -- has no metadata to keep anyway.
    """

    rank: int | None
    extents: Extents | None

    @classmethod
    def unreadable(cls, array_shape: object) -> ChunkGrid:
        """A grid nothing is known about but the rank the array pins.

        Every third-party grid, and every modelled one whose own metadata
        could not be read: the array still has a rank, and a rule about
        rank is still answerable.
        """
        rank = _rank_of(array_shape)
        return cls(rank, None if rank is None else (None,) * rank)

    @classmethod
    def derived(cls, extents: Extents) -> ChunkGrid:
        """A grid this package computed rather than read from a document."""
        return cls(len(extents), extents)

    @classmethod
    def regular(cls, lengths: Sequence[object]) -> ChunkGrid:
        """The regular grid a sharding codec's `chunk_shape` imposes."""
        return cls.derived(_uniform(lengths))

    def permuted(self, order: Sequence[int]) -> ChunkGrid:
        """This grid with its dimensions reordered by `order`.

        A transposed grid is still a grid — permuting a regular one gives
        a regular one — but it is no longer the grid the document wrote,
        so the metadata does not survive the trip.

        Declines on anything that is not a permutation of this grid's rank.
        The caller checks that too and reports it, but an order is only
        shape-validated as a tuple of integers, so this must not be the
        thing that decides whether a validator raises `IndexError`.
        """
        if self.extents is None or sorted(order) != list(range(len(self.extents))):
            return ChunkGrid(self.rank, None)
        return ChunkGrid.derived(tuple(self.extents[axis] for axis in order))

    def axis(self, dimension: int) -> frozenset[int] | None:
        """The lengths `dimension`'s chunks take, or None if undetermined."""
        if self.extents is None or dimension >= len(self.extents):
            return None
        return self.extents[dimension]

axis

axis(dimension: int) -> frozenset[int] | None

The lengths dimension's chunks take, or None if undetermined.

Source code in src/zarr_metadata/v3/_parts.py

def axis(self, dimension: int) -> frozenset[int] | None:
    """The lengths `dimension`'s chunks take, or None if undetermined."""
    if self.extents is None or dimension >= len(self.extents):
        return None
    return self.extents[dimension]

derived classmethod

derived(extents: Extents) -> ChunkGrid

A grid this package computed rather than read from a document.

Source code in src/zarr_metadata/v3/_parts.py

@classmethod
def derived(cls, extents: Extents) -> ChunkGrid:
    """A grid this package computed rather than read from a document."""
    return cls(len(extents), extents)

permuted

permuted(order: Sequence[int]) -> ChunkGrid

This grid with its dimensions reordered by order.

A transposed grid is still a grid — permuting a regular one gives a regular one — but it is no longer the grid the document wrote, so the metadata does not survive the trip.

Declines on anything that is not a permutation of this grid's rank. The caller checks that too and reports it, but an order is only shape-validated as a tuple of integers, so this must not be the thing that decides whether a validator raises IndexError.

Source code in src/zarr_metadata/v3/_parts.py

def permuted(self, order: Sequence[int]) -> ChunkGrid:
    """This grid with its dimensions reordered by `order`.

    A transposed grid is still a grid — permuting a regular one gives
    a regular one — but it is no longer the grid the document wrote,
    so the metadata does not survive the trip.

    Declines on anything that is not a permutation of this grid's rank.
    The caller checks that too and reports it, but an order is only
    shape-validated as a tuple of integers, so this must not be the
    thing that decides whether a validator raises `IndexError`.
    """
    if self.extents is None or sorted(order) != list(range(len(self.extents))):
        return ChunkGrid(self.rank, None)
    return ChunkGrid.derived(tuple(self.extents[axis] for axis in order))

regular classmethod

regular(lengths: Sequence[object]) -> ChunkGrid

The regular grid a sharding codec's chunk_shape imposes.

Source code in src/zarr_metadata/v3/_parts.py

@classmethod
def regular(cls, lengths: Sequence[object]) -> ChunkGrid:
    """The regular grid a sharding codec's `chunk_shape` imposes."""
    return cls.derived(_uniform(lengths))

unreadable classmethod

unreadable(array_shape: object) -> ChunkGrid

A grid nothing is known about but the rank the array pins.

Every third-party grid, and every modelled one whose own metadata could not be read: the array still has a rank, and a rule about rank is still answerable.

Source code in src/zarr_metadata/v3/_parts.py

@classmethod
def unreadable(cls, array_shape: object) -> ChunkGrid:
    """A grid nothing is known about but the rank the array pins.

    Every third-party grid, and every modelled one whose own metadata
    could not be read: the array still has a rank, and a rule about
    rank is still answerable.
    """
    rank = _rank_of(array_shape)
    return cls(rank, None if rank is None else (None,) * rank)

ChunkGridEntity dataclass

Bases: MetadataEntity

An entity that divides an array into the parts a pipeline encodes.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class ChunkGridEntity(MetadataEntity):
    """An entity that divides an array into the parts a pipeline encodes."""

    def shape_problems(self, array_shape: object) -> tuple[ValidationProblem, ...]:
        """Why this grid does not divide an array of `array_shape`.

        Locations are relative to the grid's `configuration`. Default:
        nothing, for a grid this package reads but has no such rule for.
        """
        return ()

    @abstractmethod
    def grid(self, array_shape: object) -> ChunkGrid:
        """What this grid divides an array of `array_shape` into.

        The array shape is a parameter because neither determines a grid
        alone: a grid whose own metadata cannot be read still has the
        array's rank, and rank is enough for several rules.
        """

grid abstractmethod

grid(array_shape: object) -> ChunkGrid

What this grid divides an array of array_shape into.

The array shape is a parameter because neither determines a grid alone: a grid whose own metadata cannot be read still has the array's rank, and rank is enough for several rules.

Source code in src/zarr_metadata/v3/_entity.py

@abstractmethod
def grid(self, array_shape: object) -> ChunkGrid:
    """What this grid divides an array of `array_shape` into.

    The array shape is a parameter because neither determines a grid
    alone: a grid whose own metadata cannot be read still has the
    array's rank, and rank is enough for several rules.
    """

shape_problems

shape_problems(
    array_shape: object,
) -> tuple[ValidationProblem, ...]

Why this grid does not divide an array of array_shape.

Locations are relative to the grid's configuration. Default: nothing, for a grid this package reads but has no such rule for.

Source code in src/zarr_metadata/v3/_entity.py

def shape_problems(self, array_shape: object) -> tuple[ValidationProblem, ...]:
    """Why this grid does not divide an array of `array_shape`.

    Locations are relative to the grid's `configuration`. Default:
    nothing, for a grid this package reads but has no such rule for.
    """
    return ()

ChunkKeyEncodingEntity dataclass

Bases: MetadataEntity

An entity that says how a chunk's coordinates become a store key.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class ChunkKeyEncodingEntity(MetadataEntity):
    """An entity that says how a chunk's coordinates become a store key."""

CodecEntity dataclass

Bases: MetadataEntity

An entity that occupies a position in the codec pipeline.

Of one of three kinds, each a base class: ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec. The kind fixes where in the pipeline the codec may stand, and what it must answer.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class CodecEntity(MetadataEntity):
    """An entity that occupies a position in the codec pipeline.

    Of one of three kinds, each a base class: `ArrayArrayCodec`,
    `ArrayBytesCodec`, `BytesBytesCodec`. The kind fixes where in the
    pipeline the codec may stand, and what it must answer.
    """

    variable_size: ClassVar[bool]
    """Whether this codec's output size depends on the bytes it is given.

    A compressor's does, so a shard index encoded with one has no size
    derivable from metadata alone, and the shard cannot be read. Every
    codec says, because a default in either direction is a verdict.
    """

    def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]:
        """Why this codec cannot be applied to the array that reaches it.

        `incoming` is None once the chain can no longer say what reaches
        here, and the default answer to that is nothing: declining beats
        guessing. Locations are relative to this codec's `configuration`;
        an empty one lands on the codec itself.
        """
        return ()

    def inner_pipelines(
        self, incoming: ArrayParts | None
    ) -> Mapping[str, tuple[Sequence[CodecEntity | Opaque], ArrayParts | None]]:
        """The pipelines this codec holds, by the member holding each, with what each is handed.

        A shard holds two: its `codecs`, handed its inner chunk, and its
        `index_codecs`, handed the shard index. Refinement walks them as
        it walks the pipeline this codec stands in, locating what it
        finds under the member, so a codec that holds pipelines says
        which and what they receive, and judges nothing inside them
        itself. Default: none.
        """
        return {}

variable_size class-attribute

variable_size: bool

Whether this codec's output size depends on the bytes it is given.

A compressor's does, so a shard index encoded with one has no size derivable from metadata alone, and the shard cannot be read. Every codec says, because a default in either direction is a verdict.

incoming_problems

incoming_problems(
    incoming: ArrayParts | None,
) -> tuple[ValidationProblem, ...]

Why this codec cannot be applied to the array that reaches it.

incoming is None once the chain can no longer say what reaches here, and the default answer to that is nothing: declining beats guessing. Locations are relative to this codec's configuration; an empty one lands on the codec itself.

Source code in src/zarr_metadata/v3/_entity.py

def incoming_problems(self, incoming: ArrayParts | None) -> tuple[ValidationProblem, ...]:
    """Why this codec cannot be applied to the array that reaches it.

    `incoming` is None once the chain can no longer say what reaches
    here, and the default answer to that is nothing: declining beats
    guessing. Locations are relative to this codec's `configuration`;
    an empty one lands on the codec itself.
    """
    return ()

inner_pipelines

inner_pipelines(
    incoming: ArrayParts | None,
) -> Mapping[
    str,
    tuple[
        Sequence[CodecEntity | Opaque], ArrayParts | None
    ],
]

The pipelines this codec holds, by the member holding each, with what each is handed.

A shard holds two: its codecs, handed its inner chunk, and its index_codecs, handed the shard index. Refinement walks them as it walks the pipeline this codec stands in, locating what it finds under the member, so a codec that holds pipelines says which and what they receive, and judges nothing inside them itself. Default: none.

Source code in src/zarr_metadata/v3/_entity.py

def inner_pipelines(
    self, incoming: ArrayParts | None
) -> Mapping[str, tuple[Sequence[CodecEntity | Opaque], ArrayParts | None]]:
    """The pipelines this codec holds, by the member holding each, with what each is handed.

    A shard holds two: its `codecs`, handed its inner chunk, and its
    `index_codecs`, handed the shard index. Refinement walks them as
    it walks the pipeline this codec stands in, locating what it
    finds under the member, so a codec that holds pipelines says
    which and what they receive, and judges nothing inside them
    itself. Default: none.
    """
    return {}

ComplexDataType dataclass

Bases: DataTypeEntity

A complex number: a [real, imag] pair of the component float type.

Source code in src/zarr_metadata/v3/data_type/_families.py

@dataclass(frozen=True)
class ComplexDataType(DataTypeEntity):
    """A complex number: a `[real, imag]` pair of the component float type."""

    configuration: Configuration = field(default_factory=Configuration)

    scalar_storage: ClassVar[StorageClass] = "multi_byte"
    component: ClassVar[type[FloatDataType]]

    def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
        pair = as_sequence(value)
        if pair is None or len(pair) != 2:
            return problem(loc, f"expected a [real, imag] pair, got {value!r}", "invalid_value")
        component = type(self).component()
        return tuple(
            ValidationProblem(found.loc, f"invalid component: {found.message}", found.kind)
            for index, part in enumerate(pair)
            for found in component.fill_value_problems(part, (*loc, index))
        )

configuration class-attribute instance-attribute

configuration: Configuration = field(
    default_factory=Configuration
)

The record of this entity's members.

An entity with members narrows it to its own record, configuration: GzipOptions, its one positional argument. An entity of a bare name defaults it to the empty record -- configuration: Configuration = field(default_factory=Configuration) -- so that Crc32cCodec() builds; coerce passes the record either way.

fill_value_problems

fill_value_problems(
    value: object, loc: Loc = ()
) -> tuple[ValidationProblem, ...]

Why value is not a fill value of this type, if it is not.

Every data type answers this; one that accepts any fill value says so with return ().

Source code in src/zarr_metadata/v3/data_type/_families.py

def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
    pair = as_sequence(value)
    if pair is None or len(pair) != 2:
        return problem(loc, f"expected a [real, imag] pair, got {value!r}", "invalid_value")
    component = type(self).component()
    return tuple(
        ValidationProblem(found.loc, f"invalid component: {found.message}", found.kind)
        for index, part in enumerate(pair)
        for found in component.fill_value_problems(part, (*loc, index))
    )

Configuration dataclass

What an entity is configured with: a record of its members, and the rules on them.

A frozen dataclass whose fields are the configuration's members, each a shape JSON takes; the entity names it in its configuration field, and an entity of a bare name defaults it to this one, empty, since the spec makes an absent configuration and an empty one the same. problems is where everything finer than a type goes -- a bound, a rule about one member, members read together -- yielding each problem as it is found, located relative to the configuration. A reader stops at the first or collects them all, as it needs: the entity's constructor stops at the first, coerce reports every one, and BloscOptions(...).problems() answers without an entity at all.

The constructor refuses a member of the wrong type, so a record is well-typed however it was built -- by hand, through replace, through an entity's with_configuration -- and the rules can trust what they read. Values are the rules' business, and the entity's constructor asks them.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class Configuration:
    """What an entity is configured with: a record of its members, and the rules on them.

    A frozen dataclass whose fields are the configuration's members,
    each a shape JSON takes; the entity names it in its `configuration`
    field, and an entity of a bare name defaults it to this one, empty,
    since the spec makes an absent configuration and an empty one the
    same.
    `problems` is where everything finer than a type goes -- a
    bound, a rule about one member, members read together -- yielding
    each problem as it is found, located relative to the configuration.
    A reader stops at the first or collects them all, as it needs: the
    entity's constructor stops at the first, `coerce` reports every one,
    and `BloscOptions(...).problems()` answers without an entity at all.

    The constructor refuses a member of the wrong type, so a record is
    well-typed however it was built -- by hand, through `replace`,
    through an entity's `with_configuration` -- and the rules can trust
    what they read. Values are the rules' business, and the entity's
    constructor asks them.
    """

    def __post_init__(self) -> None:
        """Refuse every member of the wrong type, so `GzipOptions(level="high")` raises."""
        found = mistyped(self)
        if len(found) != 0:
            raise MetadataValidationError(found)

    @classmethod
    def create_unchecked(cls, **members: object) -> Self:
        """This record with these members, built without the constructor's check.

        The one way around the check, for a caller that has just made
        it: the parser, which type-checked every member against the same
        annotations before building the record. Every field is given --
        the parser gives an absent optional member as `UNSET` -- since
        nothing here applies a default. Anything that has not checked
        the members goes through the constructor.
        """
        record = object.__new__(cls)
        for name, value in members.items():
            object.__setattr__(record, name, value)
        return record

    def problems(self) -> Iterator[ValidationProblem]:
        """Every reason these values are not allowed, yielded as found. Default: none."""
        yield from ()

__post_init__

__post_init__() -> None

Refuse every member of the wrong type, so GzipOptions(level="high") raises.

Source code in src/zarr_metadata/v3/_entity.py

def __post_init__(self) -> None:
    """Refuse every member of the wrong type, so `GzipOptions(level="high")` raises."""
    found = mistyped(self)
    if len(found) != 0:
        raise MetadataValidationError(found)

create_unchecked classmethod

create_unchecked(**members: object) -> Self

This record with these members, built without the constructor's check.

The one way around the check, for a caller that has just made it: the parser, which type-checked every member against the same annotations before building the record. Every field is given -- the parser gives an absent optional member as UNSET -- since nothing here applies a default. Anything that has not checked the members goes through the constructor.

Source code in src/zarr_metadata/v3/_entity.py

@classmethod
def create_unchecked(cls, **members: object) -> Self:
    """This record with these members, built without the constructor's check.

    The one way around the check, for a caller that has just made
    it: the parser, which type-checked every member against the same
    annotations before building the record. Every field is given --
    the parser gives an absent optional member as `UNSET` -- since
    nothing here applies a default. Anything that has not checked
    the members goes through the constructor.
    """
    record = object.__new__(cls)
    for name, value in members.items():
        object.__setattr__(record, name, value)
    return record

problems

problems() -> Iterator[ValidationProblem]

Every reason these values are not allowed, yielded as found. Default: none.

Source code in src/zarr_metadata/v3/_entity.py

def problems(self) -> Iterator[ValidationProblem]:
    """Every reason these values are not allowed, yielded as found. Default: none."""
    yield from ()

Context dataclass

The entities in scope while metadata is being read.

A value, with no reading of its own: resolve reads a field in it, and claimant is the one question it answers, which class a name belongs to. Built from classes with Context.of; extended with more by extended_with. What each class is registered as is read off it -- its kind is its base class, its key is its identifier -- so there is nothing to misfile.

Source code in src/zarr_metadata/v3/_registry.py

@dataclass(frozen=True, slots=True)
class Context:
    """The entities in scope while metadata is being read.

    A value, with no reading of its own: `resolve` reads a field in it,
    and `claimant` is the one question it answers, which class a name
    belongs to. Built from classes with `Context.of`; extended with more
    by `extended_with`. What each class is registered as is read off it
    -- its kind is its base class, its key is its `identifier` -- so
    there is nothing to misfile.
    """

    tables: Tables

    @classmethod
    def of(cls, *entities: type[MetadataEntity]) -> Context:
        """A scope of exactly these entities; a later one takes over an identifier from an earlier."""
        tables: dict[type[MetadataEntity], dict[str, type[MetadataEntity]]] = {
            kind: {} for kind in KINDS
        }
        for entity in entities:
            kind = _registrable(entity)
            tables[kind][entity.identifier] = entity
        return cls(
            MappingProxyType({kind: MappingProxyType(table) for kind, table in tables.items()})
        )

    def extended_with(self, *entities: type[MetadataEntity]) -> Context:
        """This scope, plus entities of your own.

        A name already registered under the same kind is taken over by
        what is passed here, which is how a reader substitutes its own
        reading of a codec the package already models.
        """
        return Context.of(*self.entities(), *entities)

    def entities(self) -> tuple[type[MetadataEntity], ...]:
        """Every entity in scope, kind by kind."""
        return tuple(entity for table in self.tables.values() for entity in table.values())

    def claimant(self, kind: type[EntityT], name: str) -> type[EntityT] | None:
        """The class in scope that claims `name` as an entity of `kind`; None if none does.

        Asks each class registered under the kind's kind whether the name
        is its own -- a family claims every `r<N>` -- rather than looking
        a key up, so the identifier keys exist for `extended_with` to
        take a name over, not for lookup. A class that claims the name
        but is not a `kind` -- `transpose` asked for as a
        `BytesBytesCodec` -- is none; `resolve` asks with the kind's kind
        to tell that case from a name nothing claims.
        """
        registered = kind_of(kind)
        if registered is None:
            return None
        table = self.tables.get(registered, {})
        entity = next((candidate for candidate in table.values() if candidate.accepts(name)), None)
        if entity is None or not issubclass(entity, kind):
            return None
        return entity

claimant

claimant(
    kind: type[EntityT], name: str
) -> type[EntityT] | None

The class in scope that claims name as an entity of kind; None if none does.

Asks each class registered under the kind's kind whether the name is its own -- a family claims every r<N> -- rather than looking a key up, so the identifier keys exist for extended_with to take a name over, not for lookup. A class that claims the name but is not a kind -- transpose asked for as a BytesBytesCodec -- is none; resolve asks with the kind's kind to tell that case from a name nothing claims.

Source code in src/zarr_metadata/v3/_registry.py

def claimant(self, kind: type[EntityT], name: str) -> type[EntityT] | None:
    """The class in scope that claims `name` as an entity of `kind`; None if none does.

    Asks each class registered under the kind's kind whether the name
    is its own -- a family claims every `r<N>` -- rather than looking
    a key up, so the identifier keys exist for `extended_with` to
    take a name over, not for lookup. A class that claims the name
    but is not a `kind` -- `transpose` asked for as a
    `BytesBytesCodec` -- is none; `resolve` asks with the kind's kind
    to tell that case from a name nothing claims.
    """
    registered = kind_of(kind)
    if registered is None:
        return None
    table = self.tables.get(registered, {})
    entity = next((candidate for candidate in table.values() if candidate.accepts(name)), None)
    if entity is None or not issubclass(entity, kind):
        return None
    return entity

entities

entities() -> tuple[type[MetadataEntity], ...]

Every entity in scope, kind by kind.

Source code in src/zarr_metadata/v3/_registry.py

def entities(self) -> tuple[type[MetadataEntity], ...]:
    """Every entity in scope, kind by kind."""
    return tuple(entity for table in self.tables.values() for entity in table.values())

extended_with

extended_with(*entities: type[MetadataEntity]) -> Context

This scope, plus entities of your own.

A name already registered under the same kind is taken over by what is passed here, which is how a reader substitutes its own reading of a codec the package already models.

Source code in src/zarr_metadata/v3/_registry.py

def extended_with(self, *entities: type[MetadataEntity]) -> Context:
    """This scope, plus entities of your own.

    A name already registered under the same kind is taken over by
    what is passed here, which is how a reader substitutes its own
    reading of a codec the package already models.
    """
    return Context.of(*self.entities(), *entities)

of classmethod

of(*entities: type[MetadataEntity]) -> Context

A scope of exactly these entities; a later one takes over an identifier from an earlier.

Source code in src/zarr_metadata/v3/_registry.py

@classmethod
def of(cls, *entities: type[MetadataEntity]) -> Context:
    """A scope of exactly these entities; a later one takes over an identifier from an earlier."""
    tables: dict[type[MetadataEntity], dict[str, type[MetadataEntity]]] = {
        kind: {} for kind in KINDS
    }
    for entity in entities:
        kind = _registrable(entity)
        tables[kind][entity.identifier] = entity
    return cls(
        MappingProxyType({kind: MappingProxyType(table) for kind, table in tables.items()})
    )

DataTypeEntity dataclass

Bases: MetadataEntity

An entity that says how the array's scalars are stored.

Only data types answer that, and every rule that turns on it -- a bytes codec is pointless before a single-byte type, a struct field cannot be variable-length -- asks a data type rather than consulting a table of names.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class DataTypeEntity(MetadataEntity):
    """An entity that says how the array's scalars are stored.

    Only data types answer that, and every rule that turns on it -- a
    `bytes` codec is pointless before a single-byte type, a struct field
    cannot be variable-length -- asks a data type rather than consulting
    a table of names.
    """

    scalar_storage: ClassVar[StorageClass]

    def storage_class(self) -> StorageClass | None:
        """How one scalar occupies bytes, or None if undetermined.

        None only for a composite whose parts are not all in scope: an
        answer would be a guess, and the rules that ask decline instead.
        """
        return type(self).scalar_storage

    @abstractmethod
    def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
        """Why `value` is not a fill value of this type, if it is not.

        Every data type answers this; one that accepts any fill value
        says so with `return ()`.
        """

fill_value_problems abstractmethod

fill_value_problems(
    value: object, loc: Loc = ()
) -> tuple[ValidationProblem, ...]

Why value is not a fill value of this type, if it is not.

Every data type answers this; one that accepts any fill value says so with return ().

Source code in src/zarr_metadata/v3/_entity.py

@abstractmethod
def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
    """Why `value` is not a fill value of this type, if it is not.

    Every data type answers this; one that accepts any fill value
    says so with `return ()`.
    """

storage_class

storage_class() -> StorageClass | None

How one scalar occupies bytes, or None if undetermined.

None only for a composite whose parts are not all in scope: an answer would be a guess, and the rules that ask decline instead.

Source code in src/zarr_metadata/v3/_entity.py

def storage_class(self) -> StorageClass | None:
    """How one scalar occupies bytes, or None if undetermined.

    None only for a composite whose parts are not all in scope: an
    answer would be a guess, and the rules that ask decline instead.
    """
    return type(self).scalar_storage

FloatDataType dataclass

Bases: DataTypeEntity

A binary float. A fill value may be a number, a named non-finite, or hex.

Source code in src/zarr_metadata/v3/data_type/_families.py

@dataclass(frozen=True)
class FloatDataType(DataTypeEntity):
    """A binary float. A fill value may be a number, a named non-finite, or hex."""

    configuration: Configuration = field(default_factory=Configuration)

    scalar_storage: ClassVar[StorageClass] = "multi_byte"
    hex_parser: ClassVar[Callable[[str], object]]

    largest: ClassVar[float | None]
    """The largest finite magnitude this width holds, or None for float64.

    None because a Python float *is* a float64, so no literal that reaches
    here can exceed it.
    """

    def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
        if is_integer(value) or isinstance(value, float):
            largest = type(self).largest
            if largest is not None and abs(value) > largest:
                return problem(
                    loc,
                    f"expected a {type(self).identifier} value, got {value!r}",
                    "invalid_value",
                )
            return ()
        if not isinstance(value, str):
            return problem(loc, f"expected a number or string, got {value!r}", "invalid_value")
        if value in FLOAT_SPECIALS:
            return ()
        try:
            type(self).hex_parser(value)
        except ValueError:
            return problem(
                loc,
                f"expected a number, one of 'NaN'/'Infinity'/'-Infinity', or a "
                f"{type(self).identifier} hex string, got {value!r}",
                "invalid_value",
            )
        return ()

configuration class-attribute instance-attribute

configuration: Configuration = field(
    default_factory=Configuration
)

The record of this entity's members.

An entity with members narrows it to its own record, configuration: GzipOptions, its one positional argument. An entity of a bare name defaults it to the empty record -- configuration: Configuration = field(default_factory=Configuration) -- so that Crc32cCodec() builds; coerce passes the record either way.

largest class-attribute

largest: float | None

The largest finite magnitude this width holds, or None for float64.

None because a Python float is a float64, so no literal that reaches here can exceed it.

fill_value_problems

fill_value_problems(
    value: object, loc: Loc = ()
) -> tuple[ValidationProblem, ...]

Why value is not a fill value of this type, if it is not.

Every data type answers this; one that accepts any fill value says so with return ().

Source code in src/zarr_metadata/v3/data_type/_families.py

def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
    if is_integer(value) or isinstance(value, float):
        largest = type(self).largest
        if largest is not None and abs(value) > largest:
            return problem(
                loc,
                f"expected a {type(self).identifier} value, got {value!r}",
                "invalid_value",
            )
        return ()
    if not isinstance(value, str):
        return problem(loc, f"expected a number or string, got {value!r}", "invalid_value")
    if value in FLOAT_SPECIALS:
        return ()
    try:
        type(self).hex_parser(value)
    except ValueError:
        return problem(
            loc,
            f"expected a number, one of 'NaN'/'Infinity'/'-Infinity', or a "
            f"{type(self).identifier} hex string, got {value!r}",
            "invalid_value",
        )
    return ()

IntegerDataType dataclass

Bases: DataTypeEntity

A fixed-width integer. The width is the whole difference.

Source code in src/zarr_metadata/v3/data_type/_families.py

@dataclass(frozen=True)
class IntegerDataType(DataTypeEntity):
    """A fixed-width integer. The width is the whole difference."""

    configuration: Configuration = field(default_factory=Configuration)

    bounds: ClassVar[tuple[int, int]]

    def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
        low, high = type(self).bounds
        if not is_integer(value):
            return problem(loc, f"expected an integer, got {value!r}", "invalid_value")
        if not low <= value <= high:
            return problem(
                loc, f"expected an integer in [{low}, {high}], got {value!r}", "invalid_value"
            )
        return ()

configuration class-attribute instance-attribute

configuration: Configuration = field(
    default_factory=Configuration
)

The record of this entity's members.

An entity with members narrows it to its own record, configuration: GzipOptions, its one positional argument. An entity of a bare name defaults it to the empty record -- configuration: Configuration = field(default_factory=Configuration) -- so that Crc32cCodec() builds; coerce passes the record either way.

fill_value_problems

fill_value_problems(
    value: object, loc: Loc = ()
) -> tuple[ValidationProblem, ...]

Why value is not a fill value of this type, if it is not.

Every data type answers this; one that accepts any fill value says so with return ().

Source code in src/zarr_metadata/v3/data_type/_families.py

def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
    low, high = type(self).bounds
    if not is_integer(value):
        return problem(loc, f"expected an integer, got {value!r}", "invalid_value")
    if not low <= value <= high:
        return problem(
            loc, f"expected an integer in [{low}, {high}], got {value!r}", "invalid_value"
        )
    return ()

MetadataEntity dataclass

Bases: ABC

One named entity, coerced from its metadata.

An entity is well-typed and allowed however it was built. coerce builds one only from metadata it accepted, through create_unchecked once it has; by hand, the record's constructor refuses a member of the wrong type, the entity's refuses a record that is not its own and then a value the rules disallow, and replace and with_configuration go through both. An optional member is typed | UNSET with a default of UNSET, so absence is representable -- and distinct from a null the document wrote -- and a canonical spelling can leave it out.

Frozen, so an entity of hashable members is hashable. One holding a value out of scope is not, because that value is the JSON the document wrote and a JSON object is a dict -- the same way any frozen dataclass holding a list is unhashable. It cannot be an immutable mapping instead: MappingProxyType is unhashable too, and anything else stops json.dumps from serializing what to_json returns.

A subclass names its configuration record, a Configuration whose problems holds what the spec says beyond the members' types -- so BloscCodec(BloscOptions(clevel=99)) raises on the first, and coerce reports every one instead -- and writes canonical where two spellings of its members mean the same. An entity of a bare name defaults the field to the empty record. coerce and to_json are written once here, against what the record says.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class MetadataEntity(ABC):
    """One named entity, coerced from its metadata.

    An entity is well-typed and allowed however it was built. `coerce`
    builds one only from metadata it accepted, through `create_unchecked`
    once it has; by hand, the record's constructor refuses a member of
    the wrong type, the entity's refuses a record that is not its own
    and then a value the rules disallow, and `replace` and
    `with_configuration` go through both. An optional member is typed
    `| UNSET` with a default of `UNSET`, so absence is representable --
    and distinct from a `null` the document wrote -- and a canonical
    spelling can leave it out.

    Frozen, so an entity of hashable members is hashable. One holding a
    value out of scope is not, because that value is the JSON the document
    wrote and a JSON object is a `dict` -- the same way any frozen
    dataclass holding a list is unhashable. It cannot be an immutable
    mapping instead: `MappingProxyType` is unhashable too, and anything
    else stops `json.dumps` from serializing what `to_json` returns.

    A subclass names its configuration record, a `Configuration` whose
    `problems` holds what the spec says beyond the members' types -- so
    `BloscCodec(BloscOptions(clevel=99))` raises on the first, and
    `coerce` reports every one instead -- and writes `canonical` where
    two spellings of its members mean the same. An entity of a bare
    name defaults the field to the empty record. `coerce` and `to_json`
    are written once here, against what the record says.
    """

    configuration: Configuration
    """The record of this entity's members.

    An entity with members narrows it to its own record, `configuration:
    GzipOptions`, its one positional argument. An entity of a bare name
    defaults it to the empty record -- `configuration: Configuration =
    field(default_factory=Configuration)` -- so that `Crc32cCodec()`
    builds; `coerce` passes the record either way.
    """

    identifier: ClassVar[str]
    """The name this entity is registered under.

    Usually the `name` the metadata carries. The raw-bytes data types are
    the exception: every `r<N>` spelling is one family, so the family gets
    an invented identifier that no real name can collide with.
    """

    @classmethod
    def name_problems(cls, name: str) -> Iterator[ValidationProblem]:
        """Why `name`, which `accepts` claimed, is not a well-formed name of this family.

        For a family, whose names carry data -- `r<N>` -- and which claims
        a malformed member so that it is reported rather than waved
        through as an unknown extension. Locations are relative to the
        entity: `()`. Default: none, for an entity of one name.
        """
        yield from ()

    def __post_init__(self) -> None:
        """Refuse a record that is not this entity's own, then the first problem the rules find.

        The runtime half of the entity's type, as the record's constructor
        is of the record's: `GzipCodec(BloscOptions(...))` and a family
        member carrying a name that is not a string are refused before
        any rule reads them. Then `BloscCodec(BloscOptions(clevel=99))`
        raises on the first problem the rules yield.
        """
        plan = _plan(type(self))
        if not isinstance(self.configuration, plan.record):
            raise MetadataValidationError(
                problem(
                    (),
                    f"expected a {plan.record.__name__} configuration, got "
                    f"{type(self.configuration).__name__}",
                )
            )
        name: object = self.identifier if plan.from_name is None else getattr(self, plan.from_name)
        if not isinstance(name, str):
            raise MetadataValidationError(problem((), f"expected a string name, got {name!r}"))
        first = next(type(self).name_problems(name), None)
        if first is None:
            first = next(self.configuration.problems(), None)
        if first is not None:
            raise MetadataValidationError((first,))

    @classmethod
    def accepts(cls, name: str) -> bool:
        """Whether `name` denotes this entity.

        Constant for all but the raw-bytes family, where one class covers
        every `r<N>`.
        """
        return name == cls.identifier

    @classmethod
    def create_unchecked(cls, **fields: object) -> Self:
        """This entity with these fields, built without the constructor's checks.

        The one way around them, for a caller that has just made them:
        `coerce`, which type-checked the record and ran the rules before
        building. Every field is given -- the record, and the carried
        name for a family -- since nothing here applies a default.
        Anything that has not checked goes through the constructor.
        """
        entity = object.__new__(cls)
        for name, value in fields.items():
            object.__setattr__(entity, name, value)
        return entity

    def with_configuration(self, **changes: object) -> Self:
        """This entity with these configuration members changed.

        `codec.with_configuration(typesize=UNSET)` is the record rebuilt
        through its constructor, which refuses a member of the wrong
        type, and the entity rebuilt through its own, which refuses a
        value the rules disallow -- the same checks as any construction,
        since pyright cannot see the members through `**changes`. A
        name that is not a member is refused the way `replace` refuses
        it.
        """
        return replace(self, configuration=replace(self.configuration, **changes))

    @classmethod
    def coerce(cls, value: JSONValue, context: Context) -> Coerced[Self]:
        """`value` as this entity, or the reasons it is not one: the class's validation routine.

        `resolve` relates a field's name to this class and hands it the
        field, refined JSON with arrays as tuples, which is what `value`
        is; this is what the class does with it. The configuration is
        parsed against the record the `configuration` field names,
        member by member; a member holding another entity is read in
        `context`, the scope this reading is happening in. An optional
        member the document left out is `UNSET` in the record, so no
        field's default decides what a document said. The entity is
        built only when every member of its own read -- its rules are
        written over a whole configuration -- and handed back only when
        everything inside it read too.

        The envelope is the field's, not the class's, and `resolve`
        judges it: a stray member or a `must_understand` of `false` is
        not reported here. Called on a class no scope has registered,
        this runs with none of registration's refusals having happened.
        """
        name, given, envelope = named_configuration(value)
        if name is None or not cls.accepts(name):
            return None, problem((), f"expected the {cls.identifier!r} entity")
        if len(envelope) != 0:
            return None, envelope
        plan = _plan(cls)
        carried: dict[str, str] = {} if plan.from_name is None else {plan.from_name: name}
        if given is None and plan.requires_configuration:
            return None, problem(
                ("configuration",),
                f"{cls.identifier!r} requires a configuration",
                "missing_key",
            )
        reading = _Reading(context, [])
        # A bare name's record has no members, so any key is an unknown one.
        record, own = plan.read({} if given is None else given, ("configuration",), reading)
        found = (*own, *reading.nested)
        if record is None:
            # An unknown key is survivable; a member that could not be
            # read is a hole, and judging around it would be guessing.
            return None, found
        # The rules, asked of the name and of the record before anything
        # is built: a name problem lands on the entity, a configuration
        # problem under the configuration.
        refused = (*cls.name_problems(name), *within((), tuple(record.problems())))
        if len(refused) != 0:
            # Values the spec disallows: reported rather than raised,
            # every one.
            return None, (*found, *refused)
        if any(entry.kind != "unknown_key" for entry in reading.nested):
            # A contained entity could not be read. This entity's own
            # rules ran -- an invalid inner is an `Opaque`, as an
            # out-of-scope one is -- but what is handed back is not an
            # entity that would be asked composition questions it cannot
            # answer.
            return None, found
        return cls.create_unchecked(configuration=record, **carried), found

    def canonical(self) -> Self:
        """This entity in the simplest form that means the same thing.

        A *transformation*, asked for by `canonicalize_array_metadata_v3`
        and by nothing else. `to_json` does not apply it, because writing
        a document back is not the same as asking for it to be rewritten:
        a reader that reads and writes should not change bytes it was not
        asked to change.

        The default is the entity itself. Override it where two spellings
        of the entity's members mean the same -- a rectilinear
        dimension's run-length encoding, a `typesize` that `noshuffle`
        ignores -- and, in an entity that contains entities, to put those
        in canonical form: `self.with_configuration(inner=self.inner.canonical())`.
        """
        return self

    def to_json(self) -> ZarrV3MetadataFieldJSON:
        """This entity as a document would write it.

        Written from the configuration record by the same declaration
        `coerce` reads it by, each member by the writer its annotation
        implies: the bare name when every member it holds is absent, the
        object otherwise, a contained entity through its own `to_json`,
        a JSON-valued member copied so the document is not a handle on
        the entity. Faithful to every member: read a document, write it
        back, and those come out as they went in. The envelope is
        written the entity's way -- the bare name when nothing is
        configured, the object otherwise, no `must_understand`, which
        means what absence means -- because an entity alone has no
        document to be faithful to; `ArrayDocumentV3.to_json` puts back
        the spelling the document used. Ask `canonical` first if you
        want the simplest equivalent spelling.

        An entity whose JSON is not its fields overrides this; none in
        the package does.
        """
        plan = _plan(type(self))
        name = self.identifier
        if plan.from_name is not None:
            carried = getattr(self, plan.from_name)
            name = carried if isinstance(carried, str) else name
        configuration = plan.write(self)
        if len(configuration) == 0:
            return name
        return {"name": name, "configuration": configuration}

configuration instance-attribute

configuration: Configuration

The record of this entity's members.

An entity with members narrows it to its own record, configuration: GzipOptions, its one positional argument. An entity of a bare name defaults it to the empty record -- configuration: Configuration = field(default_factory=Configuration) -- so that Crc32cCodec() builds; coerce passes the record either way.

identifier class-attribute

identifier: str

The name this entity is registered under.

Usually the name the metadata carries. The raw-bytes data types are the exception: every r<N> spelling is one family, so the family gets an invented identifier that no real name can collide with.

__post_init__

__post_init__() -> None

Refuse a record that is not this entity's own, then the first problem the rules find.

The runtime half of the entity's type, as the record's constructor is of the record's: GzipCodec(BloscOptions(...)) and a family member carrying a name that is not a string are refused before any rule reads them. Then BloscCodec(BloscOptions(clevel=99)) raises on the first problem the rules yield.

Source code in src/zarr_metadata/v3/_entity.py

def __post_init__(self) -> None:
    """Refuse a record that is not this entity's own, then the first problem the rules find.

    The runtime half of the entity's type, as the record's constructor
    is of the record's: `GzipCodec(BloscOptions(...))` and a family
    member carrying a name that is not a string are refused before
    any rule reads them. Then `BloscCodec(BloscOptions(clevel=99))`
    raises on the first problem the rules yield.
    """
    plan = _plan(type(self))
    if not isinstance(self.configuration, plan.record):
        raise MetadataValidationError(
            problem(
                (),
                f"expected a {plan.record.__name__} configuration, got "
                f"{type(self.configuration).__name__}",
            )
        )
    name: object = self.identifier if plan.from_name is None else getattr(self, plan.from_name)
    if not isinstance(name, str):
        raise MetadataValidationError(problem((), f"expected a string name, got {name!r}"))
    first = next(type(self).name_problems(name), None)
    if first is None:
        first = next(self.configuration.problems(), None)
    if first is not None:
        raise MetadataValidationError((first,))

accepts classmethod

accepts(name: str) -> bool

Whether name denotes this entity.

Constant for all but the raw-bytes family, where one class covers every r<N>.

Source code in src/zarr_metadata/v3/_entity.py

@classmethod
def accepts(cls, name: str) -> bool:
    """Whether `name` denotes this entity.

    Constant for all but the raw-bytes family, where one class covers
    every `r<N>`.
    """
    return name == cls.identifier

canonical

canonical() -> Self

This entity in the simplest form that means the same thing.

A transformation, asked for by canonicalize_array_metadata_v3 and by nothing else. to_json does not apply it, because writing a document back is not the same as asking for it to be rewritten: a reader that reads and writes should not change bytes it was not asked to change.

The default is the entity itself. Override it where two spellings of the entity's members mean the same -- a rectilinear dimension's run-length encoding, a typesize that noshuffle ignores -- and, in an entity that contains entities, to put those in canonical form: self.with_configuration(inner=self.inner.canonical()).

Source code in src/zarr_metadata/v3/_entity.py

def canonical(self) -> Self:
    """This entity in the simplest form that means the same thing.

    A *transformation*, asked for by `canonicalize_array_metadata_v3`
    and by nothing else. `to_json` does not apply it, because writing
    a document back is not the same as asking for it to be rewritten:
    a reader that reads and writes should not change bytes it was not
    asked to change.

    The default is the entity itself. Override it where two spellings
    of the entity's members mean the same -- a rectilinear
    dimension's run-length encoding, a `typesize` that `noshuffle`
    ignores -- and, in an entity that contains entities, to put those
    in canonical form: `self.with_configuration(inner=self.inner.canonical())`.
    """
    return self

coerce classmethod

coerce(value: JSONValue, context: Context) -> Coerced[Self]

value as this entity, or the reasons it is not one: the class's validation routine.

resolve relates a field's name to this class and hands it the field, refined JSON with arrays as tuples, which is what value is; this is what the class does with it. The configuration is parsed against the record the configuration field names, member by member; a member holding another entity is read in context, the scope this reading is happening in. An optional member the document left out is UNSET in the record, so no field's default decides what a document said. The entity is built only when every member of its own read -- its rules are written over a whole configuration -- and handed back only when everything inside it read too.

The envelope is the field's, not the class's, and resolve judges it: a stray member or a must_understand of false is not reported here. Called on a class no scope has registered, this runs with none of registration's refusals having happened.

Source code in src/zarr_metadata/v3/_entity.py

@classmethod
def coerce(cls, value: JSONValue, context: Context) -> Coerced[Self]:
    """`value` as this entity, or the reasons it is not one: the class's validation routine.

    `resolve` relates a field's name to this class and hands it the
    field, refined JSON with arrays as tuples, which is what `value`
    is; this is what the class does with it. The configuration is
    parsed against the record the `configuration` field names,
    member by member; a member holding another entity is read in
    `context`, the scope this reading is happening in. An optional
    member the document left out is `UNSET` in the record, so no
    field's default decides what a document said. The entity is
    built only when every member of its own read -- its rules are
    written over a whole configuration -- and handed back only when
    everything inside it read too.

    The envelope is the field's, not the class's, and `resolve`
    judges it: a stray member or a `must_understand` of `false` is
    not reported here. Called on a class no scope has registered,
    this runs with none of registration's refusals having happened.
    """
    name, given, envelope = named_configuration(value)
    if name is None or not cls.accepts(name):
        return None, problem((), f"expected the {cls.identifier!r} entity")
    if len(envelope) != 0:
        return None, envelope
    plan = _plan(cls)
    carried: dict[str, str] = {} if plan.from_name is None else {plan.from_name: name}
    if given is None and plan.requires_configuration:
        return None, problem(
            ("configuration",),
            f"{cls.identifier!r} requires a configuration",
            "missing_key",
        )
    reading = _Reading(context, [])
    # A bare name's record has no members, so any key is an unknown one.
    record, own = plan.read({} if given is None else given, ("configuration",), reading)
    found = (*own, *reading.nested)
    if record is None:
        # An unknown key is survivable; a member that could not be
        # read is a hole, and judging around it would be guessing.
        return None, found
    # The rules, asked of the name and of the record before anything
    # is built: a name problem lands on the entity, a configuration
    # problem under the configuration.
    refused = (*cls.name_problems(name), *within((), tuple(record.problems())))
    if len(refused) != 0:
        # Values the spec disallows: reported rather than raised,
        # every one.
        return None, (*found, *refused)
    if any(entry.kind != "unknown_key" for entry in reading.nested):
        # A contained entity could not be read. This entity's own
        # rules ran -- an invalid inner is an `Opaque`, as an
        # out-of-scope one is -- but what is handed back is not an
        # entity that would be asked composition questions it cannot
        # answer.
        return None, found
    return cls.create_unchecked(configuration=record, **carried), found

create_unchecked classmethod

create_unchecked(**fields: object) -> Self

This entity with these fields, built without the constructor's checks.

The one way around them, for a caller that has just made them: coerce, which type-checked the record and ran the rules before building. Every field is given -- the record, and the carried name for a family -- since nothing here applies a default. Anything that has not checked goes through the constructor.

Source code in src/zarr_metadata/v3/_entity.py

@classmethod
def create_unchecked(cls, **fields: object) -> Self:
    """This entity with these fields, built without the constructor's checks.

    The one way around them, for a caller that has just made them:
    `coerce`, which type-checked the record and ran the rules before
    building. Every field is given -- the record, and the carried
    name for a family -- since nothing here applies a default.
    Anything that has not checked goes through the constructor.
    """
    entity = object.__new__(cls)
    for name, value in fields.items():
        object.__setattr__(entity, name, value)
    return entity

name_problems classmethod

name_problems(name: str) -> Iterator[ValidationProblem]

Why name, which accepts claimed, is not a well-formed name of this family.

For a family, whose names carry data -- r<N> -- and which claims a malformed member so that it is reported rather than waved through as an unknown extension. Locations are relative to the entity: (). Default: none, for an entity of one name.

Source code in src/zarr_metadata/v3/_entity.py

@classmethod
def name_problems(cls, name: str) -> Iterator[ValidationProblem]:
    """Why `name`, which `accepts` claimed, is not a well-formed name of this family.

    For a family, whose names carry data -- `r<N>` -- and which claims
    a malformed member so that it is reported rather than waved
    through as an unknown extension. Locations are relative to the
    entity: `()`. Default: none, for an entity of one name.
    """
    yield from ()

to_json

to_json() -> ZarrV3MetadataFieldJSON

This entity as a document would write it.

Written from the configuration record by the same declaration coerce reads it by, each member by the writer its annotation implies: the bare name when every member it holds is absent, the object otherwise, a contained entity through its own to_json, a JSON-valued member copied so the document is not a handle on the entity. Faithful to every member: read a document, write it back, and those come out as they went in. The envelope is written the entity's way -- the bare name when nothing is configured, the object otherwise, no must_understand, which means what absence means -- because an entity alone has no document to be faithful to; ArrayDocumentV3.to_json puts back the spelling the document used. Ask canonical first if you want the simplest equivalent spelling.

An entity whose JSON is not its fields overrides this; none in the package does.

Source code in src/zarr_metadata/v3/_entity.py

def to_json(self) -> ZarrV3MetadataFieldJSON:
    """This entity as a document would write it.

    Written from the configuration record by the same declaration
    `coerce` reads it by, each member by the writer its annotation
    implies: the bare name when every member it holds is absent, the
    object otherwise, a contained entity through its own `to_json`,
    a JSON-valued member copied so the document is not a handle on
    the entity. Faithful to every member: read a document, write it
    back, and those come out as they went in. The envelope is
    written the entity's way -- the bare name when nothing is
    configured, the object otherwise, no `must_understand`, which
    means what absence means -- because an entity alone has no
    document to be faithful to; `ArrayDocumentV3.to_json` puts back
    the spelling the document used. Ask `canonical` first if you
    want the simplest equivalent spelling.

    An entity whose JSON is not its fields overrides this; none in
    the package does.
    """
    plan = _plan(type(self))
    name = self.identifier
    if plan.from_name is not None:
        carried = getattr(self, plan.from_name)
        name = carried if isinstance(carried, str) else name
    configuration = plan.write(self)
    if len(configuration) == 0:
        return name
    return {"name": name, "configuration": configuration}

with_configuration

with_configuration(**changes: object) -> Self

This entity with these configuration members changed.

codec.with_configuration(typesize=UNSET) is the record rebuilt through its constructor, which refuses a member of the wrong type, and the entity rebuilt through its own, which refuses a value the rules disallow -- the same checks as any construction, since pyright cannot see the members through **changes. A name that is not a member is refused the way replace refuses it.

Source code in src/zarr_metadata/v3/_entity.py

def with_configuration(self, **changes: object) -> Self:
    """This entity with these configuration members changed.

    `codec.with_configuration(typesize=UNSET)` is the record rebuilt
    through its constructor, which refuses a member of the wrong
    type, and the entity rebuilt through its own, which refuses a
    value the rules disallow -- the same checks as any construction,
    since pyright cannot see the members through `**changes`. A
    name that is not a member is refused the way `replace` refuses
    it.
    """
    return replace(self, configuration=replace(self.configuration, **changes))

MetadataValidationError

Bases: ValueError

Raised when a value fails structural metadata validation.

Carries every problem found (not just the first) in .problems, as an immutable tuple: a raised error is a finished report, and a caller inspecting it must not be able to edit the record.

Source code in src/zarr_metadata/model/_validation.py

class MetadataValidationError(ValueError):
    """Raised when a value fails structural metadata validation.

    Carries every problem found (not just the first) in `.problems`, as an
    immutable tuple: a raised error is a finished report, and a caller
    inspecting it must not be able to edit the record.
    """

    problems: tuple[ValidationProblem, ...]

    def __init__(self, problems: Sequence[ValidationProblem]) -> None:
        self.problems = tuple(problems)
        for entry in self.problems:
            # The type says so; the check is for the trap the type cannot
            # close: `problem()` returns a one-element tuple, and a list
            # of those passes here and fails far away, where a `loc` is
            # read off it.
            if not isinstance(entry, ValidationProblem):  # pyright: ignore[reportUnnecessaryIsInstance]
                msg = (
                    f"MetadataValidationError takes ValidationProblem values, got "
                    f"{type(entry).__name__}; `problem()` returns a tuple of them, so collect "
                    "with `extend`, not `append`"
                )
                raise TypeError(msg)
        super().__init__("\n".join(str(problem) for problem in self.problems))

NumpyTimeDataType dataclass

Bases: DataTypeEntity

A numpy time scalar: a signed 64-bit count of units, or NaT.

The two time types share their configuration -- a unit and a scale factor -- and the rule on it, so both live here with the family and neither sibling imports them from the other.

Source code in src/zarr_metadata/v3/data_type/_families.py

@dataclass(frozen=True)
class NumpyTimeDataType(DataTypeEntity):
    """A numpy time scalar: a signed 64-bit count of units, or `NaT`.

    The two time types share their configuration -- a unit and a scale
    factor -- and the rule on it, so both live here with the family and
    neither sibling imports them from the other.
    """

    configuration: NumpyTimeOptions

    scalar_storage: ClassVar[StorageClass] = "multi_byte"

    def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
        if value == "NaT":
            return ()
        if not is_integer(value):
            return problem(
                loc, f"expected a signed 64-bit integer or 'NaT', got {value!r}", "invalid_value"
            )
        if not -(2**63) <= value <= 2**63 - 1:
            return problem(loc, f"expected a signed 64-bit integer, got {value!r}", "invalid_value")
        return ()

configuration instance-attribute

configuration: NumpyTimeOptions

The record of this entity's members.

An entity with members narrows it to its own record, configuration: GzipOptions, its one positional argument. An entity of a bare name defaults it to the empty record -- configuration: Configuration = field(default_factory=Configuration) -- so that Crc32cCodec() builds; coerce passes the record either way.

fill_value_problems

fill_value_problems(
    value: object, loc: Loc = ()
) -> tuple[ValidationProblem, ...]

Why value is not a fill value of this type, if it is not.

Every data type answers this; one that accepts any fill value says so with return ().

Source code in src/zarr_metadata/v3/data_type/_families.py

def fill_value_problems(self, value: object, loc: Loc = ()) -> tuple[ValidationProblem, ...]:
    if value == "NaT":
        return ()
    if not is_integer(value):
        return problem(
            loc, f"expected a signed 64-bit integer or 'NaT', got {value!r}", "invalid_value"
        )
    if not -(2**63) <= value <= 2**63 - 1:
        return problem(loc, f"expected a signed 64-bit integer, got {value!r}", "invalid_value")
    return ()

Opaque dataclass

A metadata field this reading did not turn into an entity.

Carries the JSON the document wrote, so a reader holding a CodecEntity | Opaque has everything the document said in either case, and reason says which case it is. out_of_scope is a name no entity in this Context claims -- an extension this reader does not model, which is not an error and is the reader's cue to resolve it elsewhere. invalid is a name that was claimed and then refused, for the reasons reported alongside.

Answers to_json and canonical as an entity does, so a field typed CodecEntity | Opaque is written and simplified without asking which case it holds.

Built by the reader through create_unchecked, from JSON it has refined; the constructor checks one built by hand.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True, slots=True)
class Opaque:
    """A metadata field this reading did not turn into an entity.

    Carries the JSON the document wrote, so a reader holding a
    `CodecEntity | Opaque` has everything the document said in either
    case, and `reason` says which case it is. `out_of_scope` is a name no
    entity in this `Context` claims -- an extension this reader does not
    model, which is not an error and is the reader's cue to resolve it
    elsewhere. `invalid` is a name that *was* claimed and then refused,
    for the reasons reported alongside.

    Answers `to_json` and `canonical` as an entity does, so a field typed
    `CodecEntity | Opaque` is written and simplified without asking
    which case it holds.

    Built by the reader through `create_unchecked`, from JSON it has
    refined; the constructor checks one built by hand.
    """

    json: JSONValue
    reason: Literal["out_of_scope", "invalid"]

    def __post_init__(self) -> None:
        """Refuse a reason that is not one of the reader's two, and a `json` that is not JSON."""
        reason: object = self.reason
        found = (
            *(
                ()
                if reason in ("out_of_scope", "invalid")
                else problem(("reason",), f"expected 'out_of_scope' or 'invalid', got {reason!r}")
            ),
            *(
                ()
                if is_json(self.json)
                else problem(("json",), f"expected JSON, got {self.json!r}")
            ),
        )
        if len(found) != 0:
            raise MetadataValidationError(found)

    @classmethod
    def create_unchecked(cls, json: JSONValue, reason: Literal["out_of_scope", "invalid"]) -> Self:
        """An `Opaque` built without the constructor's check, for the reader, whose JSON is refined."""
        opaque = object.__new__(cls)
        object.__setattr__(opaque, "json", json)
        object.__setattr__(opaque, "reason", reason)
        return opaque

    def to_json(self) -> ZarrV3MetadataFieldJSON:
        """The JSON the document wrote, as it wrote it.

        An `Opaque` inside a built entity is out of scope -- an inner
        name no entity in scope claimed -- and its JSON passed the
        envelope check as a metadata field, which is what the cast says.
        """
        return cast("ZarrV3MetadataFieldJSON", self.json)

    def canonical(self) -> Self:
        """Itself: what was not read cannot be simplified."""
        return self

__post_init__

__post_init__() -> None

Refuse a reason that is not one of the reader's two, and a json that is not JSON.

Source code in src/zarr_metadata/v3/_entity.py

def __post_init__(self) -> None:
    """Refuse a reason that is not one of the reader's two, and a `json` that is not JSON."""
    reason: object = self.reason
    found = (
        *(
            ()
            if reason in ("out_of_scope", "invalid")
            else problem(("reason",), f"expected 'out_of_scope' or 'invalid', got {reason!r}")
        ),
        *(
            ()
            if is_json(self.json)
            else problem(("json",), f"expected JSON, got {self.json!r}")
        ),
    )
    if len(found) != 0:
        raise MetadataValidationError(found)

canonical

canonical() -> Self

Itself: what was not read cannot be simplified.

Source code in src/zarr_metadata/v3/_entity.py

def canonical(self) -> Self:
    """Itself: what was not read cannot be simplified."""
    return self

create_unchecked classmethod

create_unchecked(
    json: JSONValue,
    reason: Literal["out_of_scope", "invalid"],
) -> Self

An Opaque built without the constructor's check, for the reader, whose JSON is refined.

Source code in src/zarr_metadata/v3/_entity.py

@classmethod
def create_unchecked(cls, json: JSONValue, reason: Literal["out_of_scope", "invalid"]) -> Self:
    """An `Opaque` built without the constructor's check, for the reader, whose JSON is refined."""
    opaque = object.__new__(cls)
    object.__setattr__(opaque, "json", json)
    object.__setattr__(opaque, "reason", reason)
    return opaque

to_json

to_json() -> ZarrV3MetadataFieldJSON

The JSON the document wrote, as it wrote it.

An Opaque inside a built entity is out of scope -- an inner name no entity in scope claimed -- and its JSON passed the envelope check as a metadata field, which is what the cast says.

Source code in src/zarr_metadata/v3/_entity.py

def to_json(self) -> ZarrV3MetadataFieldJSON:
    """The JSON the document wrote, as it wrote it.

    An `Opaque` inside a built entity is out of scope -- an inner
    name no entity in scope claimed -- and its JSON passed the
    envelope check as a metadata field, which is what the cast says.
    """
    return cast("ZarrV3MetadataFieldJSON", self.json)

Pipeline dataclass

A codec pipeline refined against the array handed to it: what reaches each codec.

Source code in src/zarr_metadata/v3/_chain.py

@dataclass(frozen=True, slots=True)
class Pipeline:
    """A codec pipeline refined against the array handed to it: what reaches each codec."""

    stages: tuple[PipelineStage, ...]

PipelineStage dataclass

One position of a refined pipeline: the codec, and the array that reaches it.

Source code in src/zarr_metadata/v3/_chain.py

@dataclass(frozen=True, slots=True)
class PipelineStage:
    """One position of a refined pipeline: the codec, and the array that reaches it."""

    codec: CodecEntity | Opaque
    incoming: ArrayParts | None
    """What reaches this codec.

    None past the array->bytes boundary, where there is no array, and
    after a codec that could not say what it does to one.
    """
    inner: Mapping[str, Pipeline]
    """The pipelines this codec holds, refined, by the member that holds each; empty for most."""

incoming instance-attribute

incoming: ArrayParts | None

What reaches this codec.

None past the array->bytes boundary, where there is no array, and after a codec that could not say what it does to one.

inner instance-attribute

inner: Mapping[str, Pipeline]

The pipelines this codec holds, refined, by the member that holds each; empty for most.

RefinedArrayV3 dataclass

A v3 array document refined against its own array: the third layer's value.

parts is the array the codec pipeline is handed -- its chunks, under its grid, of its data type -- and pipeline is that pipeline resolved: at each position the codec and what reaches it, a shard's inner pipelines refined inside it. What a codec pipeline is built from, and what validating the composition finds on the way.

Source code in src/zarr_metadata/v3/_document.py

@dataclass(frozen=True, slots=True)
class RefinedArrayV3:
    """A v3 array document refined against its own array: the third layer's value.

    `parts` is the array the codec pipeline is handed -- its chunks,
    under its grid, of its data type -- and `pipeline` is that pipeline
    resolved: at each position the codec and what reaches it, a shard's
    inner pipelines refined inside it. What a codec pipeline is built
    from, and what validating the composition finds on the way.
    """

    array: ArrayDocumentV3
    parts: ArrayParts
    pipeline: Pipeline

StorageTransformerEntity dataclass

Bases: MetadataEntity

An entity that stands between the codec pipeline and the store.

Source code in src/zarr_metadata/v3/_entity.py

@dataclass(frozen=True)
class StorageTransformerEntity(MetadataEntity):
    """An entity that stands between the codec pipeline and the store."""

ValidationProblem dataclass

A single structural problem found while validating a metadata document.

loc is the path from the root of what was judged to the offending value, e.g. ("codecs", 0, "name"), and an empty loc refers to that root. The root is the document for the validators, the one field for a scope's coerce, and the configuration for an entity's rules and its constructor. kind classifies the failure mode for programmatic dispatch; message is the human-readable description.

Source code in src/zarr_metadata/model/_validation.py

@dataclass(frozen=True, slots=True)
class ValidationProblem:
    """A single structural problem found while validating a metadata document.

    `loc` is the path from the root of what was judged to the offending
    value, e.g. `("codecs", 0, "name")`, and an empty `loc` refers to that
    root. The root is the document for the validators, the one field for
    a scope's `coerce`, and the configuration for an entity's rules and
    its constructor.
    `kind` classifies the failure mode for programmatic dispatch; `message`
    is the human-readable description.
    """

    loc: tuple[str | int, ...]
    message: str
    kind: ProblemKind

    def __str__(self) -> str:
        location = ".".join(str(part) for part in self.loc) if self.loc else "<root>"
        return f"{location}: {self.message}"

is_integer

is_integer(value: object) -> TypeIs[int]

A JSON integer: an int, and not a bool.

True is an int in Python and true is not a number in JSON, so the two have to be told apart everywhere a number is expected.

Source code in src/zarr_metadata/v3/_typed_json.py

def is_integer(value: object) -> TypeIs[int]:
    """A JSON integer: an `int`, and not a `bool`.

    `True` is an `int` in Python and `true` is not a number in JSON, so
    the two have to be told apart everywhere a number is expected.
    """
    return not isinstance(value, bool) and isinstance(value, int)

named_configuration

named_configuration(
    value: object,
) -> tuple[
    str | None,
    Mapping[str, object] | None,
    tuple[ValidationProblem, ...],
]

Split metadata into (name, configuration, problems).

The shared shape every entity arrives in: a bare name, or an object carrying one. A None name means the value is not a metadata field at all; a None configuration means the bare spelling was used, or the key was left out. A configuration that is present and not an object is the one problem reported, at ("configuration",).

Source code in src/zarr_metadata/v3/_entity.py

def named_configuration(
    value: object,
) -> tuple[str | None, Mapping[str, object] | None, tuple[ValidationProblem, ...]]:
    """Split metadata into `(name, configuration, problems)`.

    The shared shape every entity arrives in: a bare name, or an object
    carrying one. A `None` name means the value is not a metadata field
    at all; a `None` configuration means the bare spelling was used, or
    the key was left out. A configuration that is present and not an
    object is the one problem reported, at `("configuration",)`.
    """
    if isinstance(value, str):
        return value, None, ()
    if not isinstance(value, Mapping):
        return None, None, ()
    entry = cast("Mapping[str, object]", value)
    name = entry.get("name")
    if not isinstance(name, str):
        return None, None, ()
    if "configuration" not in entry:
        return name, None, ()
    configuration = entry["configuration"]
    if not isinstance(configuration, Mapping):
        return name, None, problem(("configuration",), f"expected an object, got {configuration!r}")
    return name, cast("Mapping[str, object]", configuration), ()

problem

problem(
    loc: Loc,
    message: str,
    kind: ProblemKind = "invalid_type",
) -> tuple[ValidationProblem, ...]

One problem, as the one-element tuple every parser returns.

A tuple so that a parser can return it directly and a rule can found.extend(problem(...)) and raise MetadataValidationError(found) once. The default kind names a type mismatch; a value rule passes "invalid_value".

Source code in src/zarr_metadata/v3/_typed_json.py

def problem(
    loc: Loc, message: str, kind: ProblemKind = "invalid_type"
) -> tuple[ValidationProblem, ...]:
    """One problem, as the one-element tuple every parser returns.

    A tuple so that a parser can return it directly and a rule can
    `found.extend(problem(...))` and raise `MetadataValidationError(found)`
    once. The default `kind` names a type mismatch; a value rule passes
    `"invalid_value"`.
    """
    return (ValidationProblem(loc, message, kind),)

read_array_v3

read_array_v3(
    document: Mapping[str, JSONValue], context: Context
) -> tuple[ArrayDocumentV3, tuple[ValidationProblem, ...]]

The second layer: document's extension points, read in context.

Needs a scope. The one place that knows which of a document's fields holds which kind of entity; each is handed to read_field, its envelope having been judged with the document. Type-space only: what comes back is well-typed by construction, and the problems are the reasons some of it is not an entity.

Source code in src/zarr_metadata/v3/_document.py

def read_array_v3(
    document: Mapping[str, JSONValue], context: Context
) -> tuple[ArrayDocumentV3, tuple[ValidationProblem, ...]]:
    """The second layer: `document`'s extension points, read in `context`.

    Needs a scope. The one place that knows which of a document's fields
    holds which kind of entity; each is handed to `read_field`, its
    envelope having been judged with the document. Type-space only: what
    comes back is well-typed by construction, and the problems are the
    reasons some of it is not an entity.
    """
    data_type, found_1 = _read_one(context, DataTypeEntity, document, "data_type")
    chunk_grid, found_2 = _read_one(context, ChunkGridEntity, document, "chunk_grid")
    encoding, found_3 = _read_one(context, ChunkKeyEncodingEntity, document, "chunk_key_encoding")
    codecs, found_4 = _read_each(context, CodecEntity, document, "codecs")
    transformers, found_5 = _read_each(
        context, StorageTransformerEntity, document, "storage_transformers"
    )
    return (
        ArrayDocumentV3(
            document=document,
            data_type=data_type,
            chunk_grid=chunk_grid,
            chunk_key_encoding=encoding,
            codecs=codecs,
            storage_transformers=transformers,
        ),
        (*found_1, *found_2, *found_3, *found_4, *found_5),
    )

refine_array_v3

refine_array_v3(
    array: ArrayDocumentV3,
) -> tuple[RefinedArrayV3, tuple[ValidationProblem, ...]]

The third layer: array against its own array, and the pipeline resolved.

Needs the array: the fill value is judged by the data type it fills, the grid against the shape it divides, the dimension names counted against it, and the codec pipeline walked from the parts the grid and data type make, each codec handed what reaches it. A pipeline the document did not write as an array was not read, and is not judged as an empty one.

Source code in src/zarr_metadata/v3/_document.py

def refine_array_v3(array: ArrayDocumentV3) -> tuple[RefinedArrayV3, tuple[ValidationProblem, ...]]:
    """The third layer: `array` against its own array, and the pipeline resolved.

    Needs the array: the fill value is judged by the data type it fills,
    the grid against the shape it divides, the dimension names counted
    against it, and the codec pipeline walked from the parts the grid
    and data type make, each codec handed what reaches it. A pipeline
    the document did not write as an array was not read, and is not
    judged as an empty one.
    """
    parts = _parts(array)
    pipeline, composed = (
        refine_pipeline(array.codecs, parts, ("codecs",))
        if _listed(array.document, "codecs") is not None
        else (Pipeline(()), ())
    )
    problems = (
        *_fill_value_problems(array),
        *_grid_problems(array),
        *_dimension_names_problems(array),
        *composed,
    )
    return RefinedArrayV3(array, parts, pipeline), problems

resolve

resolve(
    data: object,
    kind: type[EntityT],
    context: Context,
    loc: Loc = (),
) -> tuple[EntityT | Opaque, tuple[ValidationProblem, ...]]

data, one metadata field, read as an entity of kind in context.

The reader for one field: the first two layers of reading a document, applied to a field on its own. The first needs nothing but the value -- data is refined to JSON, arrays as tuples, and judged as a metadata field, an extra member, a configuration that is not an object or a must_understand that is not a boolean or is false each a problem. The second needs context: the identifier in the field is related to a concrete class through it, and that class owns the validation routine, its coerce, which is handed the field.

What comes back is the entity, or an Opaque saying why not. A name no class in context claims is out_of_scope -- an unmodelled extension, left unjudged, which is what makes the format open. A name claimed and refused, or of another kind than this position takes, is invalid, for the reasons reported alongside; so is a value that is not JSON, or names no entity. loc prefixes the problems, so they point at where in the containing configuration the field sat.

Source code in src/zarr_metadata/v3/_entity.py

def resolve(
    data: object,
    kind: type[EntityT],
    context: Context,
    loc: Loc = (),
) -> tuple[EntityT | Opaque, tuple[ValidationProblem, ...]]:
    """`data`, one metadata field, read as an entity of `kind` in `context`.

    The reader for one field: the first two layers of reading a
    document, applied to a field on its own. The first needs nothing but
    the value -- `data` is refined to JSON, arrays as tuples, and judged
    as a metadata field, an extra member, a `configuration` that is not
    an object or a `must_understand` that is not a boolean or is `false`
    each a problem. The second needs `context`: the identifier in the
    field is related to a concrete class through it, and that class owns
    the validation routine, its `coerce`, which is handed the field.

    What comes back is the entity, or an `Opaque` saying why not. A name
    no class in `context` claims is `out_of_scope` -- an unmodelled
    extension, left unjudged, which is what makes the format open. A
    name claimed and refused, or of another kind than this position
    takes, is `invalid`, for the reasons reported alongside; so is a
    value that is not JSON, or names no entity. `loc` prefixes the
    problems, so they point at where in the containing configuration the
    field sat.
    """
    refined, problems = refine_json(data, loc)
    if refined is None:
        return Opaque.create_unchecked(None, "invalid"), problems
    return _resolve_field(refined, kind, context, loc)

well_formed_array_v3

well_formed_array_v3(
    value: object,
) -> tuple[
    Mapping[str, JSONValue] | None,
    tuple[ValidationProblem, ...],
]

The first layer: value as a refined v3 array document, with every structural problem.

Needs nothing but the value. The JSON is refined -- arrays as tuples, string keys, floats finite except in the attributes, which are user data (refine_node_json) -- and the document's shape is judged by the model layer: the keys a v3 array has, the shapes their values take, the envelope of each extension point. What comes back is refined JSON that the next layer reads without normalizing or judging JSON-ness again, and the structural problems beside it, which do not stop the next layer from reading what it can. A value that is not JSON, or not an object, is None with the reasons: not JSON is the first verdict, and there is nothing to read.

Source code in src/zarr_metadata/v3/_document.py

def well_formed_array_v3(
    value: object,
) -> tuple[Mapping[str, JSONValue] | None, tuple[ValidationProblem, ...]]:
    """The first layer: `value` as a refined v3 array document, with every structural problem.

    Needs nothing but the value. The JSON is refined -- arrays as
    tuples, string keys, floats finite except in the attributes, which
    are user data (`refine_node_json`) -- and the document's shape is
    judged by the model layer: the keys a v3 array has, the shapes their
    values take, the envelope of each extension point. What comes back
    is refined JSON that the next layer reads without normalizing or
    judging JSON-ness again, and the structural problems beside it, which
    do not stop the next layer from reading what it can. A value that is
    not JSON, or not an object, is None with the reasons: not JSON is
    the first verdict, and there is nothing to read.
    """
    refined, problems = refine_node_json(value)
    if refined is None:
        return None, problems
    if not isinstance(refined, Mapping):
        return None, problem((), f"expected a v3 array document as an object, got {refined!r}")
    document = cast("Mapping[str, JSONValue]", refined)
    return document, validate_array_metadata_v3_structure(document)

within

within(
    prefix: Loc, problems: Sequence[ValidationProblem]
) -> tuple[ValidationProblem, ...]

One entity's problems, located in the document that holds it.

An entity reports relative to its own configuration, so that is what goes between the field and the member. A problem with an empty location is about the entity itself -- a malformed r<N> name, a codec that cannot encode what reaches it -- and lands on the field.

Source code in src/zarr_metadata/v3/_entity.py

def within(prefix: Loc, problems: Sequence[ValidationProblem]) -> tuple[ValidationProblem, ...]:
    """One entity's problems, located in the document that holds it.

    An entity reports relative to its own `configuration`, so that is what
    goes between the field and the member. A problem with an empty
    location is about the entity itself -- a malformed `r<N>` name, a
    codec that cannot encode what reaches it -- and lands on the field.
    """
    return tuple(
        ValidationProblem(
            (*prefix, *(("configuration", *found.loc) if len(found.loc) != 0 else ())),
            found.message,
            found.kind,
        )
        for found in problems
    )