caput.memdata.fileformats#

Interface for file formats supported by memdata.

fileformats provides a unified interface for reading and writing data in file formats supported by memdata. The intention of this module is to abstract away the details of each file format, providing a consistent API for users and other parts of memdata.

Fileformats are implemented as subclasses of FileFormat, using the name of the file format as the subclass name.

Supported Backends#

h5py
zarr

Examples#

Opening and writing to a file:

>>> from caput.memdata import fileformats
>>>
>>> with fileformats.HDF5.open("example.h5", "w") as fh:
...     fh["dataset"] = range(10)

Compression#

File compression and chunking is specified using compression_kwargs().

The backends implemented in fileformats use the bitshuffle compression filter by default, if installed. Otherwise, no compression is used. Additional compression filters can be implemented by subclassing FileFormat and overriding the compression_kwargs() method.

Classes#

`FileFormat`	Abstract base class for file formats supported by this module.
`HDF5`	Interface for using the HDF5 file format.
`Zarr`	Interface for using zarr file format from caput.
`ZarrProcessSynchronizer`	A context manager for Zarr's ProcessSynchronizer that removes the lock files when done.

Functions#

`check_file_format`(filename, file_format, data)	Attempt to detect the format of a file.
`guess_file_format`(name[, default])	Guess the file format from the file name.
`remove_file_or_dir`(name)	Remove the file or directory with the given name.